Information processing device, face authentication system and face authentication device

Abstract

To provide an information processing device, a face authentication system and a face authentication device for improving robustness against a face attitude.SOLUTION: A face authentication system 100 includes an imaging unit 110 and an information processing unit 1000. The information processing unit 1000 detects the landmark of a face of an individual person from a face image of the individual person subjected to authentication, creates a face surface region that is a collection of regions interconnecting respective landmarks with each other, detects a blocked region that has at least a part of the face surface region blocked by the face surface region itself, creates a face normalized image by performing normalization on the face image of the individual person so as to match with the reference face attitude in a reference face image, performs masking on a region corresponding to the blocked region in the face normalized image, and performs face authentication using the face normalized image having undergone the masking.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a face authentication system, and a face authentication device. More particularly, the present invention relates to an information processing device, a face authentication system, and a face authentication device that identify individuals using facial images. [Background technology]

[0002] Facial recognition technology, a type of biometric authentication that uses biometric information, is a technology that registers a user's facial image in advance and determines whether the person is who they claim to be by comparing the facial image extracted from the person at the time of authentication with the registered facial image. Facial recognition technology has the advantage of being able to extract facial images from a distance, allowing for contactless authentication, and unlike fingerprint authentication, does not require the user to take any action to authenticate. The applications of facial recognition technology are expanding, including entrance / exit management in offices, educational institutions, event venues, etc., displaying personalized information using digital signage, and surveillance and security using surveillance camera footage.

[0003] In particular, in applications such as face recognition using digital signage or surveillance camera footage, the facial pose at the time of authentication is not limited to a frontal view, and it is expected that the face may be facing in various directions, so there is a demand for a system that can perform face recognition even when the face is not facing forward.

[0004] To address the above-mentioned problem, for example, Patent Document 1 discloses a technology for identifying individuals from various facial poses by normalizing the facial pose of a person's facial image so that it matches a reference facial pose (for example, a frontal face) and using an image in which the facial pose is aligned to a predetermined direction. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2015-125731 A Summary of the Invention [Problem to be solved by the invention]

[0006] The technique described in Patent Document 1 directly normalizes the facial pose of a facial image to match a frontal face, resulting in the generation of noise regions when applied to a facial image with occluded areas where the facial pose is significantly different from the frontal face. The generation of noise regions will be explained using an example of normalizing a left-facing facial image to a frontal face image. In a left-facing facial image, the area around the left cheek is occluded by the subject's own face area, resulting in an occluded area that is not visible in the facial image. Because pixel information is missing from the occluded area, normalizing the area to the corresponding area of ​​the frontal face results in distortion, resulting in the generation of noise regions that become a factor in identifying individuals. Since such noise regions can reduce authentication accuracy, a face authentication system that can perform normalization without generating noise regions, even for facial images with occluded areas where the facial pose is significantly different from the frontal face, and that is highly robust against facial pose is desired.

[0007] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an information processing device, a face recognition system, and a face recognition device that are robust against facial pose when identifying an individual using a face image, by being able to perform face pose normalization without generating noise areas even for face images in which the face pose differs significantly from the front and occluded areas occur, and by identifying an individual using a face image in which the face pose is aligned in a predetermined direction. [Means for solving the problem]

[0008] In order to solve the above problem, an information processing device of the present invention is an information processing device that includes a calculation device that performs face authentication to identify an individual to be authenticated, and the calculation device is configured to: acquire a face image of the individual to be authenticated and a reference face image for authentication that includes a reference face pose; detect landmarks of the individual's face from the face image of the individual; generate a face surface area that is a collection of areas connecting the landmarks; detect an occluded area in which at least a part of the face surface area is occluded by the face surface area itself; create a normalized face image for authentication by normalizing the individual's face image so that it matches the reference face pose of the reference face image for authentication; mask an area in the normalized face image for authentication that corresponds to the occluded area; and perform the face authentication using the masked normalized face image for authentication.

[0009] a reference face image for authentication that includes a face pose and a face image capturing device that captures a face image of an individual to be authenticated; a face image capturing device that captures a face image of an individual to be authenticated; an information processing device that performs face authentication to identify the individual; and a face authentication system configured to be able to transmit data to and receive data from the image capturing device and the information processing device. The information processing device is configured to: acquire, from the image capturing device, the face image of the individual to be authenticated that has been captured by the image capturing device; acquire an authentication reference face image that includes a reference face pose; detect facial landmarks of the individual from the face image; generate a face surface area that is a collection of areas connecting each landmark; detect an occluded area in which at least a part of the face surface area is occluded by the face surface area itself; create a normalized face image for authentication by normalizing the face image of the individual to match the reference face pose of the authentication reference face image; mask an area in the normalized face image for authentication that corresponds to the occluded area; and perform the face authentication using the masked normalized face image for authentication.

[0010] The face recognition device of the present invention is a face recognition device having an imaging device that captures a face image of an individual to be authenticated, and an information processing device that performs face recognition to identify the individual to be authenticated, wherein the information processing device is configured to: acquire the face image of the individual to be authenticated captured by the imaging device and a reference face image for authentication including a reference face pose; detect landmarks of the individual's face from the face image of the individual; generate a face surface area that is a collection of areas connecting each landmark; detect an occluded area in which at least a part of the face surface area is occluded by the face surface area itself; create a normalized face image for authentication by normalizing the face image of the individual so that it matches the reference face pose of the reference face image for authentication; mask an area in the normalized face image for authentication that corresponds to the occluded area; and perform the face recognition using the masked normalized face image for authentication. [Effects of the Invention]

[0011] According to the present invention, when identifying an individual using a face image, even if the face pose is significantly different from the front and an occluded area occurs, the face pose of the face image is normalized to match a reference face pose and the face pose is aligned to a predetermined direction, thereby improving robustness against the face pose and providing a highly accurate face authentication system. Problems, configurations, and effects other than those described above will become clear from the description of the following embodiments. [Brief explanation of the drawings]

[0012] [Figure 1A] FIG. 1A is a functional block diagram showing the configuration of a face authentication system according to the first embodiment. [Figure 1B] FIG. 1B is a diagram illustrating an example of the hardware configuration of a computer applied to the information processing unit. [Figure 2] FIG. 2 is a flowchart illustrating the operation of the face authentication system according to the first embodiment at the time of face registration. [Figure 3A] FIG. 3A is a diagram showing an example of face landmarks. [Figure 3B] FIG. 3B is a diagram showing an example of face landmarks. [Figure 4A] FIG. 4A is a diagram showing an example of a face surface. [Figure 4B] FIG. 4B is a diagram showing an example of a face surface. [Figure 5] FIG. 5 is a diagram showing an example of a shielded area. [Figure 6] FIG. 6 is a diagram illustrating a method for detecting an occluded area by the occluded area detection unit according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating a face pose normalization method performed by the face pose normalization unit according to the first embodiment. [Figure 8] FIG. 8 is a flowchart illustrating the operation of the face authentication system according to the first embodiment during face authentication. [Figure 9] FIG. 9 is a functional block diagram showing the configuration of a face authentication system according to the second embodiment. [Figure 10] FIG. 10 is a flowchart illustrating the operation of the face authentication system according to the second embodiment at the time of face registration. [Figure 11] FIG. 11 is a flowchart illustrating the operation of the face authentication system according to the second embodiment during face authentication. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. In all drawings of the embodiments, the same or corresponding parts may be designated by the same reference numerals. In the following description, processing may be described using functional blocks as the subject, but the subject of processing may be a CPU or a device instead of a functional block.

[0014] <<Embodiment 1>> First Embodiment: Configuration of Face Authentication System 100 1A is a block diagram showing the overall configuration of a face authentication system 100 according to Embodiment 1 of the present invention. The face authentication system 100 is a system that identifies individuals using face images.

[0015] The face authentication system 100 includes an imaging unit 110 and an information processing unit 1000. The imaging unit 110 and the information processing unit 1000 are connected via a network (not shown) so as to be able to transmit and receive data to and from each other.

[0016] The imaging unit 110 may be any unit that can acquire a two-dimensional image in which luminance information is stored in the X and Y directions. The imaging unit 110 is, for example, a camera (image capture device) that captures a facial image of a user present within an imaging range.

[0017] The information processing unit 1000 includes a face normalization unit 120, a data storage unit 130, a face registration unit 140, and a face authentication unit 150.

[0018] The face normalization unit 120 includes a face landmark detection unit 121 , an occluded region detection unit 122 , and a face normalization image generation unit 123 .

[0019] The data storage unit 130 includes (stores (memorizes)) a reference face image 131 and a normalized face image 132.

[0020] The reference face image 131 is a face image including a face pose that serves as a reference when normalizing the face pose. The reference face image 131 is, for example, a face image facing straight ahead.

[0021] Here, the facial pose includes the facial direction (yaw, roll, pitch) and the positions of facial features (X, Y, Z coordinates).

[0022] The normalized face image 132 is a face image in which the face pose of the face image of a person included in a captured image acquired by the imaging unit 110 has been normalized to match the face pose of the reference face image 131. In the normalized face image 132, if the face pose of the acquired face image of the person is significantly different from that seen from the front and there is an occluded area that is not visible in the face image, the face area after face pose normalization corresponding to the occluded area is masked with a predetermined color. This is because if the occluded area is directly normalized to the corresponding area in the reference face image 131, the face area after face pose normalization corresponding to the occluded area will be distorted and become a noise area, which will be a factor in reducing authentication accuracy. Details of this normalization method will be described later.

[0023] The reference face image 131 and the normalized face image 132 may be at least one of a two-dimensional image and feature amounts output from a feature extractor. The feature amounts in the reference face image 131 are, for example, facial landmarks that indicate the three-dimensional positions of each facial feature, such as the eyes, eyebrows, nose, mouth, chin, and contours. The feature amounts in the normalized face image 132 are, for example, feature vectors representing facial features output from a trained neural network (NN) model when the normalized face image 132 is input into the trained NN model.

[0024] The face authentication system 100 may include two or more data storage units 130. The data storage unit 130 may be located externally, and data may be received wirelessly via a data base station 160.

[0025] This embodiment can be applied to either 1:1 authentication, in which input biometric information and biometric information to be compared are in a one-to-one relationship, or 1:N authentication, in which there is a one-to-N relationship between the input biometric information and the biometric information to be compared. In 1:1 authentication, for example, a card and biometric information are used to compare the input biometric information with the biometric information registered on the card. In 1:N authentication, for example, the input biometric information is sequentially compared with all biometric information registered in a database to uniquely identify the closest user.

[0026] The configuration of the face authentication system 100 has been described above.

[0027] 1B is a diagram showing an example of the hardware configuration of a computer 2000 applied to the information processing unit 1000. Note that the information processing unit 1000 may be configured with one computer 2000 or multiple computers 2000.

[0028] The computer 2000 may be referred to as an "information processing device." The computer 2000 includes a CPU 2001, a ROM 2002, a RAM 2003, a non-volatile storage device 2004 that can read and write data, a network interface 2005, and an input / output interface 2006. These are connected to each other via a bus 2007 so that they can communicate with each other.

[0029] The CPU 2001 is a computing device that realizes various functions by loading various programs (not shown) stored in the ROM 2002 and / or storage device 2004 into the RAM 2003 and executing the programs loaded into the RAM 2003. As described above, the various programs executed by the CPU 2001 are loaded into the RAM 2003, and data used when the CPU 2001 executes the various programs is temporarily stored in the RAM 2003. The ROM 2002 and / or storage device 2004 are non-volatile storage media, and various programs are stored in the ROM 2002 and / or storage device 2004.

[0030] The network interface 2005 is an interface for connecting the computer 2000 to a network. The input / output interface 2006 is an interface for connecting the computer 2000 to an operation device and a display (display device) capable of displaying images.

[0031] The computer 2000 may be a virtual computer on the cloud. Instead of the computer 2000, a hardware device configured as a part or the whole of the computer 2000 using an FPGA (Field Programmable Gate Array) or the like may be used.

[0032] The face normalization unit 120 (the face landmark detection unit 121, the occluded area detection unit 122, and the face normalized image generation unit 123), the face registration unit 140, and the face authentication unit 150 correspond to the programs. The data storage unit 130 corresponds to the memory device 2004.

[0033] <First Embodiment: Operation of Face Authentication System 100 During Face Registration> Next, the operation of the face authentication system 100 during face registration will be described with reference to the flowchart shown in FIG.

[0034] The captured image captured by the imaging unit 110 is input to the face normalization unit 120 (S200). In the face normalization unit 120, the facial landmark detection unit 121 detects facial landmarks of the person to be registered from the captured image (S201). At this time, the detection result is judged (S202). If the captured image does not include a person and facial landmarks cannot be detected, the process ends. If facial landmarks are detected in S203, a facial surface is generated based on the detected facial landmarks (S203). Next, the facial landmark detection unit 121 inputs the detected facial landmarks and the generated facial surface to the occluded area detection unit 122. The occluded area detection unit 122 detects occluded areas that occur when the facial pose is significantly different from the front (S204). An example of the occluded area is the area around the left cheek in a face image facing left. Information about the detected occluded area is input to the face normalized image generation unit 123. The normalized face image generating unit 123 acquires a reference face image 131 including a face pose that serves as a reference when normalizing the face pose (S205).

[0035] The face pose of the person included in the captured image is normalized to match the face pose of the acquired reference face image 131 (S206). Because the occluded area detected in S204 lacks pixel information, normalizing the occluded area to match the reference face pose results in distorted areas (the aforementioned noise areas). Therefore, the normalized face area corresponding to the occluded area is masked with a predetermined color to eliminate the noise area and normalize the face pose. Next, the face image after normalization, including masking the face area corresponding to the occluded area, is subjected to processing to make it more suitable for face recognition, thereby generating a normalized face image 132 (S207). Here, the processing to make it more suitable for face recognition includes at least one of image cropping, resizing, noise and blur removal, contrast adjustment, gamma correction, backlight correction, distortion correction, and background area removal. The normalized face image 132 generated in S207 is input to the face registration unit 140. The face registration unit 140 generates authentication data based on the normalized face image 132 (S208) and records it in the data storage unit 130 (S209). When recording the data in the data storage unit 130 in S209, encryption processing or the like may be performed to improve security. The authentication data may be the normalized face image 132 itself, or features extracted from the normalized face image 132 may be used as the authentication data. In the former case, images are compared, and in the latter case, features are compared. Note that the authentication data may be referred to as "face data," and the authentication data stored in the data storage unit 130 may be referred to as "registered face data." The processing from S203 onward may be performed using a face image in which a predetermined face region has been extracted based on the results of the facial landmark detection in S201. The above is the operation of face registration in the face authentication system 100.

[0036] The following describes in detail the steps included in the flowchart of Fig. 2. Steps that do not require detailed explanation will be omitted.

[0037] <First Embodiment: Facial Landmark Detection S201> The facial landmarks detected in facial landmark detection S201 are feature points that indicate the positions of specific facial parts and features, such as the eyes, eyebrows, nose, mouth, chin, and contours, included in the facial region. In this embodiment, in S201, the facial landmarks are acquired as three-dimensional coordinates (X, Y, Z) in a predetermined three-dimensional coordinate space. In the following description, the X, Y, and Z directions are, for example, coordinates on the image plane, and the Z direction is a depth coordinate perpendicular to the image plane. Each facial landmark is assigned a unique number according to the position of a facial feature. For example, the landmark at the tip of the nose is assigned the number 5. FIGS. 3A and 3B show examples of facial landmarks. Facial landmark 300 in FIG. 3A shows a facial pose when the face is facing forward, while facial landmark 301 in FIG. 3B shows a facial pose when the face image is viewed from the front and facing left. Facial landmarks 300 and 301 show an example of detecting feature points for 468 facial features, but any detected position of at least one facial feature can be used as a facial landmark. A known method can be used to input a two-dimensional image and detect facial landmarks having three-dimensional coordinates. For example, a method using pattern matching based on an Active Appearance Model, a feature point extraction method based on Regression Trees, or a detector based on a neural network model. This concludes the description of facial landmark detection S201.

[0038] <First Embodiment: Facial Surface Generation S203> Facial surface generation S203 is a step for generating a facial surface to be used when detecting an occluded area in occluded area detection S204. In S203, a facial surface representing the surface shape of the face is generated based on the three-dimensional facial landmarks detected in facial landmark detection S201. The facial surface can be, for example, a set of triangles (Delaunay triangles) obtained by dividing a set of three-dimensional facial landmark points into triangles, a set of polygons, or a spline surface obtained by combining curves (spline curves) connecting the set of three-dimensional facial landmark points. Since well-known methods can be used to generate Delaunay triangles or spline surfaces from a set of points, a description thereof will be omitted here. FIGS. 4A and 4B show examples of facial surfaces generated using Delaunay triangles. The facial surface 400 in FIG. 4A shows a case where the facial pose is forward, while the facial surface 401 in FIG. 4B shows a case where the facial pose is leftward when viewing the face image from the front.

[0039] <First Embodiment: Occluded Area Detection S204> Occluded area detection S204 is a step for detecting an occluded area using the positional relationship between the facial surface generated in facial surface generation S203 and three-dimensional facial landmarks. The occluded area will be described in detail with reference to FIG. 5. When a facial surface 401 is turned significantly to the left from the front, a portion of the facial surface 401 is occluded by the facial surface 401 itself, resulting in an occluded area 500. The occluded area 500 is an area that cannot be seen in the image, and therefore is an area lacking pixel information. For example, in FIG. 5, a Delaunay triangle including a facial landmark 502 (star mark in FIG. 5) on the occluded area 500 constitutes the occluded area 500, and is an area lacking pixel information. On the other hand, a Delaunay triangle including a facial landmark 501 (square mark in FIG. 5) on the obverse side area that is not occluded by the facial surface 401 itself is an area containing pixel information.

[0040] In the face pose normalization S206, if the occluded region 500 is directly normalized to the corresponding region in the reference face image 131, the corresponding face region after face pose normalization will be distorted and a noise region will occur. Therefore, the occluded region 500 is detected in this step (S204) to prevent the occurrence of a noise region during face pose normalization in S206.

[0041] Next, a method for detecting an occluded region 500 using a facial surface and three-dimensional facial landmarks will be described with reference to FIG. 6. FIG. 6 shows a view of the facial surface 401, in which the facial pose in FIG. 5 is facing far left from the front, rotated -90 degrees around the Y axis (the right-hand screw direction is positive), as viewed from the YZ plane. The star marks in FIG. 6 correspond to the facial landmarks 502 on the occluded region 500 shown as an example in FIG. 5, and the square marks in FIG. 6 correspond to the facial landmarks 501 on the front side region shown as an example in FIG. 5. In this method, a reference position 600 is set in front of the facial surface 401 (in the +Z direction in the example in FIG. 6). Straight lines are connected between the reference position 600 and each facial landmark, and it is determined whether the straight lines intersect with the facial surface 401. Since the occluded region 500 is an area occluded by the facial surface 401 itself, if the straight lines intersect with the facial surface 401, it can be determined that the facial landmark forming one end of the line is located on the occluded region 500. On the other hand, if the line does not intersect with the facial surface 401, it can be determined that the facial landmark constituting one end of the line exists in the front-side region. For example, in the case of Fig. 6, the facial landmark 502 on the occluded region 500 has an intersection 601 with a Delaunay triangle constituting the facial surface 401 on a line (dashed line in Fig. 6) connecting the facial landmark 502 to the reference position 600, so it can be determined that the facial landmark 502 exists in the occluded region 500, and the Delaunay triangle including the facial landmark 502 constitutes the occluded region 500. On the other hand, the facial landmark 501 on the front-side region does not have an intersection with the Delaunay triangle constituting the facial surface 401 on a line (dotted line in Fig. 6) connecting the facial landmark 501 to the reference position 600, so it can be determined that the facial landmark 501 exists in the front-side region, and the Delaunay triangle including the facial landmark 501 constitutes the front-side region.

[0042] Here, the reference position 600 can be set arbitrarily, but in this method, the closer the straight line connecting the reference position 600 and each facial landmark is to a perpendicular line to the image plane (the XY plane in the example of FIG. 6), the better the detection accuracy, and therefore it is desirable that the Z coordinate of the reference position 600 set in front of the facial surface 401 is as close to infinity as possible. Also, the reference position 600 may be set for each facial landmark, and in this case, if the X and Y coordinates of the reference position 600 are made the same as the X and Y coordinates of each facial landmark, the straight line connecting the reference position 600 and each facial landmark will be equivalent to a perpendicular line to the image plane, which is desirable for good accuracy.

[0043] As described above, for each facial landmark that constitutes the facial surface 401, by determining whether or not a line connecting the reference position 600 and the facial surface 401 (in the example of Figure 6, a Delaunay triangle that constitutes the facial surface 401) has an intersection 601, it is possible to determine whether or not the facial landmark is located on the occluded area 500, and an area including a facial landmark that is determined to be located on the occluded area 500 can be detected as an area that constitutes the occluded area 500 (in the example of Figure 6, a Delaunay triangle including the facial landmark 502).

[0044] The method in this step is highly reliable because it can detect the occluded area 500 based on the positional relationship between the face surface 401 and face landmarks without using a threshold value.

[0045] This concludes the detailed description of the occluded area detection step S204.

[0046] <First Embodiment: Face Pose Normalization S206> In face pose normalization S206, the face pose of the person included in the captured image is normalized based on the detection result of occluded area detection S204 so that it matches the face pose of the reference face image 131 acquired in S205. The face pose normalization method will be described with reference to FIG. 7. In FIG. 7, the left side of the figure is a face image before normalization in which the face pose is facing left when viewed from the front, and the right side is a face image including a face pose facing forward, which serves as the reference for normalization. In the case of a face landmark 501 in the front side area (a square mark in FIG. 7), the shape of the area 700 including the face landmark 501 is converted so that it matches the shape of the area 701 including the corresponding face landmark (face landmark with the same number) in the reference face image. By performing a similar shape conversion process on each face landmark in the front side area, face images in which the face pose is not frontal but the positions of each facial feature are different can be normalized to a frontal face pose. In the case of facial landmark 502 on occluded region 500 (star mark in FIG. 7), pixel information is missing in occluded region 500, so if shape conversion processing is performed in the same way as for facial landmark 501 on the obverse region, distortion occurs in the corresponding region after normalization, resulting in a noise region. For this reason, a predetermined masking process is performed on region 702 in the reference face image that includes a facial landmark corresponding to (having the same number as) facial landmark 502. By performing a similar masking process on each facial landmark on occluded region 500, it is possible to avoid the occurrence of noise regions that can reduce authentication accuracy.

[0047] The shape transformation process can use known image shape transformation methods such as affine transformation and homography transformation.

[0048] The masking process may be performed using, for example, black, white, skin color, or the average value of the facial image, etc. In addition, masking may be performed using a region corresponding to the facial landmark 502 on the occluded region 500 included in the other facial image, such as a partial region of the reference facial image 131.

[0049] This normalization method can normalize not only the facial pose but also the positions of facial features so that they match the reference face image 131. Therefore, even if the facial expression varies, such as a face with an open mouth, a face with turned-up mouth corners, a face with wide-open eyes, or a face with lowered eyebrows, it can be normalized to a neutral face. By using this normalized image to identify individuals and aligning the facial expression at the time of registration and authentication, the robustness of face recognition can be improved even with respect to facial expressions.

[0050] This concludes the detailed description of face pose normalization S206.

[0051] <First Embodiment: Matching Data Generation S208> In the matching data generation S208, matching data is generated based on the face normalized image 132 generated in the face normalized image generation S207. The generated matching data is converted into a format suitable for the algorithm adopted by the face authentication unit 150, and may be a feature vector or an image. An example of generating a feature vector is described below. In S208, the input face normalized image 132 is input to a predetermined feature extractor, thereby generating a feature vector representing the facial features of the face normalized image 132. Examples of feature extractors that can be used include those based on an NN model trained using training data, or an algorithm based on Local Binary Patterns. Here, it is preferable to use a feature extractor trained on face images generated using the face normalization technique described above in the face normalization unit 120, as this will enable the generation of more accurate feature vectors.

[0052] This concludes the detailed description of the verification data generation S208.

[0053] <First Embodiment: Operation of Face Recognition System 100 During Face Recognition> Next, the operation of face authentication in the face authentication system 100 will be described using the flowchart shown in Fig. 8. Compared to the flowchart showing the operation of face registration shown in Fig. 2, S800 to S803 have been added. In Fig. 8, steps having the same functions as those in Fig. 2 are given the same reference numerals, and detailed description thereof will be omitted. The following mainly describes the steps (S800 to S803) added to Fig. 2.

[0054] During face authentication, as in face registration, steps S200 to S207 are performed on the image captured by imaging unit 110, and a face normalized image 132 is generated using a face normalization method based on the detection result of occluded region 500 described above. Face normalized image 132 generated by face normalization unit 120 is input to face authentication unit 150, which executes steps S208 and S800 to S803. As in face registration, face authentication unit 150 generates matching data (authentication data) for generated face normalized image 132 (S208). This matching data (authentication data) may be referred to as "authentication face data." After generating the matching data, the matching data (authentication data) recorded at the time of registration in data storage unit 130 during face registration is acquired (S800). Here, in the case of 1:N authentication, a plurality of pieces of matching data at the time of registration are acquired in S800. By calculating the similarity between the verification data generated at the time of authentication and the verification data at the time of registration obtained from the data storage unit 130, it is possible to verify whether the person included in the captured image at the time of authentication is a registered person (S801). The result is output to an output device or other system as an authentication result and presented to the user (S802). Methods that can be used to present the authentication result to the user include displaying it on a display, sounding it through a sound system, vibrating it through a vibration function, lighting it up using an LED light, etc.

[0055] Here, when normalizing the face pose (S206), it is desirable to use the reference face image 131 used during face registration in order to align the face pose with that during face registration. By aligning the face pose between registration and authentication, the accuracy of deriving similarity increases regardless of the face pose, and robustness to the face pose can be improved.

[0056] During face recognition, the face pose is not limited to a frontal face, and various face poses are expected, so this normalization method, which does not generate noise areas based on the detection results of the occluded area 500 described above, is particularly useful.

[0057] Although not shown in Fig. 8, the face authentication unit 150 may perform attribute estimation processing such as age and gender on the face normalized image 132 generated in S207. The face normalized image 132 generated in this embodiment does not produce noise areas even when the face pose is significantly different from the front, and the face pose can be aligned to a predetermined direction, making it useful for attribute estimation. For attribute estimation processing such as age and gender, a known method using a trained NN model trained with a data set of predetermined attributes can be used, for example.

[0058] The operations performed by the face authentication system 100 during face authentication have been described above.

[0059] The following describes in detail the steps included in the flowchart of Fig. 8. Steps that do not require detailed explanation will be omitted.

[0060] <First Embodiment: Face Matching Process S801> For the calculation of similarity in the face matching process S801, for example, if the authentication data is a feature vector, known calculation methods such as Euclidean Norm or Cosine Similarity can be used. For image morphology, known methods based on pattern recognition or image luminance distribution vectors can be used. Since similarity calculation based on feature vectors generally requires less calculation effort than when image morphology is used, similarity calculation using feature vectors is useful when speeding up matching processing. To improve authentication accuracy, similarity using feature vectors and similarity using image morphology may be combined. Specifically, similarity using feature vectors and similarity using image morphology may be calculated separately, and the sum of these values ​​based on a predetermined weighting may be used as the similarity.

[0061] In S801, in order to eliminate differences in masking areas included in normalized face image 132 at the time of enrollment and at the time of authentication, the similarity may be calculated using face images in which the masking areas are the same at the time of enrollment and at the time of authentication. Specifically, the masking area included in normalized face image 132 at the time of enrollment is superimposed on normalized face image 132 at the time of authentication, and the masking area included in normalized face image 132 at the time of authentication is superimposed on normalized face image 132 at the time of enrollment.

[0062] This concludes the detailed description of the face matching process S801.

[0063] <First Embodiment: Output of Authentication Result S802> In the authentication result output S802, face authentication is performed by checking whether the similarity calculated in the face matching process S801 satisfies a predetermined threshold condition. For example, the following authentication methods can be used. In the case of 1:1 authentication, if the calculated similarity satisfies a predetermined threshold condition, the person included in the captured image is determined to be a registered person. If the threshold condition is not met, it is determined as "mismatched person." In the case of 1:N authentication, among registered people whose calculated similarity satisfies a predetermined threshold condition, the person with the greatest similarity is determined to be the person at the time of authentication. If the threshold condition is not met, it is determined as "unregistered." This concludes the detailed explanation of the authentication result output S802.

[0064] <First embodiment: Summary> The face authentication system 100 according to the first embodiment can normalize various face poses to the frontal face pose without generating noise areas, even when the face pose is significantly different from the frontal face pose and an occluded area 500 with missing pixel information occurs. Therefore, since the face poses can be aligned during authentication, the accuracy of deriving similarity increases regardless of differences in face pose at the time of capture, enabling face authentication that is robust against face poses.

[0065] <<Embodiment 2>> <Second Embodiment: Configuration of Face Authentication System 100> In the second embodiment of the present invention, a configuration for quickly performing face normalization processing in the present invention by detecting the face direction will be described.

[0066] Fig. 9 is a diagram showing a detailed configuration of face authentication system 100 in the second embodiment. Compared to the configuration of the first embodiment shown in Fig. 1, this configuration has a face direction detection unit 900 added to the face normalization unit 120. The other configuration is the same as that of the first embodiment. In the following, explanations of blocks that share processing with the first embodiment will be omitted by assigning the same reference numerals, and the internal processing of face normalization unit 120 equipped with face direction detection unit 900 will be mainly explained.

[0067] <Second embodiment: Operation of face normalization unit 120 equipped with face direction detection unit 900> The operation of face normalization unit 120 including face direction detection unit 900 will be described using the flowchart shown in Fig. 10, which shows the processing during face registration in the second embodiment. Compared to the flowchart in the first embodiment shown in Fig. 2, this flowchart has S1000 to S1002 added. Steps similar to those explained in the first embodiment are given the same step numbers, and explanations thereof will be omitted. The following mainly describes the differences between the first embodiment and the second embodiment.

[0068] After the facial landmark detection unit 121 detects facial landmarks (S201) and generates a facial surface (S203), the facial direction detection unit 900 detects the facial direction of a person included in a captured image (S1000). It is determined whether the detected facial direction is within a predetermined angle range (S1001). If it is within the predetermined angle range (YES in FIG. 10), the occluded area detection (S204) is skipped and the process proceeds to the step of acquiring a reference facial image (S205). On the other hand, if the predetermined angle is not satisfied (NO in FIG. 10), a process of narrowing down the facial landmarks to be targeted in the occluded area detection (S1002) is performed based on the detected facial direction. The occluded area detection unit 122 performs a process of detecting an occluded area 500 (S204) using the narrowed down facial landmarks. Based on the detection result of the occluded region 500, the normalized face image generating unit 123 generates the normalized face image 132 (S205 to S207).

[0069] The above is the operation of the face normalization unit 120 equipped with the face direction detection unit 900 during face registration.

[0070] In the flowchart shown in FIG. 11, which shows the processing during face authentication in the second embodiment, the above-mentioned steps S1000 to S1002 are added between the facial surface generation step S203 and the reference face image acquisition step S205 in the flowchart of FIG. 8, and the operation of the face normalization unit 120, including steps S1000 to S1002, is as described above.

[0071] Below, steps S1000 to S1002 included in the flowcharts of FIGS. 10 and 11 will be described in detail.

[0072] <Embodiment 2: Face Direction Detection S1000> The face direction detected by face direction detection S1000 is, for example, the angle (amount of rotation) in the yaw, roll, and pitch directions when facing directly ahead is set to 0°. A known method can be used to detect the face direction. For example, a method of estimation based on changes in the relative positions and arrangements of each landmark within the face landmarks, or a method using a trained NN model can be used. As the face landmarks are detected in S201, the detection results can be used.

[0073] This concludes the detailed description of face direction detection S1000.

[0074] <Embodiment 2: Determining Face Direction S1001> In S1001, it is determined whether the face direction detected in face direction detection S1000 is within a predetermined angle range. Here, the predetermined angle range can be set arbitrarily, but it is desirable to set it to an angle range in which the face direction can be considered to be approximately forward, such as an angle range of +10° to -10° in the yaw, roll, and pitch directions. This is because when the face of a person included in a captured image is facing approximately forward, occlusion region 500 does not occur (or the area of ​​occlusion region 500 is so small that its influence can be ignored), so the influence of the noise region during face normalization described above can be ignored, and occlusion region detection S204 may be skipped.

[0075] Therefore, if the face direction detected in S1000 is determined to be substantially forward in S1001, the occluded area detection S204 can be skipped, thereby reducing the amount of processing and speeding up the internal processing of the face normalization unit 120.

[0076] Note that in S1001, it may be determined whether the face direction is within a predetermined angle range as follows: It is determined whether at least any one of the angles (absolute value of the angle) in the yaw, roll, and pitch directions detected in S1000 is equal to or greater than a predetermined first threshold. If at least one of the at least any one of the angles is equal to or greater than the predetermined first threshold, it is determined that the face direction is not within the predetermined angle range, and if all of the at least any one of the angles are less than the predetermined first threshold, it is determined that the face direction is within the predetermined angle range.

[0077] This concludes the detailed explanation of S1001.

[0078] <Second Embodiment: Facial Landmark Narrowing Down S1002> In facial landmark narrowing down S1002, facial landmarks to be processed in occluded area detection S204 are narrowed down according to the facial direction detected in facial direction detection S1001. For example, if the facial pose is facing left when viewed from the front, an occluded area 500 occurs around the left cheek area. Facial landmarks that exist outside the area around the left cheek area exist in the front side area, so there is no need to determine whether they exist in the occluded area 500. Therefore, facial landmarks that may exist in the occluded area 500 are narrowed down to those around the left cheek, and the number of facial landmarks to be processed in S204 can be reduced.

[0079] As described above, by narrowing down in advance the facial landmarks that may exist in the occluded area 500 according to the face orientation detected in S1000, it is possible to reduce the computational load when detecting the occluded area in S204 and speed up the processing.

[0080] This concludes the detailed description of the facial landmark narrowing down step S1002.

[0081] <Embodiment 2: Summary> The face authentication system 100 according to the second embodiment includes a face direction detection unit 900. Since it is possible to skip occluded area detection S204 and narrow down the face landmarks to be processed in occluded area detection S204 according to the detected face direction, the processing speed in the face normalization unit 120 can be increased.

[0082] <Modifications of the present invention> The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0083] In the above embodiment, one image capturing unit 110 is shown in FIG. 1, but two image capturing units 110 may be provided for face registration and authentication.

[0084] In the above embodiment, the method of determining an occluded area is not limited to the above.

[0085] In the above embodiments, the authentication data (registered face data) stored in data storage unit 130 may be, for example, an unmasked face image (normalized face image) or data created based on an unmasked face image (normalized face image). In this case, when performing face authentication, an area of ​​the registered unmasked normalized face image corresponding to an occluded area detected in the face image for authentication may be masked, and this masked normalized face image (data created based on the normalized face image) may be compared with the masked normalized face image for authentication (data created based on the normalized face image), thereby performing face authentication.

[0086] In the above embodiment, the face authentication system 100 is illustrated as an integrated unit, but the imaging unit 110, face normalization unit 120, data storage unit 130, face registration unit 140, and face authentication unit 150 may be located in different spatial locations and may each communicate data wirelessly from a data base station 160.

[0087] In the above embodiments, the face normalization unit 120, face registration unit 140, and face authentication unit 150 (and the respective functional units arranged as internal blocks thereof) can be configured by hardware such as a circuit device that implements these functions, or can be configured by a computing device (e.g., Central Processing Unit: CPU) executing software that implements these functions.

[0088] The present invention can also have the following configuration.

[0089] [1] an imaging device that captures a facial image of an individual to be authenticated; an information processing device that performs face authentication to identify the individual; A face authentication device having: The information processing device includes: acquiring the face image of the individual to be authenticated, captured by the imaging device, and an authentication reference face image including a reference face pose; Detecting facial landmarks of the individual from the facial image of the individual, generating a facial surface area that is a collection of areas connecting the landmarks, and detecting an occluded area in which at least a part of the facial surface area is occluded by the facial surface area itself; creating a normalized face image for authentication by normalizing the face image of the individual so that it matches the reference face posture of the reference face image for authentication, masking an area in the normalized face image for authentication that corresponds to the occluded area, and performing the face authentication using the masked normalized face image for authentication; It was configured as follows: Facial recognition device.

[0090] [2] In the face authentication device according to [1], The information processing device includes: Acquires registered face data of one or more registered users from a storage device in which the registered face data is stored, the registered face data being data relating to the faces of a plurality of users; performing the face authentication by comparing the masked normalized face image for authentication or authentication face data, which is authentication data created based on the masked normalized face image for authentication, with one or more of the registered face data; It was configured as Facial recognition device.

[0091] [3] [2] In the face authentication device described in The information processing device includes: Acquire a registration face image of the user to be registered, captured by the imaging device, and a registration reference face image including a reference face pose; Detecting facial landmarks of the individual from the facial image for registration, generating a facial surface area that is a collection of areas connecting the landmarks, and detecting an occluded area in which at least a part of the facial surface area is occluded by the facial surface area itself; creating a normalized face image for registration by normalizing the face image for registration so that it matches the reference face posture of the reference face image for registration, masking an area in the normalized face image for registration that corresponds to the occluded area, and storing the masked normalized face image for registration or registration face data, which is authentication data for registration created based on the masked normalized face image for registration, as the registration face data in the storage device; It was configured as follows: Facial recognition device. [Explanation of symbols]

[0092] 100...Face recognition system, 110...Imaging unit, 120...Face normalization unit, 121...Face landmark detection unit, 122...Occluded area detection unit, 123...Face normalized image generation unit, 130...Data storage unit, 131...Reference face image, 132...Face normalized image, 140...Face registration unit, 150...Face recognition unit, 160...Data base station, 300...Frontal face landmark, 301...Left-facing face landmark, 400...Frontal face surface, 401...Left-facing face surface, 500...Occluded area, 501...Table Facial landmarks on the side area, 502...facial landmarks on the occluded area, 600...reference position, 601...intersection, 700...partial area of ​​the front side area, 701...partial area of ​​the front side area after normalization, 702...masking, 900...face direction detection unit, 1000...information processing unit, 2000...computer, 2001...CPU, 2002...ROM, 2003...RAM, 2004...storage device, 2005...network interface, 2006...input / output interface, 2007...bus

Claims

1. An information processing device including a computing device that performs face authentication to identify an individual to be authenticated, The computing device acquiring a face image of the individual to be authenticated and a reference face image for authentication including a reference face pose; Detecting landmarks of the face of the individual from the face image of the individual, generating a face surface area that is a collection of areas connecting the landmarks, and detecting an occluded area in which at least a part of the face surface area is occluded by the face surface area itself; creating a normalized face image for authentication by normalizing the individual's face image so that it matches the reference face posture of the authentication reference face image, masking an area in the normalized face image for authentication that corresponds to the occluded area, and performing the face authentication using the masked normalized face image for authentication; It was configured as follows: Information processing device.

2. 2. The information processing device according to claim 1, a storage device storing a plurality of registered face data relating to faces of a plurality of users; The computing device acquiring the registered face data of one or more registered users; performing the face authentication by comparing the masked normalized face image for authentication or authentication face data, which is authentication data created based on the masked normalized face image for authentication, with one or more of the registered face data; It was configured as follows: Information processing device.

3. 3. The information processing device according to claim 2, The computing device Acquire a registration face image of the user to be registered and a registration reference face image including a reference face pose; Detecting facial landmarks of the individual from the face image for registration, generating a facial surface area that is a collection of areas connecting the landmarks, and detecting an occluded area in which at least a part of the facial surface area is occluded by the facial surface area itself; creating a normalized face image for registration by normalizing the face image for registration so that it matches the reference face posture of the reference face image for registration, masking an area in the normalized face image for registration that corresponds to the occluded area, and storing the masked normalized face image for registration or registration face data, which is authentication data for registration created based on the masked normalized face image for registration, as the registration face data in the storage device; It was configured as follows: Information processing device.

4. 2. The information processing device according to claim 1, The computing device determining whether a line connecting a reference position set in front of the face surface area and the landmark has an intersection with a partial area of ​​the face surface area, detecting an area including the landmark having the intersection based on the determination as the occluded area, and generating the normalized face image for authentication based on the detected occluded area; It was configured as follows: Information processing device.

5. 4. The information processing device according to claim 3, The computing device determining whether a line connecting a reference position set in front of the face surface area and the landmark has an intersection with a partial area of ​​the face surface area, detecting an area including the landmark having an intersection based on the determination as the occluded area, and generating the normalized face image for registration based on the detected occluded area; It was configured as follows: Information processing device.

6. 4. The information processing device according to claim 3, The computing device The same reference face image is used as the authentication reference face image and the registration reference face image. It was configured as follows: Information processing device.

7. 2. The information processing device according to claim 1, The facial surface area comprises: the landmarks of the face of the individual are either a set of triangles obtained by dividing the landmarks of the face of the individual by triangles, a set of polygons obtained by dividing the landmarks of the face of the individual by polygons, or a spline surface formed by combining curves connecting the landmarks of the face of the individual. Information processing device.

8. 5. The information processing device according to claim 4, a straight line connecting the reference position and the landmark on the face of the individual is a normal to an image plane of the face image of the individual; Information processing device.

9. 2. The information processing device according to claim 1, The computing device generating the normalized face image for authentication from the individual's face image by converting the shape of a partial area of ​​the face surface area so that the shape matches the shape of a partial area of ​​the face surface area included in the reference face image for authentication; It was configured as follows: Information processing device.

10. 5. The information processing device according to claim 4, The computing device Detecting at least one of a yaw, a roll, and a pitch angle by which the facial posture of the individual is rotated with respect to a face of the individual facing forward based on the facial landmarks of the individual; If the detected rotation angle of the face pose of the individual is equal to or greater than a first threshold, performing the determination of the occluded area; If the rotation angle of the detected face pose of the individual is less than a first threshold, the determination of the occluded area and the detection of the occluded area are not performed, and the face authentication is performed using the unmasked normalized face image for authentication. It was configured as follows: Information processing device.

11. 11. The information processing device according to claim 10, The computing device narrowing down the area where the occlusion area may exist based on an angle at which the face pose of the individual is rotated; performing a determination of the occluded area for the landmarks included in the narrowed area; The determination of the occluded area is not performed for the landmarks included in the area excluding the narrowed area. It was configured as follows: Information processing device.

12. 4. The information processing device according to claim 3, The computing device further masking an area of ​​the masked normalized face image for registration that corresponds to the occluded area at the time of authentication, further masking an area of ​​the masked normalized face image for authentication that corresponds to the occluded area at the time of registration, and performing the face authentication using the further masked normalized face image for registration and the further masked normalized face image for authentication. It was configured as follows: Information processing device.

13. an imaging device that captures a facial image of an individual to be authenticated; an information processing device that performs face authentication to identify the individual; and A face authentication system configured so that the imaging device and the information processing device can transmit and receive data to each other, The information processing device includes: acquiring a facial image of the individual to be authenticated, the facial image being captured by the imaging device, from the imaging device; obtaining a reference face image for authentication including a reference face pose; Detecting landmarks of the face of the individual from the face image of the individual, generating a face surface area that is a collection of areas connecting the landmarks, and detecting an occluded area in which at least a part of the face surface area is occluded by the face surface area itself; creating a normalized face image for authentication by normalizing the individual's face image so that it matches the reference face posture of the authentication reference face image, masking an area in the normalized face image for authentication that corresponds to the occluded area, and performing the face authentication using the masked normalized face image for authentication; It was configured as follows: Facial recognition system.

14. The face authentication system according to claim 13, The information processing device includes: a storage device storing a plurality of registered face data relating to faces of a plurality of users; acquiring the registered face data of one or more registered users from the storage device; performing the face authentication by comparing the masked normalized face image for authentication or authentication face data, which is authentication data created based on the masked normalized face image for authentication, with one or more of the registered face data; It is configured as follows: The information processing device includes: acquiring, from an imaging device, a facial image for registration of the user to be registered, the facial image being captured by the same or a different imaging device as the imaging device that captures the facial image of the individual to be authenticated; obtaining a reference face image for registration including a reference face pose; Detecting facial landmarks of the individual from the face image for registration, generating a facial surface area that is a collection of areas connecting the landmarks, and detecting an occluded area in which at least a part of the facial surface area is occluded by the facial surface area itself; creating a normalized face image for registration by normalizing the face image for registration so that it matches the reference face posture of the reference face image for registration, masking an area in the normalized face image for registration that corresponds to the occluded area, and storing the masked normalized face image for registration or registration face data, which is authentication data for registration created based on the masked normalized face image for registration, as the registration face data in the storage device; It was configured as follows: Facial recognition system.

15. an imaging device that captures a facial image of an individual to be authenticated; an information processing device that performs face authentication to identify the individual; A face authentication device having: The information processing device includes: acquiring a face image of the individual to be authenticated, captured by the imaging device, and an authentication reference face image including a reference face pose; Detecting landmarks of the face of the individual from the face image of the individual, generating a face surface area that is a collection of areas connecting the landmarks, and detecting an occluded area in which at least a part of the face surface area is occluded by the face surface area itself; creating a normalized face image for authentication by normalizing the individual's face image so that it matches the reference face posture of the authentication reference face image, masking an area in the normalized face image for authentication that corresponds to the occluded area, and performing the face authentication using the masked normalized face image for authentication; It was configured as follows: Facial recognition device.

Images