IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, FACE AUTHENTICATION SYSTEM AND PROGRAM

Abstract

To provide an image processing device capable of extracting a face image of a living body suitable for face authentication by exactly distinguishing the living body from a non-living body from a photographed image, an image processing method, a face authentication system and a program.SOLUTION: An image processing device includes: an acquisition part for acquiring a first image generated by photographing a subject by using light of a first wavelength, a second image generated by photographing the subject by using light of a second wavelength, and depth information of the subject; a detection part for detecting a face photographed in the second image; a determination part for determining whether the face detected by the detection part is a face obtained by photographing a living body on the basis the depth information; and an extraction part for extracting a face image from the first image on the basis of information on the face determined, by the determination part, to be obtained by photographing the living body.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, a face authentication system, and a program. [Background technology]

[0002] Recently, biometric authentication, which uses biometric information, which is information on a person's physical characteristics and behavioral characteristics, has come to be used in situations where identity verification is required. Face recognition, which is one type of biometric authentication, has the advantages of having little psychological resistance from the person being authenticated, being able to authenticate from a distance, and having a psychological deterrent effect against fraud.

[0003] Facial recognition is a technology that collates information obtained from a facial image of a person taken by a camera with information on a registered person, and authenticates whether the photographed person is the same as the registered person. Patent Documents 1 and 2 disclose a facial recognition method using an infrared image as a technology to prevent so-called impersonation, which is an attempt to pass facial recognition using a non-biological object such as a photograph. Since printed matter such as a photograph does not absorb infrared light, it is possible to determine from the captured infrared image whether the subject is a living body or a photograph, and to eliminate fraudulent acts. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2009 / 107237 Brochure [Patent Document 2] JP 2008-158597 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, when a person with dark skin, such as a black person, is photographed with an infrared camera, the reflected infrared light cannot be detected sufficiently due to the high infrared absorption rate, and it can be difficult to determine whether the photograph is of a living body. Also, when performing face recognition from an infrared image, the accuracy of face recognition is poor compared to when performing face recognition from a visible light image.

[0006] An object of the present invention is to provide an image processing device, an image processing method, a face recognition system, and a program that can accurately distinguish between living and non-living objects from a captured image and extract a living face image suitable for face recognition. [Means for solving the problem]

[0007] According to one aspect of the present invention, an acquisition unit acquires a first image generated by photographing a subject using light of a first wavelength and a second image generated by photographing the subject using light of a second wavelength; and the second image a detection unit that detects a face in the second image based on depth information acquired from the second image; Face detected from the second image a liveness determination unit for determining whether the first image is a live body or not; Facial features of the detected face and 、 The second image a matching unit that matches a facial feature amount of a face detected from the first image and determined to be a living body with a facial feature amount of a face detected from the second image and determines whether or not the person whose face is detected from the first image is identical to any of the people whose faces are detected from the second image and determined to be a living body; and when the matching unit determines that the person whose face is detected from the first image is identical to any of the people whose faces are detected from the second image and determined to be a living body, and a determination unit that determines whether or not a person is a registered person.

[0008] According to yet another aspect of the present invention, a first image is generated by photographing a subject using light of a first wavelength, and a second image is generated by photographing the subject using light of a second wavelength, and the first image is generated by photographing the subject using light of a second wavelength. and the second image and detecting a face in the second image based on depth information obtained from the second image. Face detected from the second image determining whether the first image is a living body or not, Facial features of the detected face and 、 The second image a face feature amount of a face detected from the first image and determined to be a living body, and a face feature amount of a face detected from the first image is compared with a face feature amount of a face detected from the second image and determined to be a living body, and it is determined whether or not the person whose face is detected from the first image is identical to any of the people whose faces are detected from the second image and determined to be a living body, and when it is determined that the person whose face is detected from the first image is identical to any of the people whose faces are detected from the second image and determined to be a living body, Image processing method for determining whether a person is a registered person method is provided. Effect of the Invention

[0009] According to the present invention, it is possible to accurately distinguish between living and non-living objects from a captured image and extract a living face image suitable for face authentication. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a schematic configuration of a face authentication system according to a first embodiment of the present invention. [Diagram 2] 3 is a flowchart showing a face authentication method in the face authentication system according to the first embodiment of the present invention. [Diagram 3] FIG. 11 is a block diagram showing a schematic configuration of a face authentication system according to a second embodiment of the present invention. [Figure 4] 10 is a flowchart showing a face authentication method in a face authentication system according to a second embodiment of the present invention. [Diagram 5] FIG. 13 is a diagram showing an example of erroneous extraction of a face image when a subject including a two-dimensional face image is photographed. [Figure 6] 10 is a flowchart showing a face authentication method in a face authentication system according to a third embodiment of the present invention. [Figure 7] 1A and 1B are diagrams illustrating an example of a visible light image and a near-infrared image. [Figure 8] 8 is a diagram showing an example of a face recognition result when the face recognition process of the present embodiment is executed based on the image of FIG. 7. FIG. [Figure 9] FIG. 1 is a schematic diagram illustrating an example of a computing device. [Figure 10] FIG. 11 is a block diagram showing a functional configuration of an image processing device according to another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] [First embodiment] A face authentication system according to a first embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram showing a schematic configuration of the face authentication system according to this embodiment.

[0012] The face authentication system 100 according to this embodiment includes an image capturing unit 10, an image processing unit 20, a face authentication unit 40, a storage unit 50, and a control unit 60, as shown in FIG.

[0013] The photographing unit 10 is for acquiring a first image generated by photographing a subject using light of a first wavelength, a second image generated by photographing a subject using light of a second wavelength, and depth information (three-dimensional information) of the subject. The light of the first wavelength is, for example, visible light. The light of the second wavelength is, for example, near-infrared light. The photographing unit 10 may include, for example, a camera 12 that photographs a visible light image, a camera 14 that photographs a near-infrared image, and a projector 16 that irradiates a specific pattern of laser light on the subject. In this embodiment, an example will be described in which the visible light image is photographed by the camera 12 and the near-infrared image is photographed by the camera 14.

[0014] The method of acquiring depth information of the subject is not particularly limited. For example, in the example using the image capturing unit 10 having the above configuration, a method of capturing an image with the camera 14 while projecting a specific pattern of laser light from the projector 16 can be applied. The distance to the subject can be measured by analyzing the specific pattern of reflected light from the captured image. Alternatively, a method of measuring the distance to the subject from the parallax of images captured by multiple cameras, or a method of measuring the distance to the subject by irradiating an infrared laser to the subject and utilizing the time difference of the received reflected light (TOF method) can also be used.

[0015] Examples of commercially available products having the same functions as the imaging unit 10 having the above configuration include Kinect (registered trademark) made by Microsoft Corporation and RealSense (registered trademark) made by Intel Corporation. These products are equipped with an RGB camera as a means for capturing visible light images, and a near-infrared projector and a near-infrared camera as a means for capturing images including depth information of a subject.

[0016] The image processing unit 20 is a functional block that extracts a facial image of a living body from an image captured by the photographing unit 10, and includes an image data acquisition unit 22, a face detection unit 24, a liveness determination unit 26, and a facial image extraction unit 30.

[0017] The image data acquisition unit 22 acquires visible light image data captured by the camera 12, near-infrared image data captured by the camera 14, and depth information of the subject from the photographing unit 10. The face detection unit 24 detects faces appearing in the visible light image and near-infrared image acquired by the image data acquisition unit 22. The liveness determination unit 26 determines whether or not the face in the near-infrared image detected by the face detection unit 24 is a photograph of a living body, based on the depth information of the subject acquired by the image data acquisition unit 22. The face image extraction unit 30 extracts a face image to be used for face authentication from the visible light image, based on information about the face appearing in the near-infrared image.

[0018] The face authentication unit 40 is a functional block that determines whether or not a person in a face image extracted by the image processing unit 20 corresponds to a person registered in advance, and includes a face feature amount extraction unit 42 and a matching unit 44. Note that, although the image processing unit 20 and the face authentication unit 40 are separated here for convenience, the image processing unit 20 may have some of the functions of the face authentication unit 40.

[0019] The facial feature extraction unit 42 extracts facial feature amounts, which are parameters representing facial features, from the facial image in the visible light image extracted by the facial image extraction unit 30. The matching unit 44 performs a matching process to match the facial feature amounts of the facial image in the visible light image extracted by the facial feature amount extraction unit 42 with the facial feature amounts of a facial image of a person registered in the storage unit 50.

[0020] The control unit 60 is a control unit that controls the overall processing in the photographing unit 10, the image processing unit 20, and the face authentication unit 40.

[0021] Next, a face authentication method using the face authentication system 100 according to this embodiment will be specifically described with reference to Fig. 2. Fig. 2 is a flowchart showing the face authentication method in the face authentication system according to this embodiment.

[0022] Some commercially available imaging devices that combine a visible light camera and a near-infrared camera are equipped with a coordinate transformation function that relates the coordinates of an image captured by the visible light camera to the coordinates of an image captured by the near-infrared camera. In this embodiment, an image processing method using such a coordinate transformation function will be described. Note that a visible light camera and a near-infrared camera configured as separate devices may be used, and a coordinate transformation function that relates the coordinates of an image captured by the visible light camera to the coordinates of an image captured by the near-infrared camera may be separately prepared.

[0023] The face recognition process using the face recognition system 100 according to this embodiment is performed in accordance with steps S101 to S107 in Fig. 2. Note that the processes of each unit, which will be described later, are executed under the control of the control unit 60.

[0024] First, the photographing unit 10 photographs a visible light image of the subject with the camera 12, and photographs a near-infrared image of the subject with the camera 14. The photographing unit 10 also acquires depth information of the subject using the camera 14 and the projector 16 (step S101). It is desirable to photograph the visible light image and the infrared image synchronously. The photographed visible light image and near-infrared image are images of at least one same person as a subject. The depth information of the subject is generated using information of near-infrared light reflected by the subject.

[0025] The image data of the visible light image and the near-infrared image captured by the image capturing unit 10 are sent to the image data acquiring unit 22 of the image processing unit 20 together with the depth information of the subject. The image data acquiring unit 22 may perform additional image processing such as correction processing on the received image data as necessary.

[0026] Next, the face detection unit 24 performs face detection processing to detect a face appearing in the near-infrared image sent to the image data acquisition unit 22 (step S102). The algorithm used by the face detection unit 24 for face detection is not particularly limited, and various algorithms can be applied.

[0027] Next, liveness determination unit 26 determines whether or not the face image detected from the near-infrared image is an image acquired by camera 14 capturing an image of a living body (step S103). Specifically, liveness determination unit 26 refers to the depth information of the subject acquired by imaging unit 10 and determines whether or not the part of the face detected from the near-infrared image contains depth information corresponding to the unevenness that represents the characteristics of the living body.

[0028] For example, since a living body contains depth information that differs from place to place according to the contours of the face, if a distribution of depth information is recognized in the face, it can be determined that the detected face image is a photograph of a living body. On the other hand, since a two-dimensional image such as a photograph does not contain depth information that differs from place to place according to the contours of the face, if the depth information in the face is uniform, it can be determined that the detected face image is not a photograph of a living body.

[0029] In step S103, if the liveness determination unit 26 determines that the detected face image is an image of a living body ("Y" in the figure), the process proceeds to step S104. On the other hand, in step S103, if the liveness determination unit 26 determines that the detected face image is not an image of a living body ("N" in the figure), the face authentication is determined to have failed, and the process returns to step S101. Note that cases in which the detected face is determined to not be an image of a living body also include cases in which no face was detected from the near-infrared image in step S102.

[0030] Next, the face image extraction unit 30 uses a coordinate transformation function provided in the imaging device to transform the coordinates in the near-infrared image of the face image determined to be a photograph of a living body into coordinates in the visible light image (step S104). By using the coordinate transformation function provided in the imaging device, it is possible to easily associate the face determined to be a photograph of a living body in the near-infrared image with the face of a person captured in the visible light image.

[0031] Next, the face image extraction unit 30 extracts a face image from the visible light image based on the coordinates in the visible light image of the face determined to be a photograph of a living body (step S105). That is, the face image extraction unit 30 extracts a face image corresponding to the face in the near-infrared image from the visible light image using information about the face in the near-infrared image, which in this embodiment is information about the position of the face in the near-infrared image. At this time, it is desirable to cut out the image slightly wider, taking into consideration the conversion error from the coordinates in the near-infrared image to the coordinates in the visible light image.

[0032] In this way, only face images of living bodies can be extracted from the captured visible light images.

[0033] Next, the face authentication unit 40 executes face authentication processing using the face image extracted from the visible light image by the image processing unit 20. By performing face authentication using the visible light image, it is possible to improve the accuracy of face authentication compared to the case where a near-infrared image is used.

[0034] First, the facial feature extraction unit 42 extracts facial features, which are parameters representing facial features, from the facial image extracted in step S105 (step S106). The facial feature extraction may be performed after the face detection unit 24 performs face detection on the facial image cut out from the visible light image.

[0035] A facial feature amount is a vector amount, which is a combination of scalar amount components expressing the features of a facial image. The components of the feature amount are not particularly limited, and various types can be used. For example, the components of the feature amount can be positional relationships such as distances and angles between feature points set at the centers or end points of facial organs such as the eyes, nose, and mouth, the curvature of the facial contour, color distribution and shading values ​​of the facial surface, etc. The number of components of the feature amount is also not particularly limited, and can be appropriately set according to the required matching accuracy, processing speed, etc.

[0036] Next, the matching unit 44 performs a matching process to check whether the person in the face image extracted from the visible light image corresponds to any of the people registered in the storage unit 50 (step S107). Specifically, the matching unit 44 compares the facial feature amount of the face image extracted in step S106 with the facial feature amount of the face image of the person registered in the storage unit 50. If a person whose facial feature amount similarity exceeds a predetermined threshold exists in the storage unit 50, the matching unit 44 determines that the person in the face image extracted from the visible light image is the same as the registered person, that is, determines that the face authentication has been successful. On the other hand, if a person whose facial feature amount similarity exceeds a predetermined threshold does not exist in the storage unit 50, the matching unit 44 determines that the person in the face image extracted from the visible light image is a person not registered in the storage unit 50, that is, determines that the face authentication has failed.

[0037] After that, the control unit 60 executes a predetermined process according to the result of the matching process. For example, when the face authentication system of this embodiment is used in a gate system for immigration control or entrance / exit management, the gate system is controlled to permit entry only when face authentication is successful.

[0038] If the similarity of the facial feature amount does not meet a predetermined threshold, the facial authentication system 100 may be configured to issue an alert as fraud. Furthermore, if a face is detected in a near-infrared image but it is determined that the detected face is not a photograph of a living body, there is a possibility that fraud has been committed by attempting to pass through the gate system using a photograph or the like. In such a case, an alert may be issued as fraud. On the other hand, if the storage unit 50 stores facial feature amounts of a facial image of a person included in a blacklist, the facial authentication system 100 may be configured to issue an alert if authentication is successful.

[0039] In this way, according to the present embodiment, it is possible to accurately distinguish between living and non-living objects from a captured image and extract a living face image suitable for face recognition, thereby realizing a face recognition system capable of realizing highly accurate face recognition.

[0040] [Second embodiment] A face authentication system and a face authentication method according to a second embodiment of the present invention will be described with reference to Figures 3 to 5. The same components as those in the face authentication system according to the first embodiment are given the same reference numerals, and their description will be omitted or simplified.

[0041] First, a schematic configuration of the face authentication system according to the present embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing a schematic configuration of the face authentication system according to the present embodiment.

[0042] As shown in Fig. 3, the face authentication system 100 according to this embodiment is similar to the face authentication system according to the first embodiment shown in Fig. 1, except that the image processing unit 20 further includes a face number determination unit 28. The face number determination unit 28 determines the number of faces appearing in the visible light image and the number of faces determined to be images of living organisms among the faces appearing in the near-infrared image.

[0043] In the face recognition system of this embodiment, the association between the visible light image and the near-infrared image is performed based on the number of faces captured in the visible light image and the number of faces determined to be images of living bodies among the faces captured in the near-infrared image, rather than on the coordinate conversion function used in the first embodiment. Therefore, the face recognition system of this embodiment does not necessarily need to use an imaging device equipped with a coordinate conversion function between the visible light image and the near-infrared image.

[0044] Next, a face authentication method using the face authentication system 100 according to this embodiment will be specifically described with reference to Fig. 4 and Fig. 5. Fig. 4 is a flowchart showing the face authentication method in the face authentication system according to this embodiment. Fig. 5 is a diagram showing an example of erroneous extraction of a face image when a subject including a two-dimensional face image is photographed.

[0045] The face recognition process using the face recognition system 100 according to this embodiment is performed in accordance with steps S201 to S207 in Fig. 4. The processes of each unit, which will be described later, are executed under the control of the control unit 60.

[0046] First, the photographing unit 10 photographs a visible light image of the subject with the camera 12, and photographs a near-infrared image of the subject with the camera 14. The photographing unit 10 also acquires depth information of the subject using the camera 14 and the projector 16 (step S201). Since a coordinate conversion function is not used in this embodiment, the cameras 12 and 14 may be separate imaging devices, but it is desirable to perform calibration so that the fields of view of the cameras 12 and 14 are approximately equal. The image data of the visible light image and the near-infrared image photographed by the photographing unit 10 are sent to the image data acquisition unit 22 of the image processing unit 20 together with the depth information of the subject.

[0047] Next, the face detection unit 24 performs face detection processing for detecting faces appearing in each of the visible light image and the near-infrared image sent to the image data acquisition unit 22 (step S202).

[0048] Next, liveness determination unit 26 determines whether or not the face image detected from the near-infrared image is an image acquired by camera 14 capturing an image of a living body (step S203). When multiple faces are detected from the near-infrared image, liveness determination unit 26 determines whether or not each face image is an image acquired by capturing an image of a living body. The method of determination is the same as step S103 in the first embodiment.

[0049] Next, the face number determination unit 28 determines the number of faces detected from the visible light image and the number of faces detected from the near-infrared image that are judged to be photographs of living body faces, and judges whether the two numbers are the same (step S204).

[0050] If the face number determination unit 28 determines in step S204 that the number of faces is the same ("Y" in the figure), it determines that all of the face images detected in the visible light images are images of living bodies, and proceeds to step S205. On the other hand, if the face number determination unit 28 determines in step S204 that the number of faces is different ("N" in the figure), it determines that at least some of the faces detected in the visible light images are images of two-dimensional images such as photographs, and returns to step S201. In such a case, an alert may be issued as a case where fraudulent activity has occurred. Furthermore, it may be possible to specify the faces detected only from the visible light images. Furthermore, the face number determination unit 28 may output the face images of the faces detected only from the visible light images as faces where fraudulent activity has occurred.

[0051] Next, the face image extraction unit 31 extracts face images (face images) detected from the visible light image (step S205). This step is performed when the number of faces detected from the visible light image is the same as the number of faces detected from the near-infrared image and determined to be images of the faces of a living body. That is, in this step, face images corresponding to the faces appearing in the near-infrared image are extracted from the visible light image using information about the faces appearing in the near-infrared image, which in this embodiment is the number of faces detected from the near-infrared image and determined to be images of the faces of a living body. When extracting face images from the visible light image, the coordinate conversion function described in the first embodiment may be used to extract face images corresponding to the faces appearing in the near-infrared image from the visible light image.

[0052] Next, the facial feature amount extraction unit 42 extracts facial feature amounts, which are parameters representing facial features, from the facial image extracted from the visible light image in step S205 (step S206). The method of extracting the facial feature amounts is the same as in step S106 in the first embodiment.

[0053] Next, the matching unit 44 performs a matching process (step S207) to check whether the person in the face image extracted from the visible light image corresponds to any of the people registered in the storage unit 50. The method of the matching process is the same as that of step S107 in the first embodiment.

[0054] Thereafter, the control unit 60 executes a predetermined process depending on the result of the collation process.

[0055] As described above, in this embodiment, only when the number of faces detected from the visible light image is the same as the number of faces detected from the near-infrared image and determined to be images of living bodies, the detected face images are subject to face recognition processing. The reason for this will be described below.

[0056] If even one of the facial images detected from the near-infrared images is determined to not have been obtained by camera 14 photographing a living subject, it is possible to immediately determine that fraudulent activity has occurred using a two-dimensional image such as a photograph.

[0057] However, for example, if a poster with a person's face on it happens to be stuck on a wall within the field of view of the camera, a problem may occur, such as being unable to proceed to face recognition processing.

[0058] Furthermore, in cases where a subject exists that can be detected as a face in a visible light image but is difficult to detect as a face in a near-infrared image, for example a photograph of a fair-skinned face, a liveness check of the near-infrared image alone will not be able to eliminate the two-dimensional facial image.

[0059] For example, as shown in FIG. 5(a), in a near-infrared image of a subject including photographs of person A and person B, the face of person A is detected, but the face of person B is not detected. In this case, since no liveness check is performed on the facial image of person B, it is determined that no fraudulent activity using a two-dimensional image has been performed. Then, in the subsequent face recognition process, as shown in FIG. 5(b), the faces of person A and person B in the visible light image are determined to be processing targets, and face recognition is performed on each of them.

[0060] In this way, when there is a subject that is detected as a face in the visible light image but is difficult to detect as a face in the near-infrared image, the number of faces detected from the visible light image will differ from the number of faces detected from the near-infrared image.

[0061] Therefore, in this embodiment, in addition to the liveness check of the near-infrared image, the number of faces detected in the visible light image is compared with the number of faces detected in the near-infrared image. If a two-dimensional image in which a face can be detected from the near-infrared image exists within the field of view of the camera, this face image can be eliminated by the liveness check. If a two-dimensional image in which a face cannot be detected from the near-infrared image exists within the field of view of the camera, the presence of this face image can be known based on the difference between the number of faces detected in the visible light image and the number of faces detected in the near-infrared image. This makes it possible to appropriately select face images to be included in the target of face recognition processing from the captured images, and to execute face recognition processing.

[0062] In this way, according to the present embodiment, it is possible to accurately distinguish between living bodies and non-living bodies from a captured image and extract a living body face image suitable for face recognition without converting the coordinates of a visible light image into the coordinates of a near-infrared image, thereby realizing a face recognition system capable of realizing highly accurate face recognition.

[0063] [Third embodiment] A face recognition method according to a third embodiment of the present invention will be described with reference to Figures 6 to 8. The same components as those in the face recognition systems according to the first and second embodiments are given the same reference numerals, and their description will be omitted or simplified.

[0064] Fig. 6 is a flowchart showing a face authentication method in the face authentication system according to the present embodiment. Fig. 7 is a diagram showing an example of a visible light image and a near-infrared image captured by the image capture unit. Fig. 8 is a diagram showing an example of a face authentication result when the face authentication process of the present embodiment is executed based on the image in Fig. 7.

[0065] In this embodiment, another face authentication method using the face authentication system according to the first or second embodiment will be described. The face authentication method of this embodiment associates an image captured by a visible light camera with an image captured by a near-infrared camera by face authentication. In this embodiment, as in the second embodiment, the coordinate transformation function used in the first embodiment does not necessarily have to be used.

[0066] The face recognition process according to this embodiment is performed in accordance with steps S301 to S307 in Fig. 6. The processes of each unit, which will be described later, are executed under the control of the control unit 60.

[0067] First, the photographing unit 10 photographs a visible light image of the subject with the camera 12, and photographs a near-infrared image of the subject with the camera 14. The photographing unit 10 also acquires depth information of the subject using the camera 14 and the projector 16 (step S301). The image data of the visible light image and the near-infrared image photographed by the photographing unit 10 are sent to the image data acquisition unit 22 of the image processing unit 20 together with the depth information of the subject.

[0068] Next, the face detection unit 24 performs face detection processing for detecting faces appearing in each of the visible light image and the near-infrared image sent to the image data acquisition unit 22 (step S302).

[0069] As an example, assume that photographs of person A, person C, and person B are present in the field of view of cameras 12 and 14. Assume that the faces of person A', person B', and person C' are detected from the visible light image, for example, as shown in FIG. 7(a). Assume that the faces of person A", person B", and person C" are detected from the near-infrared image, for example, as shown in FIG. 7(b). Assume here that person A and person B are persons who have been registered in storage unit 50, and person C is a person who has not been registered in storage unit 50.

[0070] Next, the liveness determining unit 26 determines whether or not the face detected from the near-infrared image is an image of a living body (step S303). The method of determination is the same as that of step S103 in the first embodiment.

[0071] In the example of FIG. 7, the faces of person A″ and person C″ are determined to be images of living bodies, while the face of person B″ is determined to be a photograph. As a result, the face of person B″ is excluded from the group of faces detected from the near-infrared image.

[0072] Next, the facial image extraction unit 30 extracts a facial image (facial image) detected from the visible light image and a facial image detected from the near-infrared image and determined to be a photograph of a living body (hereinafter referred to as a "living body facial image") (step S304).

[0073] Next, the facial feature extraction unit 42 extracts facial feature parameters representing facial features from each of the facial image detected from the visible light image and the facial image of the living body detected from the near-infrared image (step S305). The method of extracting the facial feature parameters is the same as that of step S106 in the first embodiment.

[0074] Next, the matching unit 44 judges whether the person in the face image detected from the visible light image is the same as any of the people in the face image of the living body detected from the near-infrared image (step S306). Specifically, the matching unit 44 compares the facial feature amount of the face image detected from the visible light image with the facial feature amount of the face image of the living body detected from the near-infrared image for all combinations of the face image detected from the visible light image and the face image of the living body detected from the near-infrared image. Then, the matching unit 44 determines the face image to be extracted from the visible light image based on the matching between the facial feature amount of the face image detected from the visible light image and the facial feature amount of the face image of the living body detected from the near-infrared image. For example, the matching unit 44 judges that a combination in which the similarity of the facial feature amount exceeds a predetermined threshold is the same person.

[0075] Usually, the number of face images detected from a visible light image is the maximum face detection number. Also, the number of face images of living organisms detected from a near-infrared image is equal to or less than the maximum face detection number. If the maximum face detection number is N, then each person will be matched with a maximum of N face images. If there are multiple people whose facial feature similarities exceed a predetermined threshold, the person with the face image with the highest score can be determined to be the same person.

[0076] When the matching unit 44 determines that at least a part of the persons in the face images detected from the visible light image corresponds to the persons in the live face images detected from the near-infrared image ("Y" in the figure), the process proceeds to step S307. The face images in the visible light images for which the face images of the same persons are not found in the near-infrared images are excluded from the group of face images detected from the visible light images. In this sense, the matching unit 44 can also be said to be an extraction unit that extracts face images of persons who also appear in the near-infrared image and who are shown as live face images in the near-infrared image from among all face images of persons appearing in the visible light image.

[0077] On the other hand, if the matching unit 44 determines that none of the people in the face images detected from the visible light image correspond to the people in the biometric face images detected from the near-infrared image ("N" in the figure), it determines that face authentication has failed and returns to step S301.

[0078] In the example of FIG. 7, the facial image of person A' in the visible light image and the facial image of person A" in the near-infrared image are determined to be photographs of the same person A. Also, the facial image of person C' in the visible light image and the facial image of person C" in the near-infrared image are determined to be photographs of the same person C. On the other hand, since the facial image of person B" has been excluded from the group of facial images detected from the near-infrared image, it is determined that the person corresponding to person B' in the visible light image does not appear in the near-infrared image.

[0079] The liveness check in step S303 may be performed after step S306. In this case, however, it is necessary to perform processing such as extraction of facial features on the captured 2D facial image, so from the viewpoint of reducing the processing load, it is desirable to perform the liveness check before step S305.

[0080] Next, the matching unit 44 judges whether the person in the face image extracted from the visible light image corresponds to any of the people registered in the storage unit 50 (step S307). Specifically, for each face image extracted from the visible light image, the matching unit 44 compares the facial feature amount of the face image extracted from the visible light image with the facial feature amount of the face image of the person registered in the storage unit 50. If a face image of a person whose similarity in the facial feature amount exceeds a predetermined threshold is present in the storage unit 50, the matching unit 44 judges that the person in the face image extracted from the visible light image is the same as the registered person, that is, judges that the face authentication has been successful. On the other hand, if a face image of a person whose similarity in the facial feature amount exceeds a predetermined threshold is not present in the storage unit 50, the matching unit 44 judges that the person in the face image extracted from the visible light image is a person not registered in the storage unit 50, that is, judges that the face authentication has failed.

[0081] If the face authentication is successful in step S307 ("Y" in the figure), the control unit 60 ends the series of face authentication processes. On the other hand, if the face authentication is unsuccessful in step S307 ("N" in the figure), the control unit 60 returns to step S301.

[0082] 7, the facial image of person A' in the visible light image is determined to be the facial image of person A registered in the storage unit 50, and facial authentication is successful. On the other hand, since person C is not registered in the storage unit 50, facial authentication of the facial image of person C' in the visible light image fails.

[0083] FIG. 8 shows a summary of the determination results at each step when the face recognition process of this embodiment is executed based on the image of FIG.

[0084] In the face recognition process of this embodiment, face recognition is successful when all three conditions are met: 1) the person in the visible light image and the person in the near-infrared image are the same person, 2) the person in the near-infrared image is a photograph of a living body, and 3) the person in the visible light image is a registered person. In FIG. 8, face images where the conditions are met are connected with black bars, and face images where the conditions are not met are connected with white bars. Since person A satisfies all of the above conditions 1) to 3), face recognition is successful. Since person B satisfies the above conditions 1) and 3), but not condition 2), face recognition is unsuccessful. Since person C satisfies the above conditions 1) and 2), but not condition 3), face recognition is unsuccessful.

[0085] As described above, the face recognition method according to the present embodiment realizes matching between an image captured by a visible light camera and an image captured by a near-infrared camera by face recognition, and therefore does not require calibration between the visible light camera and the near-infrared camera. In this way, according to the present embodiment, it is possible to accurately distinguish between living and non-living objects from a captured image and extract a living face image suitable for face recognition, thereby realizing a face recognition system capable of realizing highly accurate face recognition.

[0086] [Fourth embodiment] A computer device according to a fourth embodiment of the present invention will be described with reference to Fig. 9. In this embodiment, an example of a computer device used to realize the processing of each unit in the face matching system according to the first to third embodiments will be described.

[0087] Fig. 9 shows an example of a computer device used to realize the processing of each unit in the face matching system according to the first to third embodiments. The computer device 1000 shown in Fig. 9 is not particularly limited and may be of various types and forms. For example, the computer device 1000 may be a laptop computer, a desktop computer, a workstation, a personal digital assistant, a server, a blade server, a mainframe, an embedded system, etc.

[0088] The computer device 1000 has a processor 1002, a memory 1004, and a storage device 1006. The computer device 1000 also has a high-speed controller 1008 including a high-speed interface, and a low-speed controller 1010 including a low-speed interface. The high-speed controller 1008 is connected to the memory 1004 and a high-speed expansion port 1012. The high-speed controller 1008 is also connected to external input / output devices such as a display 1016. On the other hand, the low-speed controller 1010 is connected to a low-speed expansion port 1014 and the storage device 1006.

[0089] The processor 1002, memory 1004, storage device 1006, high-speed controller 1008, low-speed controller 1010, and high-speed expansion port 1012 are interconnected by various buses, and the processor 1002, memory 1004, storage device 1006, high-speed controller 1008, low-speed controller 1010, and high-speed expansion port 1012 may be implemented on a common motherboard or in other suitable forms.

[0090] The processor 1002 may be, for example, a central processing unit (CPU) and is capable of processing instructions to be executed within the computer device 1000. Such instructions include instructions stored in the memory 1004 or in the storage device 1006 for displaying graphical information of a graphical user interface (GUI) on an external input / output device such as a display 1016.

[0091] Also, multiple processors, multiple buses, or multiple processors and multiple buses may be used, along with multiple memories and multiple types of memory, as appropriate. Also, multiple computing devices 1000 may be connected together with each device performing some portion of the required processing. For example, multiple computing devices 1000 may be connected together as a server bank, a group of blade servers, or a multi-processor system.

[0092] The memory 1004 stores information within the computer device 1000. For example, the memory 1004 may be a volatile memory unit, a non-volatile memory unit, or another computer-readable medium, such as a magnetic disk, an optical disk, etc.

[0093] The storage device 1006 may constitute mass storage for the computer device 1000. The storage device 1006 may be or include a computer-readable medium, such as, for example, a floppy disk drive, a hard disk drive, an optical disk drive, a tape drive, a flash memory or other solid-state memory device, a disk array, etc. The storage device 1006 may include a storage area network, other configurations of devices. The computer program product may be tangibly embodied in an information carrier. The computer program product may also store instructions that, when executed, perform one or more processes as described above. The information carrier may be a computer-readable medium or a machine-readable medium, such as the memory 1004, the storage device 1006, a memory on the processor 1002, or a propagated signal.

[0094] The high speed controller 1008 manages bandwidth intensive operations for the computing device 1000, while the low speed controller 1010 manages less bandwidth intensive operations. However, this allocation of functions is exemplary only and is not intended to be limiting. Also, some or all of the high speed controller 1008 may be incorporated within the processor 1002.

[0095] The high speed controller 1008 is connected to the memory 1004 and to a high speed expansion port 1012 that can accept various expansion cards. The high speed controller 1008 is also connected to a display 1016, for example via a graphics processor or accelerator.

[0096] The low-speed controller 1010 is also connected to the storage device 1006 and a low-speed expansion port 1014. The low-speed expansion port 1014 may include communication ports of various standards, such as USB (Universal Serial Bus), Bluetooth (registered trademark), and wired or wireless Ethernet (registered trademark). One or more input / output devices, such as a keyboard, a pointing device, and a scanner, may be connected to the low-speed expansion port 1014. One or more network devices, such as a switch and a router, may be connected to the low-speed expansion port 1014 via a network adapter, for example.

[0097] The computing device 1000 can be implemented in many different forms. For example, the computing device 1000 can be implemented in the form of a standard server or a group of such servers. The computing device 1000 can also be implemented as part of a rack server system. Additionally, the computing device 1000 can be implemented in the form of a personal computer, such as a laptop computer, a desktop computer, or the like.

[0098] The computer device 1000 can function as at least the image processing unit 20, the face authentication unit 40, and the control unit 60 of the face authentication system 100 according to the first to third embodiments. The processor 1002 controls the overall operation of the computer device 1000, and includes the function of the control unit 60. The processor 1002 can function as the image processing unit 20 by executing a program that realizes the functions of the image data acquisition unit 22, the face detection unit 24, the liveness determination unit 26, the number of faces determination unit 28, and the face image extraction units 30 and 31. The processor 1002 can function as the face authentication unit 40 by executing a program that realizes the functions of the face feature amount extraction unit 42 and the matching unit 44.

[0099] That is, the processor 1002 executes a program that realizes the functions of each of the image data acquisition unit 22, the face detection unit 24, the liveness determination unit 26, the number of faces determination unit 28, the face image extraction units 30 and 31, the face feature amount extraction unit 42, and the matching unit 44. This allows the processor 1002 to function as each of the image data acquisition unit 22, the face detection unit 24, the liveness determination unit 26, the number of faces determination unit 28, the face image extraction units 30 and 31, the face feature amount extraction unit 42, and the matching unit 44. In addition, the storage device 1006 of the computer device 1000 can function as the storage unit 50.

[0100] In addition, some or all of the programs to be executed by the processor 1002 of the computer device 1000 can be provided by a computer-readable recording medium on which they are recorded, such as a DVD-ROM (Digital Versatile Disc-Read Only Memory), a CD-ROM (Compact Disc-Read Only Memory), a USB memory, or other flash memory.

[0101] [Other embodiments] The face matching systems described in the first to third embodiments above also have a function as an image processing device for performing image processing to extract a face image of a living body from a visible light image and an image including depth information of a subject. The image processing device has the same function as the image processing unit 20. That is, according to another embodiment, the image processing device 200 can be configured as shown in Fig. 10. Fig. 10 is a block diagram showing the functional configuration of an image processing device according to another embodiment.

[0102] As shown in FIG. 10, the image processing device 200 includes an acquisition unit 202, a detection unit 204, a determination unit 206, and an extraction unit 208. The acquisition unit 202 acquires visible light image data and image data including depth information of a subject (for example, near-infrared image data). The detection unit 204 detects a face appearing in the image acquired by the acquisition unit 202. The determination unit 206 determines whether or not the face in the image detected by the detection unit 204 is a photograph of a living body. The extraction unit 208 extracts a face image used for face authentication from the visible light image based on information about the face appearing in the near-infrared image. The specific functions of each unit are the same as the functions of each unit of the image processing unit 20 described in the first to third embodiments. The image processing device 200 may further include a function as a determination unit similar to the face number determination unit 28 of the image processing unit 20 of the second embodiment.

[0103] Also included in the scope of each embodiment is a processing method in which a program that operates the configuration of the embodiment to realize the functions of the above-mentioned embodiment is recorded on a recording medium, the program recorded on the recording medium is read as code, and executed on a computer. That is, a computer-readable recording medium is also included in the scope of each embodiment. Also included in each embodiment is not only the recording medium on which the above-mentioned program is recorded, but also the program itself.

[0104] Examples of the recording medium that can be used include a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a non-volatile memory card, and a ROM. In addition, the scope of each embodiment is not limited to a program recorded on the recording medium that executes processing by itself, but also includes a program that executes processing by operating on an OS in cooperation with other software and functions of an extension board.

[0105] Furthermore, an example in which part of the configuration of any of the embodiments is added to another embodiment, or an example in which part of the configuration of another embodiment is substituted therefor, is also an embodiment of the present invention.

[0106] It should be noted that the above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0107] A part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0108] (Appendix 1) an acquisition unit that acquires a first image generated by photographing a subject using light of a first wavelength, a second image generated by photographing the subject using light of a second wavelength, and depth information of the subject; A detection unit that detects a face appearing in the second image; a determination unit that determines whether or not the face detected by the detection unit is an image of a living body based on the depth information; an extraction unit that extracts a face image from the first image based on face information that the determination unit has determined to be an image of a living body; 13. An image processing device comprising:

[0109] (Appendix 2) The extraction unit extracts, from the first image, a face image corresponding to the face appearing in the second image, using information about a position in the second image of the face that the determination unit has determined to be an image of a living body. 2. The image processing device according to claim 1 .

[0110] (Appendix 3) A determination unit for determining the number of faces appearing in the image, The detection unit further detects a face appearing in the first image, the determination unit determines the number of faces appearing in the first image and the number of faces appearing in the second image based on a detection result of the detection unit; The extraction unit extracts face images from the first image when the number of faces appearing in the first image and the number of faces appearing in the second image match. 3. The image processing device according to claim 1 or 2.

[0111] (Appendix 4) The depth information of the object is information generated using the light of the second wavelength reflected by the object. 4. The image processing device according to claim 1,

[0112] (Appendix 5) The detection unit further detects a face appearing in the first image, The extraction unit determines a face image to be extracted from the first image based on a comparison between a facial feature amount of the face captured in the second image and a facial feature amount of the face captured in the first image. 3. The image processing device according to claim 1 or 2.

[0113] (Appendix 6) the light of the first wavelength is visible light; The second wavelength of light is near infrared. 6. The image processing device according to claim 1,

[0114] (Appendix 7) an image processing unit that extracts a face image from a first image generated by photographing a subject using light of a first wavelength; a face authentication unit that determines whether the person in the face image extracted by the image processing unit is the same as a registered person, The image processing unit includes: an acquisition unit that acquires the first image, a second image generated by capturing an image of the subject using light of a second wavelength, and depth information of the subject; A detection unit that detects a face appearing in the second image; a determination unit that determines whether or not the face detected by the detection unit is an image of a living body based on the depth information; and an extraction unit that extracts the face image from the first image based on information about the face that the determination unit has determined to be an image of a living body. A face recognition system characterized by:

[0115] (Appendix 8) The extraction unit extracts, from the first image, a face image corresponding to the face appearing in the second image, using information about a position in the second image of the face that the determination unit has determined to be an image of a living body. 8. The face recognition system according to claim 7.

[0116] (Appendix 9) the image processing unit further includes a determination unit that determines the number of faces appearing in the image; The detection unit further detects a face appearing in the first image, the determination unit determines the number of faces appearing in the first image and the number of faces appearing in the second image based on a detection result of the detection unit; The extraction unit extracts face images from the first image when the number of faces appearing in the first image and the number of faces appearing in the second image match. 9. The face recognition system according to claim 7 or 8.

[0117] (Appendix 10) The depth information of the object is information generated using the light of the second wavelength reflected by the object. 10. The face authentication system according to claim 7, wherein:

[0118] (Appendix 11) The method further includes a notification unit that issues an alert when the number of faces appearing in the first image and the number of faces appearing in the second image are different, indicating that the image is fraudulent. 11. The face authentication system according to claim 9 or 10.

[0119] (Appendix 12) The detection unit further detects a face appearing in the first image, The extraction unit determines a face image to be extracted from the first image based on a comparison between a facial feature amount of the face captured in the second image and a facial feature amount of the face captured in the first image. 9. The face recognition system according to claim 7 or 8.

[0120] (Appendix 13) The face authentication unit determines that authentication has been successful when the person in the face image extracted by the extraction unit is the same as a registered person and the person whose face appears in the first image and the person whose face appears in the second image are the same person. 13. The face authentication system according to claim 7,

[0121] (Appendix 14) the light of the first wavelength is visible light; The second wavelength of light is near infrared. 14. The face authentication system according to claim 7,

[0122] (Appendix 15) Acquiring a first image generated by photographing a subject using light of a first wavelength, a second image generated by photographing the subject using light of a second wavelength, and depth information of the subject; Detecting a face in the acquired second image; determining whether the detected face represents a living subject based on the depth information; A face image is extracted from the first image based on face information determined to be a photograph of a living body. 13. An image processing method comprising:

[0123] (Appendix 16) Using information about the position in the second image of a face determined to be an image of a living body, a face image corresponding to the face appearing in the second image is extracted from the first image. 16. The image processing method according to claim 15,

[0124] (Appendix 17) Further detecting a face in the acquired first image; determining the number of faces appearing in the first image and the number of faces appearing in the second image based on a result of the face detection; extracting face images from the first image when the number of faces appearing in the first image and the number of faces appearing in the second image match; 17. The image processing method according to claim 15 or 16.

[0125] (Appendix 18) The depth information of the object is information generated using the light of the second wavelength reflected by the object. 18. The image processing method according to any one of claims 15 to 17.

[0126] (Appendix 19) Further detecting a face in the acquired first image; A face image to be extracted from the first image is determined based on a similarity between a facial feature amount of the face captured in the second image and a facial feature amount of the face captured in the first image. 17. The image processing method according to claim 15 or 16.

[0127] (Appendix 20) the light of the first wavelength is visible light; The second wavelength of light is near infrared. 20. The image processing method according to any one of claims 15 to 19.

[0128] (Appendix 21) A computer device includes: acquiring a first image generated by photographing a subject using light of a first wavelength, a second image generated by photographing the subject using light of a second wavelength, and depth information of the subject; detecting a face in the acquired second image; determining whether the detected face is a living body based on the depth information; extracting a face image from the first image based on face information determined to be a living body image; A program characterized by executing the above.

[0129] (Appendix 22) A computer-readable recording medium having the program described in Appendix 21 recorded thereon. [Explanation of symbols]

[0130] 10. Photography Department 20...Image processing unit 22...Image data acquisition section 24…Face detection section 26…Liveness assessment section 28…Number of faces determination section 30, 31...Facial image extraction section 40…Facial recognition section 42…Facial feature extraction unit 44…Collation section 50...Storage section 60...Control unit 100…Facial recognition system

Claims

1. an acquisition unit that acquires a first image generated by photographing a subject using light of a first wavelength and a second image generated by photographing the subject using light of a second wavelength; a detection unit that detects a face appearing in the first image and the second image; a liveness determination unit that determines whether or not a face detected from the second image is a live body based on depth information acquired from the second image; a matching unit that matches a facial feature amount of a face detected from the first image with a facial feature amount of a face detected from the second image and determined to be a living body, and determines whether or not a person whose face is detected from the first image is identical to any of the people whose faces are detected from the second image and determined to be a living body; a determination unit that determines whether or not the person whose face is detected from the first image is a registered person when the matching unit determines that the person whose face is detected from the first image is identical to any of the people whose faces are detected from the second image and determined to be living bodies; and 13. An image processing device comprising:

2. the light of the first wavelength is visible light; The second wavelength of light is near infrared.

2. The image processing device according to claim 1,

3. Obtaining a first image generated by photographing a subject using light of a first wavelength and a second image generated by photographing the subject using light of a second wavelength; Detecting a face appearing in the first image and the second image; determining whether the face detected from the second image is a living body based on depth information acquired from the second image; comparing a facial feature amount of the face detected from the first image with a facial feature amount of the face detected from the second image and determined to be a living body, and determining whether or not the person whose face was detected from the first image is identical to any of the people whose faces were detected from the second image and determined to be a living body; When it is determined that the person whose face is detected from the first image is identical to any of the people whose faces are detected from the second image and determined to be living bodies, it is determined whether or not the person whose face is detected from the first image is a registered person.

13. An image processing method comprising:

4. A computer device includes: obtaining a first image generated by photographing an object using light of a first wavelength and a second image generated by photographing the object using light of a second wavelength; detecting a face appearing in the first image and the second image; determining whether a face detected from the second image is a living body based on depth information acquired from the second image; a step of comparing a facial feature amount of the face detected from the first image with a facial feature amount of the face detected from the second image and determined to be a living body, and determining whether or not the person whose face was detected from the first image is identical to any of the people whose faces were detected from the second image and determined to be a living body; a step of determining whether or not the person whose face was detected from the first image is a registered person when it is determined that the person whose face was detected from the first image is the same as any of the people whose faces were detected from the second image and determined to be a living body; A program characterized by executing the above.

Images