Image processing device, image processing method, program

Abstract

The present invention provides an imaging processing device, imaging processing method, and program that reduce unnecessary reshoots and misauthentication when the person to be authenticated is unlikely to be a pre-registered individual. [Solution] The image processing device 100, which is an information processing device, includes a face image storage unit 106 in which a person's face image 107 is registered, an authentication unit 109 that authenticates whether the person in the input face image is the same person as the person in the face image storage unit 106, a matching unit 102 that calculates the degree of match between the input face image and the registered face image, a determination unit 103 that uses the authentication result from the authentication unit and the degree of match to determine whether the person in the input face image is likely to be the same person as the person in the registered face image, and an output unit 104 that outputs information corresponding to the determination result output by the determination unit 103 to a display device 101 located outside the image processing device.

Description

Technical Field

[0001] The present invention relates to an imaging processing device, an imaging processing method, and a program that perform reshooting when the authentication result of a face authentication device is unauthenticated.

Background Art

[0002] In a face authentication system using images, the face image of an authentication target person is compared with a previously registered face image to determine whether the face image of the authentication target person matches any of the previously registered face images. However, due to the quality of the face image of the authentication target person, there are cases where the authentication target person is unauthenticated even though the face image is registered. When the face authentication result is unauthenticated, Patent Document 1 discloses a method for improving the quality of face images by presenting guidance on the actions to be taken by the authentication target person to improve the shooting method.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

[0005] In the method described in Patent Document 1, guidance for reshooting is provided assuming that the person to be authenticated is a pre-registered individual. As a result, unnecessary reshoots or misauthentication may occur if the person to be authenticated is not a pre-registered individual. In view of the above problems, the present invention aims to reduce the occurrence of reshoots or misauthentication when the authentication result is unauthenticated. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device comprising: a face image storage unit in which a person's face image is registered; an authentication unit that authenticates whether the person in the input face image is the same person as the person in the face image stored in the face image storage unit; a matching unit that calculates the degree of match between the input face image and the registered face image; a determination unit that uses the authentication result from the authentication unit and the degree of match to determine whether the person in the input face image is likely to be the same person as the person in the registered face image; and an output unit that outputs information corresponding to the determination result output by the determination unit. [Effects of the Invention]

[0007] According to the present invention, it is possible to reduce the need for reshoots when the authentication result is unauthenticated, and to reduce the occurrence of misauthentication. [Brief explanation of the drawing]

[0008] [Figure 1] This is an example of the configuration of an image processing apparatus according to the first embodiment. [Figure 2] This flowchart shows an example of the procedure for determining the likelihood of the person being identified according to the first embodiment. [Figure 3] This is an example of an image taken using procedure S200. [Figure 4] This is an example of an image created in procedure S201. [Figure 5] This is an example of the information presented in procedure S206. [Figure 6] This is an example of a procedure for determining the threshold used in procedure S201. [Figure 7] This is a scatter plot showing the frequency distribution of pairs of facial images of the same person. [Figure 8] This is an example of the procedure for determining the threshold used in procedure S204. [Figure 9] This is an example of a scatter plot of pairs of facial images. [Figure 10] This is a schematic diagram illustrating how a facial image is divided into rectangular regions. [Figure 11] This is a schematic diagram that investigates the number of pairs in which the corresponding rectangular regions of two facial images both contain faces. [Figure 12] This is a schematic diagram showing the result of extracting pixels containing faces from a facial image. [Figure 13] This is a schematic diagram showing the result of overlaying two face regions onto a single image region. [Figure 14] This is a schematic diagram showing the results of detecting the location of facial organs from a facial image. [Figure 15] This is a schematic diagram showing the result of correlating the positions of facial organs detected from two facial images. [Figure 16] This figure shows an example of the target position of facial organs. [Figure 17] It is a diagram showing an example of an image created by normalizing a face image. [Figure 18] It is a schematic diagram of the result of drawing face organ positions on two face images. [Figure 19] It is a schematic diagram showing the distance between the centers of the right eyes on two face images. [Figure 20] It is a flowchart showing an example of the detailed procedure of step S204. [Figure 21] It is an example of a scatter diagram of a pair of face images. [Figure 22] It is a diagram in which a class division line is drawn on the scatter diagram of FIG. 21. [Figure 23] It is a diagram in which a region with a face similarity below a certain value in the scatter diagram of FIG. 21 is divided into a plurality of regions. [Figure 24] It is a flowchart showing an example of a procedure for calculating a matching degree. [Figure 25] It is an example of a method of displaying a matching result without displaying a pre-registered face image. [Figure 26] It is an example of a method of displaying a matching result with a pre-registered face image blurred. [Figure 27] It is an example of a method of displaying a matching result using an illustration based on a pre-registered face image.

BEST MODE FOR CARRYING OUT THE INVENTION

[0009] Hereinafter, based on the preferred embodiments of the present invention, an explanation will be given with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

[0010] (First Embodiment) In this embodiment, an information processing apparatus having an authentication function and a function of determining whether there is a possibility that the authentication target person is a person pre-registered (hereinafter also referred to as the possibility of being the person himself / herself) and displaying the determination result will be described. The information processing apparatus according to this embodiment will be described using FIGS. 1 to 9.

[0011] Figure 1 shows an example of the configuration of the information processing device in this embodiment. The image processing device 100 is an example of an information processing device and includes a matching unit 102, a determination unit 103, an output unit 104, a receiving unit 105, an authentication unit 109, a face image storage unit 106, and a statistical value storage unit 110. Furthermore, each unit is connected to a network via a bus, and can send and receive data from each other.

[0012] The receiving unit 105 receives a face image 107 input from outside the image processing device 100. The received face image 107 is stored in the face image storage unit 106.

[0013] The facial image storage unit 106 has registered facial images stored in it beforehand.

[0014] The authentication unit 109 uses the registered face image stored in the face image storage unit 106 and the face image 107 to create an authentication result between the person whose registered face image is registered and the person depicted in the face image 107.

[0015] The matching unit 102 performs matching processing between each of the registered face images pre-stored in the face image storage unit 106 and the input face image 107. The matching unit 102 calculates the degree of match between each of the registered face images and face image 107 through the matching processing and transmits the calculated degree of match to the determination unit 103. Here, the degree of match is a value that quantifies the level of spatial correspondence between the two faces by calculating the correspondence relationship and consistency between the regions and organ points of the two face images.

[0016] The statistical value storage unit 110 has the statistical values ​​described later already stored in it.

[0017] The determination unit 103 determines whether or not an image of the same person as the person in the face image 107 exists among the registered face images, based on the statistical values ​​stored in the statistical value storage unit 110, the authentication result received from the authentication unit 109, and the match degree received from the matching unit 102. The determination unit 103 transmits the determination result to the output unit 104.

[0018] The output unit 104 outputs the determination result received from the determination unit 103 to the outside of the image processing device 100. The determination result 108 output to the outside is displayed, for example, on a display device 101 located outside the image processing device 100.

[0019] Figures 2 to 5 illustrate an example of the procedure for creating a determination result 108 using an image processing system including an image processing device 100.

[0020] In step S200, an image of the person to be authenticated is created. That is, the person to be authenticated is photographed with a camera. An example of the created image is shown in Figure 3. The person to be authenticated is photographed in such a way that the image includes the entire face area of ​​the person to be authenticated.

[0021] In step S201, facial recognition processing is performed on the person to be authenticated using the image of the person to be authenticated created in step S200 and a pre-stored registered facial image. In this embodiment, a method of authentication using a partial image, which is a rectangular region that roughly circumsects the face area in the image, is used. First, from the image obtained in step S200, a rectangular region containing the face is extracted by image processing using a face detector to create a facial image of the person to be authenticated. An example of the created facial image is shown in Figure 4. The created image is input to the image processing device 100 as facial image 107. Next, the similarity is calculated by comparing facial image 107 with each of the multiple registered facial images stored in the image processing device 100, and the maximum similarity is found. Here, each of the multiple registered facial images stored in the image processing device 100 is a partial image that includes a rectangular region that roughly circumsects the face area. If the maximum similarity between the input facial image and the registered facial image is greater than or equal to a predetermined threshold, it is determined that the person in the registered facial image with the highest similarity matches the person to be authenticated, and the authentication process is determined to be successful. On the other hand, if the maximum similarity score obtained in this process is less than a predetermined threshold, the authentication process is determined to have failed. In this embodiment, the method described in Non-Patent Document 1 is used as the method for calculating the similarity score, but the specific method is not limited to this, and other methods may be used. The method for determining the threshold used in this procedure will be described in detail separately.

[0022] In step S202, the next step is determined based on whether the facial recognition process performed in step S201 was successful or not. If the facial recognition process is successful, the process is terminated. If the facial recognition process fails, the process proceeds to step S203.

[0023] In step S203, the match score is calculated using the face image of the person to be authenticated and the registered face image that shows the maximum similarity to the said face image. In order to calculate the match score, this embodiment uses a dense matching method that densely estimates corresponding points between images. Specifically, pairs of corresponding points between images are detected using the method described in Non-Patent Literature 2, and the number of detected pairs is taken as the match score. The matching method is not limited to this, and other methods that estimate points or regions that match between two objects depicted in an image may be used. A person's face is composed of several characteristic parts such as the eyes, nose, mouth, and cheeks. Therefore, even if the way the face is depicted in the two images being compared is different, there are points or regions that match between the two face images. The match score may also be calculated using the number or proportion of these matching points or regions.

[0024] In step S204, the likelihood that the person to be authenticated is a person who has been previously registered, i.e., a person for whom a registered face image exists, is determined based on the relationship between the match score calculated in step S203 and a predetermined threshold. If the match score calculated in step S203 is less than the threshold, it is determined that the person to be authenticated is likely to be a person who has been previously registered. This is because even if the person to be authenticated is a person who has been previously registered, authentication may fail if, for example, the authentication process was performed using a frontal image and a side profile image. On the other hand, if the maximum value of the match score calculated in step S203 is greater than or equal to the threshold, it is determined that the person to be authenticated is not likely to be a registered person.

[0025] In step S205, the next step is determined based on the result of the determination made in step S204. If there is a possibility that the person to be authenticated is a registered person, proceed to step S206. If there is no possibility that the person to be authenticated is a registered person, terminate the process.

[0026] In step S206, information is presented to the person to be authenticated for re-capture of their image. Using the dense matching method employed in step S203, the matching results, which closely estimate corresponding points between images, are used to create the information presented to the person to be authenticated. An example of the information presented is shown in Figure 5. In Figure 5, image 500 is the face image of the person to be authenticated created in step S201, and image 501 is the registered face image that had the highest similarity to image 500 in step S201. Point clouds 502 and 503 are sets of points that constitute pairs of corresponding points detected from images 500 and 501. From the information shown in Figure 5, the person to be authenticated can learn how to re-capture their image in order to successfully authenticate. In order to avoid being incorrectly judged as authentication failure, it is necessary to increase the number of pairs of corresponding points between face images to improve the matching degree. For example, when presented with the information in Figure 5, it can be seen that the area around the mouth and the area around the bangs are not matching. This is thought to be because there is a large difference between the pre-registered image and the currently captured image in the areas where the matching is not possible. Therefore, it is clear that when reshooting, the certifier should make adjustments to ensure better matching around the mouth and bangs. For example, the certifier could lift their bangs or close their mouth.

[0027] In step S207, the next step is determined based on whether or not to re-photograph the person to be authenticated. If re-photographing of the person to be authenticated is required, proceed to step S200. If re-photographing of the person to be authenticated is not required, terminate the process.

[0028] In step S203, the face image of the person to be authenticated generated in step S201 is compared with the registered face image that had the highest similarity in step S201 to calculate the match score. However, the calculation of the match score is not limited to this. Multiple registered face images may be selected, and the match score between each registered face image and the face image of the person to be authenticated may be calculated, and the maximum value of the calculated match score may be used. Alternatively, multiple images of the same person taken from various angles may be registered in advance, and all of the multiple images of the person corresponding to the registered face image that had the highest similarity in step S201 may be used as the target for calculating the match score, and the maximum value of the match score may be calculated.

[0029] The method for determining the threshold for the face recognition process used in procedure S201 is shown with reference to Figures 6 and 7. Figure 6 is an example of the threshold determination procedure, and Figure 7 is a graph showing the frequency distribution of face similarity and face image pairs.

[0030] In step S600, a set of facial images is prepared. For the sake of explanation, we will prepare 10 facial images for each of 100 people.

[0031] In step S601, facial similarity is calculated from the set of facial images prepared in step S600. Facial similarity is calculated for each pair of facial images of the same person that can be created from the set of images prepared in S600. Since there are 10 facial images for each of 100 people, there are 4500 pairs of facial images of the same person. As an example of a method for calculating the similarity of two faces, the method described in Non-Patent Document 1, which was also used in step S201, is used. The calculated facial similarity is stored in the statistical value storage unit 110 in Figure 1.

[0032] In step S602, a threshold for facial similarity for identity determination is set. Based on the distribution of facial similarity calculated in step S601, a threshold α for facial similarity is determined to determine that the people in the two images are the same person. In this embodiment, the threshold is set such that the probability of incorrectly determining a pair of facial images that are actually of the same person as a pair of facial images of different people (hereinafter referred to as the unrecognized rate) is 0.1, but the threshold may be determined by other criteria or methods.

[0033] Using Figure 7, we will detail an example of procedure S602. First, as shown in Figure 7, we create a frequency distribution where the horizontal axis is face similarity and the vertical axis is the frequency of pairs of face images. 700 is the frequency distribution of pairs of face images of the same person, and contains 4500 pairs. Next, we calculate the product M of the total number of pairs included in frequency distribution 700 and the unauthenticated rate, that is, the number of pairs that can be left unauthenticated. In this example, M is 4500 × 0.1 = 450. Finally, we calculate a face similarity value α such that the frequency of pairs with a face similarity less than α is M, and α is set as the face similarity threshold. 702 is a subset of frequency distribution 700 and contains M = 450 pairs of face images.

[0034] The method for determining the match threshold used in procedure S204 is shown using Figures 8 and 9. Figure 8 is an example of the threshold determination procedure, and Figure 9 is an example of a scatter plot of match and face similarity.

[0035] In step S800, a group of face images is prepared. From the pairs of face images of the same person prepared in step S600, pairs whose similarity calculated in step S601 is less than the threshold α are selected and used as the group of face images for this procedure.

[0036] In step S801, the match degree is calculated for each pair of face images prepared in step S800. Using the method described in Non-Patent Literature 2, corresponding pairs of points are detected between the two images constituting the pair, and the number of point pairs is taken as the match degree. The calculated match degree is stored in the statistical value storage unit 110.

[0037] In step S802, a threshold for the match score is determined based on the match score distribution calculated in step S801, in order to determine whether the person in the two images is likely to be the same person. In this embodiment, the maximum match score of pairs of face images of the same person is set as the threshold, but the threshold may be determined by other criteria or methods.

[0038] Figure 9 is an example of a scatter plot of face image pairs with the horizontal axis representing the match degree and the vertical axis representing the face similarity. In Figure 9, 900 is the threshold α set in step S602, and 901 is the set of points plotted on the scatter plot for the pair of data for which the match degree was calculated in step S801. Point 902 is the point with the highest match degree among the points included in 901, and 903 represents the match degree β of point 902. In this example, 901 originally contains 450 points, but is simplified to show 17 points. In this embodiment, the maximum value of the match degree is set as the threshold. Therefore, in the example shown in Figure 9, the match degree β is set as the threshold for determining whether or not the person is likely to be the same person.

[0039] In image-based facial recognition systems, guidance for re-taking a photo may be provided to the person being authenticated if authentication fails. This is because one reason for failure to authenticate is problems with the quality of the person's facial image. In this case, guidance for re-taking is issued assuming that the person being authenticated is a pre-registered individual but has failed to authenticate. Therefore, even if the person being authenticated is not actually registered, re-taking may be performed, resulting in unnecessary retakes. Furthermore, performing authentication processing with a re-taken facial image may lead to false authentication. To address this, the above process is used to limit re-taking to cases where there is a high probability that the person being authenticated is a registered individual. This reduces the risk of unnecessary re-taking and unnecessary false authentication.

[0040] (Second embodiment) In the first embodiment, a dense matching method is used to calculate the degree of match between an input face image and a pre-registered face image, by densely estimating corresponding points between the images. However, depending on the quality and characteristics of the images, it may be difficult to reliably detect corresponding points.

[0041] Furthermore, in the first embodiment, a method was described in which, in order to determine whether or not there is a possibility of the person being the same, the degree of match between images was checked to see if it was below a threshold, and only if it was below the threshold was it determined that there was a possibility of the person being the same. In this case, the degree of match between a large number of pre-prepared pairs of images of the same person was calculated, and the maximum value of the match degree was set as the threshold. However, depending on the quantity and quality of the registered images, it may be difficult to determine with high accuracy whether the person in the input face image is a person who has been pre-registered.

[0042] Therefore, in this embodiment, a method different from the dense matching method is used to calculate the degree of match. In addition to image pairs of the same person, the degree of match of image pairs of different people and the similarity when the image pairs are authenticated are also considered to determine whether or not the person to be authenticated can be registered. The configuration of the image processing apparatus in this embodiment is the same as in the first embodiment, using the configuration shown in Figure 1. The processing procedure is also the same as in the first embodiment, using the procedure shown in Figure 2, and explanations common to the first embodiment are omitted. The difference between the first embodiment and this embodiment lies in the processing content in steps S203 and S204.

[0043] As an example of the method for calculating the match score in this embodiment, the input face image of the person to be authenticated and the registered face image are each divided into multiple rectangular regions, and the region containing the face is extracted from the rectangular regions to calculate the match score. In other words, the match score is calculated using a part of the face image. Specifically, the method described in Non-Patent Document 3 is used, but other methods may also be used. This will be explained with reference to Figures 10 and 11.

[0044] First, the input face image and the pre-registered face images are resized to a predetermined size, thus making their sizes consistent.

[0045] Next, each image is divided into a specific number of rectangular regions. In this embodiment, for illustrative purposes, each image is divided into 5x5 rectangular regions. In Figure 10, Image 1000 shows a face image input from an external source divided into 5x5 rectangular regions, and Image 1000 shows a pre-registered face image divided into 5x5 rectangular regions.

[0046] Next, we determine whether each of the divided 5x5 rectangular regions contains a face.

[0047] Finally, count the number of pairs of rectangular regions in corresponding positions between the two images that both contain faces, and use this number of pairs as the match score. Figure 11 shows the case where there were 17 pairs of rectangular regions in corresponding positions between image 1100 and image 1101 that both contained faces. In Figure 11, the rectangular regions marked with a "check" are those determined to be rectangular regions that constitute a pair of rectangular regions in corresponding positions between the two images that both contain faces. Here, we counted the number of pairs of rectangular regions in corresponding positions between the two images that both contain faces, but it is also possible to count the number of pairs of rectangular regions that do not contain faces and use that as the match score.

[0048] As described above, the image can be divided into rectangular regions, and the match score can be calculated using the results of extracting the region containing the face from the rectangular regions. This method has the advantage of allowing the person being authenticated to know how to adjust the position and orientation of their face relative to the camera and retake the photo in order to succeed in authentication. In order to prevent authentication from being incorrectly judged as a failure even though the face image has been registered in advance, it is necessary to increase the number of pairs of rectangular regions and improve the match score. For example, when presented with the information in Figure 11, it can be seen that there are few pairs of rectangular regions mainly at both ends of the image. This is thought to be because the face of the person being authenticated was at an angle to the camera when the image was taken. Therefore, the authenticater should take measures such as facing directly towards the camera when taking the photo.

[0049] Another method for calculating the match score is described, which uses the results of extracting pixels containing faces. In this embodiment, the method described in Non-Patent Document 4 is used, but other methods may be used. This will be explained with reference to Figures 12 and 13.

[0050] First, the input face image and the pre-registered face images are resized to a predetermined size, thus making their sizes consistent.

[0051] Next, each image is processed using the method described in Non-Patent Document 4 to calculate the face region within the image. In Figure 12, image 1200 shows a face image input from an external source, and image 1201 shows a pre-registered face image. Region 1202 is the face region detected from image 1200, and region 1203 is the face region detected from image 1201.

[0052] Finally, the face regions of each image are superimposed into a single image, and the number of pixels contained within the common region is defined as the match degree. Image 1300 in Figure 13 is drawn by superimposing regions 1202 and 1203 onto a region of the same size as images 1200 and 1201. Similarly, image 1301 is drawn by superimposing the common region 1302 of regions 1202 and 1203 onto a region of the same size as images 1200 and 1201. In this example, the match degree is the number of pixels contained in the common region 1302. This method has the advantage, similar to the method using the rectangular region described above, that it allows us to know how the person being authenticated should adjust their face position and orientation relative to the camera and retake the image in order to succeed in authentication. Furthermore, since the image is divided into finer segments than the rectangular region matching method described above to calculate the match degree, the threshold for determining whether or not the person is likely to be the real person can be set more precisely. In other words, it can effectively reduce the occurrence of retakes and false authentications even for authentication targets that show a high probability of being the real person near the threshold for whether or not they are likely to be the real person.

[0053] Another method for calculating the degree of match is to use the results of detecting the positions of facial organs, such as the center of the eyes and the corners of the mouth. In this embodiment, the method for calculating the degree of match based on the difference in the three-dimensional distribution of facial organ positions described in Non-Patent Document 5 is used, but other methods may also be used. This will be explained with reference to Figures 14 and 15.

[0054] First, the positions of facial organs are detected from both the input facial image and the pre-registered facial images. This yields a point cloud detected from the input facial image (hereinafter referred to as point cloud 1) and a point cloud detected from the pre-registered facial images (hereinafter referred to as point cloud 2).

[0055] In Figure 14, image 1400 shows a face image input from an external source, and image 1401 shows a pre-registered face image. 1402 is point cloud 1 detected from image 1400, and 1403 is point cloud 2 detected from image 1401. In this embodiment, five points are used as facial organ locations: the center of both eyes, the tip of the nose, and both ends of the mouth. Therefore, 1402 and 1403 each contain five points. Each of the five points has a two-dimensional coordinate on the image and a three-dimensional coordinate in space. It also contains information about what type of organ point each point represents.

[0056] Next, using the 3D coordinates of point cloud 1 and point cloud 2, a homogeneous transformation matrix (hereinafter referred to as R12) is calculated to transform point cloud 1 into point cloud 2. Then, the sum of the distances (hereinafter referred to as RS12) between points in point cloud 2 and the points in the transformed point cloud 1 that correspond to the points in point cloud 2 is calculated. Figure 15 shows the results of matching points in image 1400 and points in image 1401 with points of the same type. Based on the point correspondence results shown in Figure 15, R12 is calculated, and after transforming point cloud 1 with R12, RS12 is calculated. Finally, the reciprocal of RS12 is calculated and used as the match score.

[0057] As described above, the degree of match can be calculated based on the difference in the three-dimensional distribution of facial feature positions. This method has the advantage of allowing the user to know how to adjust the facial feature positions and retake the image in order to successfully authenticate the user. In order to prevent authentication from being incorrectly judged as a failure despite the registration of a facial image, it is necessary to adjust the relative positions of the facial features in three-dimensional space. For example, when presented with the information in Figure 15, it can be seen that the positions of the corners of the mouth are mainly different from those in the registered image. This is thought to be because the user's mouth was open when the image was taken. Therefore, the authenticater should take measures such as closing the mouth when taking the image.

[0058] As an alternative method for calculating the degree of match using facial organ positions, we will explain a method that calculates the degree of match based on the difference in the two-dimensional distribution of facial organ positions.

[0059] First, point clouds 1 and 2 are obtained using a procedure similar to the one used to calculate the degree of match based on the difference in the three-dimensional distribution of facial organ positions.

[0060] Next, using the two-dimensional coordinates of point cloud 1 and point cloud 2, an affine transformation matrix is ​​calculated to transform point cloud 1 into point cloud 2, and the sum of the distances between corresponding points after the transformation (hereinafter referred to as AS12) is calculated. Finally, the reciprocal of AS12 is calculated and used as the match score. Calculating the match score from the difference in the two-dimensional distribution of facial organ positions has the advantage, similar to calculating it from the difference in the three-dimensional distribution of facial organ positions, of allowing us to know how the person being authenticated should adjust their facial organ positions and retake the image in order to successfully authenticate. Furthermore, since the calculation of the affine transformation matrix is ​​performed using two-dimensional data, it can be calculated faster than the homogeneous transformation matrix calculated using three-dimensional data.

[0061] As yet another method for calculating the degree of match using facial organ positions, we will explain, using Figures 16 to 19, a method in which facial images are normalized and the degree of match is calculated based on the amount of positional shift of the corresponding organ positions.

[0062] First, point clouds 1 and 2 are obtained using a procedure similar to the one used to calculate the degree of match based on the difference in the three-dimensional distribution of facial organ positions.

[0063] Next, normalized face images are created from both the externally input face image and the pre-registered face image. Here, a normalized image is an image created by pre-setting the image size and the target positions of facial features on the image, and then performing an affine transformation so that the positions of the facial features after the transformation are as close as possible to the target positions. In Figure 16, 1600 is an image of the pre-set size, and 1601 shows the set of target positions of facial features set within image 1600. In Figure 17, image 1700 is a normalized face image from an externally input face image, and image 1701 is a normalized face image from a pre-registered face image.

[0064] For two normalized images created through normalization, the sum of the distances between corresponding points (hereinafter referred to as NS12) is calculated. In Figure 18, points 1702, 1703, 1704, 1705, and 1706 indicate the organ locations on image 1700. Points 1707, 1708, 1709, 1710, and 1711 indicate the organ locations on image 1701. Figure 19 shows the result of displaying point 1702, the center of the right eye on image 1700, and point 1707, the center of the right eye on image 1701, on the same image. Line segment 1900 connects the corresponding points 1702 and 1707, and the length of line segment 1900 is the distance between points 1702 and 1707. Similarly, the distances between other corresponding points are calculated, and the sum of these distances is taken as NS12. Finally, the reciprocal of NS12 is calculated and used as the match score. This method has the advantage that the person being authenticated can intuitively know whether or not facial features have been properly detected from the externally input facial image by comparing the two normalized images. For example, when presented with the information in Figure 18, it can be seen that the angle of the face is different in the two normalized images, mainly because the position of the points detected from the nose of the person being authenticated is different. Therefore, the authenticater can take measures to ensure that the position of the points detected from the nose matches, such as taking the picture with the nose facing directly towards the camera.

[0065] In this embodiment, a method for determining whether a person in a face image input from an external source may be a person who has been registered in advance is described. A detailed example of step S204 in this embodiment is shown in Figure 20.

[0066] In S2000, prepare a set of facial images. For the sake of explanation, we will prepare 10 facial images for each of 100 people, as in procedure S600.

[0067] In step S1001, facial similarity is calculated from the facial images prepared in step S2000. For 4500 pairs of facial images of the same person and 495000 pairs of facial images of different people, facial similarity is calculated using the method described in Non-Patent Document 1. Furthermore, the facial similarity is stored in the statistical value storage unit 110.

[0068] In step S2002, a threshold for facial similarity is set for facial recognition. Similar to step S602, a threshold α is set such that the rate of unrecognized faces is 0.1 for pairs of facial images of the same person.

[0069] In S2003, the match score is calculated from the face images prepared in step S2000. First, from the group of face images created in S2000, only pairs of face images of the same person and pairs of face images of different people are selected, provided that the similarity calculated in step S2001 is less than the threshold α. Next, the match score is calculated for each of the selected pairs. Furthermore, the calculated match score is saved in the statistical value storage unit 110.

[0070] In S2004, it is determined whether or not a person in an externally input facial image is likely to be a person who has been pre-registered. First, a two-class classifier is created using a nonlinear support vector machine (SVM) based on the match score distribution calculated in procedure S2003. Here, the two classes are the class for pairs of images of the same person and the class for pairs of images of different people.

[0071] Next, the match score is calculated for the pair of registered images that have the highest similarity to the externally inputted face image. The similarity and match scores of this pair are processed by a two-class classifier. If the pair is classified into the class of images of the same person, it is determined that there is a possibility that the images are of the same person; if the pair is classified into the class of images of different people, it is determined that there is no possibility that the images are of the same person.

[0072] The processing details of procedure S2004 will be explained in more detail using Figures 21-23.

[0073] Figure 21 is a scatter plot with the horizontal axis representing the match score and the vertical axis representing the face similarity score. In Figure 21, 2100 is the threshold α determined in procedure S2002, and 2101 is the set of points plotted on the scatter plot for pairs of images of the same person for which the match score was calculated in procedure S2003. Also, 2102 is the set of points plotted on the scatter plot for pairs of images of different people for which the match score was calculated in procedure S603.

[0074] Figure 22 shows an example of plotting a class division line 2201, calculated using a nonlinear SVM, on the scatter plot shown in Figure 21. The classes to the left of the division line represent pairs of the same person's images, and the classes to the right of the division line represent pairs of images of different people.

[0075] In this embodiment, a nonlinear SVM was used to create a two-class classifier, but other methods may be used. For example, a linear SVM may be used to calculate the boundary line. Alternatively, multiple classes may be created using methods such as the k-means algorithm, and each class may be used to represent pairs of images of the same person and pairs of images of other people.

[0076] Furthermore, depending on the distribution trends of data between images of the same person and data between images of different people, accurate classification may be difficult. Therefore, the region where the facial similarity is less than the threshold α may be divided into multiple regions, and classification may be performed in each region. For example, as shown in Figure 23, a new threshold γ2300 may be used to divide the region below the threshold α into two regions, region 2301 and region 2302, and two-class classification may be performed in each region.

[0077] In the method described above, the determination is made using the similarity and match scores stored in the statistical value storage unit 110. The method of determination is not limited to this, and the determination may be made using other methods. For example, a rule-based determination method may be used, such as investigating and using an optimal threshold for a specific environment in which the facial recognition system is used in advance. Alternatively, a classifier may be created in advance by machine learning to learn the correlation between the distributions of similarity and match scores, and used for determination. Alternatively, a method may be used in which a classifier is created using any of these methods, and then if the classification performance is below a certain value, it is always determined that there is no possibility of it being the person in question.

[0078] Dense matching techniques, which densely estimate corresponding points between images, can sometimes struggle to consistently detect these points. Similarly, determining the likelihood of an image being the same person using a threshold calculated from the match degree of image pairs of the same individual can be difficult depending on the quantity and quality of registered images. Therefore, by utilizing the process described above, stable and highly accurate determinations can be achieved.

[0079] (Variation 1) In the first and second embodiments, a method was described for calculating the match score between an externally input facial image and the pre-registered facial image with the highest similarity. Here, all registered facial images were images of actual people.

[0080] However, using only the image with the highest similarity to calculate the match score can lead to unstable results that may not meet expectations. For example, even if the person in the externally input facial image and the person in the image with the highest similarity are different people, the match score may be calculated as low, leading to the incorrect conclusion that they are the same person. Therefore, this modified version describes a method to improve the stability of the match score calculation results using artificially generated facial images.

[0081] The configuration of the image processing apparatus in this modified example will be the same as in the first embodiment, using the configuration shown in Figure 1. Similarly, the processing procedure will be the same as in the first embodiment, using the procedure shown in Figure 2. The difference between the first embodiment and this modified example lies in the processing content of step S203. Explanations common to both the first embodiment and this modified example will be omitted, and the differences from the first embodiment will be explained in detail.

[0082] Figure 24 shows the procedure for calculating the degree of match in this modified example.

[0083] In step S2400, a template face image is created. In this modified example, an average brightness value image is created from face images previously registered in the face image storage unit 106 and used as the template face image.

[0084] In step S2401, similar to step S203, the match score between the input face image and the registered face image with the highest similarity to the input face image is calculated.

[0085] In step S2402, the degree of match between the input face image and the template face image generated in S2400 is calculated. The method described in Non-Patent Document 2 is used to calculate the degree of match, similar to the method described in step S203.

[0086] In step S2403, the average of the match scores calculated in step S2401 and step S2402 is calculated as the match score. The method of calculating the match score is not limited to this; for example, a weighted average of the match scores calculated in step S2401 and step S2402 may be used as the match score to determine whether the person to be authenticated is likely to be the person whose face image has been registered.

[0087] This modified example can improve the stability of the match score calculation results.

[0088] (Modification 2) In the first and second embodiments, the matching results performed when calculating the match score are displayed using both externally input facial images and pre-registered facial images. However, if a person other than the person in the registered facial image sees or photographs the pre-registered facial image, it could become reference information for someone attempting to succeed in authentication through impersonation, which could pose a security risk. Furthermore, if the person being authenticated and the person in the registered facial image are different people, there are privacy concerns. Therefore, we will describe an example of a modified method that changes the way pre-registered facial images are displayed and does not directly display the facial images.

[0089] Figure 25 shows an example of how matching results are displayed in this modified example. The left image in Figure 25 is a face image input from an external source. For pre-registered face images, only the image outline is displayed, as shown in the right image in Figure 25, and the matching results are displayed without showing the image itself.

[0090] Furthermore, the matching results may be displayed after some kind of processing has been applied to the pre-registered facial images. For example, as shown in Figure 26, the matching results may be displayed with an externally input facial image on the left and a blurred pre-registered facial image on the right.

[0091] Furthermore, as shown in Figure 27, the matching results may also be displayed using illustrations based on pre-registered face images. Figure 27 shows the matching results using the method of the first embodiment. That is, the right figure in Figure 2700 is an input face image from a real photograph, and the left figure is a pre-registered face image from a real photograph. On the other hand, Figure 27 shows the matching results using illustration 2703, which is based on a pre-registered face image 2702.

[0092] By preventing registered facial images from being directly displayed when showing matching results, the possibility of authentication by others, such as impersonation, can be reduced.

[0093] (Variation 3) In the first and second embodiments, procedure S201, which is a facial recognition process, is performed, and procedure S203, which is a match degree calculation process, is performed only if the facial recognition process fails. However, it is also possible to omit the judgment process in S202 and always calculate the match degree. By calculating the match degree each time the facial recognition process is performed, regardless of whether the facial recognition process is successful or not, data on the match degree and matching results when the facial recognition process is successful can be obtained. By using the obtained data to set the threshold for the match degree, it is possible to determine the likelihood of the person being identified with greater accuracy.

[0094] Furthermore, for example, the authentication and determination of the likelihood of identity in the first embodiment may be used as preprocessing to perform more detailed and accurate facial recognition in a later stage. In conventional facial recognition systems that register multiple registered images for a single person, the registered image with the highest degree of match with the input image may be prioritized for more detailed and accurate facial recognition processing.

[0095] (Modification 4) In the first and second embodiments, face recognition processing is performed first, followed by the calculation of the match score. However, when face recognition processing is performed on a large number of registered face images, the time required for face recognition processing may be long. Therefore, the match score may be calculated before face recognition processing, and the images to be processed may be selected based on the match score. In this case, face recognition processing is performed on the registered face images selected based on the match score, and if face recognition processing fails, the likelihood of the person being identified is determined based on the match score. Even when performing face recognition processing on face images obtained through re-shooting based on the likelihood of the person being identified, the images may be selected based on the match score.

[0096] This modified version reduces unnecessary retakes and misidentification, as well as shortening the time required for facial recognition processing.

[0097] (Other embodiments) Furthermore, the present invention can also be realized by performing the following process: that is, supplying software (program) that realizes the functions of the above-described embodiment to a system or device via a network or various storage media, and having the computer (or CPU or MPU, etc.) of that system or device read and execute the program.

[0098] It should be noted that the above embodiments are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0099] Furthermore, this disclosure includes the following components.

[0100] (Composition 1) A facial image storage unit in which a person's facial image is registered, An authentication unit that authenticates if the person in the input facial image is the same person as the person in the facial image stored in the facial image storage unit, A matching unit that calculates the degree of match between the input facial image and the registered facial image, A determination unit that uses the authentication result from the authentication unit and the match score to determine whether or not the person in the input face image is likely to be the person in the registered face image, An information processing device characterized by comprising an output unit that outputs information corresponding to the determination result output by the determination unit.

[0101] (Configuration 2) If the determination unit determines that the above possibility exists, The information processing apparatus according to configuration 1, characterized in that the output unit outputs information based on the results of the matching process for calculating the degree of match.

[0102] (Composition 3) The information processing apparatus according to configuration 1 or 2, characterized in that the output unit outputs the input face image.

[0103] (Composition 4) The information processing apparatus according to any one of configurations 1 to 3, characterized in that the output unit does not output the registered face image.

[0104] (Composition 5) The information processing apparatus according to any one of configurations 1 to 3, characterized in that the output unit outputs an image obtained by processing the registered face image.

[0105] (Composition 6) If the determination unit determines that the above possibility does not exist, The information processing apparatus according to configuration 1, characterized in that the output unit does not output the result of the matching process.

[0106] (Composition 7) If the authentication unit fails to authenticate that the person in the input facial image is the same person as the person in the registered facial image, The information processing apparatus according to any one of configurations 1 to 6, characterized in that the matching unit calculates the degree of match.

[0107] (Composition 8) The matching unit includes a statistical value storage unit, The statistical value storage unit stores values ​​based on the similarity between pairs of facial images of a certain person and another facial image of the same person, and the similarity between a facial image of a certain person and a facial image of a person other than the same person. The information processing apparatus according to any one of configurations 1 to 7, characterized in that the determination unit performs the determination using the value.

[0108] (Composition 9) The information processing apparatus according to configuration 8, characterized in that the determination unit determines whether or not the determination is possible based on the degree of match and the value.

[0109] (Composition 10) The information processing device according to any one of configurations 1 to 9, characterized in that the determination unit comprises a classifier created by machine learning, and the classifier performs the determination.

[0110] (Composition 11) The information processing apparatus according to any one of configurations 1 to 10, characterized in that the determination unit performs the determination based on the relationship between the degree of match and a predetermined threshold.

[0111] (Composition 12) The information processing apparatus according to configuration 11, characterized in that the determination unit determines that there is a possibility when the degree of match is smaller than the predetermined threshold.

[0112] (Composition 13) The aforementioned face image storage unit stores artificially generated face images. The information processing apparatus according to any one of configurations 1 to 12, characterized in that the matching unit calculates the match degree from the artificially generated face image, the input face image, and the registered face image.

[0113] (Composition 14) The information processing apparatus according to any one of configurations 1 to 13, characterized in that the matching unit calculates the degree of match based on the number of point pairs detected by the dense matching method.

[0114] (Composition 15) The information processing apparatus according to any one of configurations 1 to 14, characterized in that the matching unit calculates the degree of match from a portion of the input face image.

[0115] (Composition 16) The information processing device according to any one of configurations 1 to 15, characterized in that the matching unit calculates the degree of match based on the result of detecting the organ positions in the input facial image.

[0116] (Composition 17) A program that causes a computer to function as an information processing device as described in any one of Configuration 1 to Configuration 16.

[0117] (Composition 18) An information processing method for an information processing device having a face image storage unit in which a person's face image is registered, Authentication step to verify that the person in the input facial image is the same person as the person in the facial image stored in the facial image storage unit, A matching process that calculates the degree of match between the input face image and the registered face image, A determination step that uses the authentication result in the authentication step and the match score to determine whether or not the person in the input face image is likely to be the person in the registered face image, An information processing method characterized by comprising an output step that outputs information corresponding to the determination result in the determination step. [Explanation of symbols]

[0118] 100 Image Processing Devices 102 Matching Department 103 Judgment section 104 Output section 106 Face Image Recording Unit 109 Certification Department

Claims

1. A facial image storage unit in which a person's facial image is registered, An authentication unit that authenticates if the person in the input facial image is the same person as the person in the facial image stored in the facial image storage unit, A matching unit that calculates the degree of match between the input face image and the registered face image, A determination unit that uses the authentication result from the authentication unit and the match score to determine whether or not the person in the input face image is likely to be the person in the registered face image, An information processing device characterized by comprising an output unit that outputs information corresponding to the determination result output by the determination unit.

2. If the determination unit determines that the above possibility exists, The information processing apparatus according to claim 1, characterized in that the output unit outputs information based on the results of the matching process for calculating the degree of match.

3. The information processing apparatus according to claim 2, characterized in that the output unit outputs the input face image.

4. The information processing apparatus according to claim 2, characterized in that the output unit does not output the registered face image.

5. The information processing apparatus according to claim 2, characterized in that the output unit outputs an image obtained by processing the registered face image.

6. If the determination unit determines that the above possibility does not exist, The information processing apparatus according to claim 2, characterized in that the output unit does not output the result of the matching process.

7. If the authentication unit fails to authenticate that the person in the input facial image is the same person as the person in the registered facial image, The information processing apparatus according to claim 1, characterized in that the matching unit performs the calculation of the degree of match.

8. The matching unit includes a statistical value storage unit, The statistical value storage unit stores values ​​based on the similarity between pairs of facial images of a certain person and another facial image of the same person, and the similarity between a facial image of a certain person and a facial image of a person other than the same person. The information processing apparatus according to claim 1, characterized in that the determination unit performs the determination using the value.

9. The information processing apparatus according to claim 8, characterized in that the determination unit determines whether or not the determination is possible based on the degree of match and the value.

10. The information processing apparatus according to claim 1, wherein the determination unit comprises a classifier created by machine learning, and the classifier performs the determination.

11. The information processing apparatus according to claim 1, characterized in that the determination unit performs the determination based on the relationship between the degree of match and a predetermined threshold.

12. The information processing apparatus according to claim 11, characterized in that the determination unit determines that there is a possibility when the degree of match is smaller than the predetermined threshold.

13. The aforementioned face image storage unit stores artificially generated face images. The information processing apparatus according to claim 1, characterized in that the matching unit calculates the match degree from the artificially generated face image, the input face image, and the registered face image.

14. The information processing apparatus according to claim 1, characterized in that the matching unit calculates the degree of match based on the number of pairs of points detected by the dense matching method.

15. The information processing apparatus according to claim 1, characterized in that the matching unit calculates the degree of match from a portion of the input face image.

16. The information processing apparatus according to claim 1, characterized in that the matching unit calculates the degree of match based on the result of detecting the organ positions in the input facial image.

17. A program that causes a computer to function as an information processing device according to any one of claims 1 to 16.

18. An information processing method for an information processing device having a face image storage unit in which a person's face image is registered, Authentication step to verify that the person in the input facial image is the same person as the person in the facial image stored in the facial image storage unit, A matching process that calculates the degree of match between the input face image and the registered face image, A determination step that uses the authentication result in the authentication step and the match score to determine whether or not the person in the input face image is likely to be the person in the registered face image, An information processing method characterized by comprising an output step that outputs information corresponding to the determination result in the determination step.

Images