Information processing device, information processing method and program

Abstract

To enable specification of an image that is not appropriate to be used in object recognition processing.SOLUTION: An information processing device includes: extraction means for extracting feature information from an image; first storage means for storing feature information of an evaluation image determined to cause an inappropriate result in evaluation of the evaluation image related to object recognition performed by using the extraction means; and image specification means for specifying an inappropriate image in the object recognition on the basis of the feature information extracted from an object area image of a processing object by the extraction means, and the feature information stored in the first storage means.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, many techniques have been proposed to extract useful information from images by advanced processing. Among them, object recognition techniques that use a multi-layered neural network called a deep net to identify which of a number of pre-registered classes an object in an image belongs to have been actively researched and developed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2013-73439 A [Patent Document 2] JP 2019-67194 A Summary of the Invention [Problem to be solved by the invention]

[0004] There has been remarkable progress in object recognition technology using deep networks. However, object recognition processing has not always reached a level of perfection that allows for accurate identification. Depending on the conditions under which the input image was taken, misidentification may occur, where an object is mistakenly identified as belonging to a different class than the correct answer. Therefore, studies are being conducted to improve the accuracy of identification by filtering (selecting) the input images and identifying only images that are appropriate for identification.

[0005] Patent Document 1 proposes a technique for preparing a degradation judgment dictionary for judging a degraded image, and judging whether or not to reject a recognition result based on the degree of degradation of an image calculated using the degradation judgment dictionary. However, the technique described in Patent Document 1 requires the preparation of a degradation judgment dictionary, and also requires learning of the degradation judgment dictionary. Patent Document 2 proposes a technique for collecting high-quality training images by image search by closely examining keywords used in image search. However, Patent Document 2 does not mention improving the quality of images once collected.

[0006] An object of the present invention is to make it possible to identify images that are not suitable for use in object recognition processing. [Means for solving the problem]

[0007] The information processing device of the present invention is characterized by having an extraction means for extracting feature information from an image, a first storage means for storing feature information of an evaluation image that is determined to produce an inappropriate result in an evaluation of the evaluation image for object recognition performed using the extraction means, and an image identification means for identifying an inappropriate image for object recognition based on the feature information extracted by the extraction means from the object area image to be processed and the feature information stored in the first storage means. Effect of the Invention

[0008] According to the present invention, it is possible to specify images that are not suitable for use in object recognition processing. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of the configuration of an information processing device according to a first embodiment. [Diagram 2] 4 is a flowchart showing an example of processing by the information processing device in the first embodiment. [Diagram 3] FIG. 11 is a diagram illustrating an example of the configuration of an information processing device according to a second embodiment. [Figure 4] 11 is a flowchart showing an example of processing by the information processing device in the second embodiment. [Diagram 5] FIG. 11 is a diagram illustrating an example of the configuration of an information processing device according to a third embodiment. [Figure 6] 13 is a flowchart showing an example of processing by an information processing device in embodiment 3. [Figure 7] FIG. 10 is a diagram illustrating an example of a configuration related to a learning process. [Figure 8] FIG. 13 is a diagram illustrating a process in a feature vector extraction unit. [Figure 9] FIG. 13 is a diagram illustrating a process in a classifying unit. [Figure 10] 13 is a flowchart showing an example of a process during learning. [Figure 11] FIG. 13 is a diagram illustrating an example of the configuration of an information processing device according to a fourth embodiment. [Figure 12] 13 is a flowchart showing an example of processing by an information processing device according to the fourth embodiment. [Figure 13] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. This embodiment relates to an object recognition technology that identifies which of the preregistered classes an object present in an input image belongs to. In the following, an example will be described in which the object recognition technology of this embodiment is applied to a face recognition (personal recognition) system that identifies which preregistered person's face the face of a person present in an input image belongs to, with the target object being a person's face. In face recognition, for example, a multi-layered neural network called a deep net is used to compare an input face image with face images of multiple people that have been registered in advance, and it is determined who the input face image is. The deep net is also called a deep neural net or deep learning. Note that an example of application to a face recognition system will be described, but the target object in object recognition is not limited to a face, and each embodiment described below can be applied to other target objects.

[0011] <Embodiment 1> In the first embodiment, an information processing device according to the present embodiment is applied to a system that performs face authentication. In the present embodiment, face authentication is performed using a feature extractor that extracts information (feature vector) that distinguishes a person from a face image, for example, by a neural network in which feature extraction parameters are set. If the similarity score obtained by comparing two feature vectors obtained from two images is high, it is determined that the people whose faces appear in the two images are the same person, and if the similarity score is low, it is determined that the people whose faces appear in the two images are different people.

[0012] First, the misidentification-inducing images and the method for collecting them will be described. When an object recognizer that recognizes target objects in images is obtained by learning or other methods, evaluation of the object recognizer is a necessary task in order to know the accuracy of the obtained object recognizer. Evaluation refers to the task of inputting images (evaluation images) into the object recognizer and checking whether it can correctly recognize the target object (whether the recognition result is correct or not). When evaluating an object recognizer, it is common to input a variety of images to check whether there is any bias in the characteristics of the object recognizer, that is, whether there are any images that it is particularly good at or not good at.

[0013] In the following, evaluation images that produce inappropriate recognition results as the object recognizer proceeds with such evaluation work are also referred to as "misidentification-inducing images." The term "misidentification-inducing" is used to mean that the object recognizer cannot properly extract feature information of the target object from the evaluation image, which induces misidentification.

[0014] For example, when a facial image feature extractor is used as an object recognizer, a wide variety of facial images are input to the feature extractor to evaluate it. Then, two feature vectors obtained from the two images are compared to determine whether the faces in the images are the same person or different people (other people).

[0015] In this evaluation task, when two very blurred face images of different people (other people) are input to a feature extractor, the feature extractor extracts similar feature vectors from the two face images, resulting in a high similarity score. The reason for this is that feature vectors cannot be extracted successfully from images that are so blurred that the contours of the face and the positions of the organs are unclear. Moreover, if both images are similarly blurred, the feature extractor may extract similar feature vectors. However, since the face images of different people are input, the result of face recognition should be that the people in the two images are different people, not the same person, and such an image can be said to be an image that induces misidentification. In other words, it can be said that a very blurred image is an image from which it is difficult for a face image feature extractor to extract information that identifies an individual.

[0016] The above-mentioned example of a blurred image has been described as an example of a face image of a different person (a different person). However, even if the face image is of the same person (the same person), if the image is very blurred, it is considered appropriate to determine that it is a different person (a different person) rather than the same person. This is because if the image is so blurred that a human cannot determine whether it is the same person or not, it is considered impossible to properly extract information (feature vector) that distinguishes an individual from the image. If information that distinguishes an individual cannot be properly extracted, it is dangerous to determine that it is the same person based on that information, and it is more natural to determine that it is a different person. Also, when considering applications that use a face recognition system (for example, control of entrance and exit), it can be said that it is safer to determine that an image that is so blurred that information that distinguishes an individual cannot be extracted is a different person (a different person).

[0017] In addition, in a system that performs face recognition, a face detector is generally provided in front of the feature extractor. In many cases, the face detector is configured to identify a face area from an image, extract a face area image, and then input the face area image to the feature extractor. In such a case, if the input image is very blurred, the face detector may erroneously detect a face. When a face is erroneously detected, an image area in which a face does not exist is input to the feature extractor as a face area image. Even for such an image with a false face detection, a high similarity score may be generated in the face recognition process.

[0018] For example, it is easy for a face detector to mistakenly detect a simulacra pattern (a phenomenon in which points and lines arranged in an inverted triangle look like a human face) as a face. If similar simulacra patterns are mistakenly detected from two images and input to a feature extractor, it is easy to imagine that similar feature vectors will be extracted from the two input images, resulting in a high similarity score. It is unnatural to compare images of such simulacra patterns and determine that they are the same person, and it would be more natural to determine that they are different people. In this way, images that have been mistakenly detected as faces can also be considered images that induce misidentification.

[0019] False face detection does not only occur with simulacra patterns, but can also occur with face images with extremely large occlusion areas, images of the back of the head, images with multiple overlapping faces (multiple faces are detected as a single face image), etc. It is considered difficult to extract information (feature vectors) that distinguish individuals from such false face detection images. Such false face detection images can also be considered as images that induce misidentification.

[0020] In this way, for a feature extractor that assumes that a facial image is to be input, images that are not faces (falsely detected facial images) and facial images from which it is difficult to extract information that distinguishes an individual (such as severely blurred or occluded images) can be said to be images that induce misidentification.

[0021] It is not uncommon to encounter such misidentification-inducing images during the evaluation of an object recognizer (feature extractor), etc. Therefore, it is easy to collect and store misidentification-inducing images during evaluation work, etc. This collection of misidentification-inducing images can be done automatically or manually.

[0022] The following method is an example of automatic collection of misidentification-inducing images. For example, when collecting images of erroneously detected faces as misidentification-inducing images, automatic collection is possible by using evaluation images of a face detector. In the evaluation images of the face detector, GT (Grand Truth, correct answer) is added to the face position, and the face detector is evaluated based on whether or not a face is detected at that position. Therefore, if detection results that indicate face detection failure are collected during the evaluation of the face detector, it is possible to collect misidentification-inducing images. Similarly, if an image that does not contain a face at all is input to the face detector and a face is detected, this is a misdetection of a face, and the result may be collected.

[0023] Also, for example, when severely blurred images are automatically collected as misidentification-inducing images, a classifier for determining the blur of an image is prepared, and the blur classifier is applied to the image of the face detection result to collect the images determined to be blurred images. A publicly known method may be used to determine the blur of an image. For example, a method may be used in which an edge extraction filter such as a Laplacian filter is applied to the image, the edge density is calculated, and whether or not the image is blurred is determined based on the edge density.

[0024] Furthermore, when misidentification inducing images are collected manually, for example, the following may be done. When collecting misidentification inducing images by having a person visually view the images of the face detection results, it is possible to collect the images based on the criterion of whether information that can identify an individual can be extracted from the images of the face detection results. It can be determined that it is not possible to extract sufficient information to identify an individual from images such as the above-mentioned severely blurred face images, face images with extremely large areas occluded, face images with extreme angles of face orientation (face images with a face orientation that is significantly different from a frontal orientation), and face images with multiple faces. Such images can be collected manually as misidentification inducing images.

[0025] The following describes an information processing device in embodiment 1. Fig. 1 is a block diagram showing an example of the configuration of an information processing device 100 in embodiment 1. The information processing device 100 has an image input unit 101, an object detection unit 102, a feature vector extraction unit 103, a similarity acquisition unit 104, an image storage unit 105, a feature vector storage unit 106, an image specification unit 107, a similarity adjustment unit 108, and a threshold processing unit 109.

[0026] An image to be processed for object recognition (in this example, an image to be processed for face recognition) is input to the image input unit 101. In this embodiment, it is assumed that a reproduced image of a captured recorded image is input to the image input unit 101.

[0027] The object detection unit 102 detects a target object in an image input from the image input unit 101 (hereinafter also referred to as an "input image"), extracts an area including the target object from the input image, obtains a detected image of the target object, and outputs it as an object area image. In the example of this embodiment, the object detection unit 102 detects a face, which is a target object, in the input image. The face detection method may be an existing publicly known method. For example, the object detection unit 102 extracts shapes corresponding to components of a face area such as a nose, a mouth, and eyes from the input image, and estimates the size of the face from the size of both eyes and the distance between them. Then, the object detection unit 102 may set a region in the input image that is surrounded by a region of the estimated size based on a position corresponding to the center of the nose as an image of the face area. The image of the face area detected by the object detection unit 102 is normalized to a fixed size by a predetermined method and output as a face area image.

[0028] The feature vector extraction unit 103 performs feature vector extraction processing on the object area image input from the object detection unit 102, and obtains and outputs the feature vector of the input object area image. The feature vector is an example of feature information, and the feature vector extraction unit 103 is an example of an extraction means for extracting feature information from an image. The feature vector extraction unit 103 is realized by a neural network in which feature extraction parameters obtained in advance by learning using learning data are set. In other words, the feature vector extraction unit 103 is a trained model. In this embodiment, the feature vector extraction unit 103 performs feature vector extraction processing on the face area image input from the object detection unit 102, and obtains and outputs the feature vector of the face area image.

[0029] The similarity acquisition unit 104 acquires the similarity between two feature vectors derived from two images based on the feature vectors obtained by the feature vector extraction unit 103. Any method for acquiring the similarity is available, but in this embodiment, a method is selected that indicates that the higher the similarity, the more similar the feature amounts (feature vectors), that is, the more likely it is that one image and the other image are the face of the same person. For example, there is a method of taking the cosine of the angle formed by the two feature vectors and adding 1 to it as the similarity score (score range is 0 to 2), or a method of taking the inverse of the Euclidean distance between the two feature vectors as the similarity score.

[0030] The image storage unit 105 stores the misidentification inducing images. The misidentification inducing images and their collection are as described above, so detailed description will be omitted. Images collected as misidentification inducing images in the evaluation work of evaluation images (evaluation images determined to produce inappropriate results) are stored in the image storage unit 105. The feature vector storage unit 106 stores the feature vectors of the misidentification inducing images stored in the image storage unit 105. The feature vectors of the misidentification inducing images stored in the feature vector storage unit 106 are acquired using a neural network in which the same feature extraction parameters as those of the feature vector extraction unit 103 are set, that is, using the same trained model as that of the feature vector extraction unit 103. The feature vectors of the misidentification inducing images may be acquired, for example, using the feature vector extraction unit 103. The image storage unit 105 is an example of a second storage means, and the feature vector storage unit 106 is an example of a first storage means.

[0031] The image specification unit 107 specifies an inappropriate image for object recognition based on the feature vector of the object region image (face region image in this example) obtained by the feature vector extraction unit 103 and the feature vector of the misidentification inducing image stored in the feature vector storage unit 106. Here, the inappropriate image refers to an image having a feature vector similar to the feature vector of the misidentification inducing image. Although the misidentification inducing image that may induce misidentification is specified during the evaluation work, the same thing cannot be done during the actual operation of the face recognition system. Therefore, the image specification unit 107 specifies an object region image that is assumed to be similar to the misidentification inducing image as an inappropriate image based on the feature vector of the misidentification inducing image that has already been specified.

[0032] First, the image specification unit 107 obtains a similarity between the feature vector of the object area image obtained by the feature vector extraction unit 103 and the feature vector of the misidentification inducing image stored in the feature vector storage unit 106. There is no particular limitation on the method of obtaining the similarity, but for example, if the same method as that used for obtaining the similarity by the similarity obtaining unit 104 is used, it is possible to use the similarity obtaining unit 104 for obtaining the similarity, which is preferable. Furthermore, the image specification unit 107 judges whether or not the object area image is an inappropriate image based on the similarity between the feature vector of the obtained object area image and the feature vector of the misidentification inducing image. For example, if the obtained similarity is higher than a preset threshold (inappropriate image judgment threshold), the image specification unit 107 judges that the object area image is an inappropriate image.

[0033] Here, it is generally assumed that a plurality of misidentification-inducing images are stored in the image storage unit 105, and feature vectors obtained from each of the plurality of misidentification-inducing images are stored in the feature vector storage unit 106. That is, it is assumed that a plurality of feature vectors are stored in the feature vector storage unit 106. Therefore, for one object region image, a plurality of similarities with the feature vectors of the misidentification-inducing images are also obtained, and a plurality of comparison results with the inappropriate image determination threshold are also present. Various methods such as those exemplified below can be considered for handling the plurality of comparison results, and the method that produces the most appropriate result may be adopted. For example, the image specification unit 107 may determine that the object region image is an inappropriate image if at least one of the plurality of similarities is higher than the inappropriate image determination threshold. Also, for example, the image specification unit 107 may determine that the object region image is an inappropriate image if half of the plurality of similarities are higher than the inappropriate image determination threshold.

[0034] The similarity adjustment unit 108 adjusts the similarity acquired by the similarity acquisition unit 104 based on the determination result by the image specification unit 107. As described above, it is more natural to determine that an error-inducing image is a different person rather than the same person even if the similarity is high. Similarly, it is better to determine that an inappropriate image similar to an error-inducing image is a different person rather than the same person. The similarity adjustment unit 108 adjusts the similarity score so that the inappropriate image is determined to be a different person. The adjustment method is not particularly limited, but in this embodiment, it is performed as follows as an example.

[0035] Since the similarity acquisition unit 104 acquires similarity for two object area images, the similarity adjustment unit 108 also receives the judgment results for two images, ie, whether the images are inappropriate or not, from the image judgment unit 107. If both of the two object area images are judged to be inappropriate images, the similarity adjustment unit 108 sets the similarity score for these two object area images to 0 (multiplies the similarity score acquired by the similarity acquisition unit 104 by 0). If either one of the two object area images is judged to be an inappropriate image, the similarity adjustment unit 108 halves (1 / 2) the similarity score for these two object area images (multiplies the similarity score acquired by the similarity acquisition unit 104 by 0.5). If neither of the two object area images is determined to be an inappropriate image, the similarity adjustment unit 108 does not adjust the similarity scores for the two object area images (it maintains the similarity scores acquired by the similarity acquisition unit 104).

[0036] By adjusting the similarity scores in this way, the similarity scores related to inappropriate images are adjusted to have smaller values. The similarity scores adjusted by the similarity adjustment unit 108 are output as adjusted similarity scores.

[0037] The threshold processing unit 109 judges whether the people whose faces appear in the two object area images are the same person or different people, based on the adjusted similarity score input from the similarity adjustment unit 108. The threshold processing unit 109 compares the adjusted similarity score with a preset threshold (face authentication judgment threshold) and judges whether the two object area images are images of the same person or different people. If the adjusted similarity score is higher than the face authentication judgment threshold, the threshold processing unit 109 judges that the people are the same person, and if the adjusted similarity score is lower than the face authentication judgment threshold, the threshold processing unit 109 judges that the people are different people. Note that, when the adjusted similarity score is the same as the face authentication judgment threshold, it is sufficient to determine in advance whether to judge that the people are the same person or different people.

[0038] During actual operation of face authentication by the information processing device 100 shown in FIG. 1, if an image similar to an error-inducing image is input as an image to be processed, the image is determined to be an inappropriate image, and the similarity score is adjusted to be smaller. By adjusting the similarity score in this manner, the adjusted similarity score related to the inappropriate image is less likely to exceed the face authentication judgment threshold, and as a result, it is more likely to be determined that the image is a different person in face authentication. According to the information processing device 100 shown in FIG. 1, even during actual operation of face authentication, it is possible to specify an inappropriate image, and to suppress the output of an inappropriate face authentication result such as that caused by an error-inducing image during evaluation.

[0039] The operation of the information processing device 100 shown in Fig. 1 will be described with reference to Fig. 2. Fig. 2 is a flowchart showing an example of processing by the information processing device 100 shown in Fig. 1. Fig. 2 shows the flow of face recognition processing for determining whether face images present in two input images are the faces of the same person or different people. For the sake of simplicity, the description will be given assuming that one input image contains one face image.

[0040] In step S201, the image input unit 101 inputs an image to be subjected to face recognition. In step S202, the object detection unit 102 executes a process of detecting a face, which is a target object, for the image input in step S201 (input image), normalizes the image of the detected face region to a fixed size using a predetermined method, and outputs it as a face region image. Note that if no face is detected in the input image, the subsequent processes are skipped. In step S203, the feature vector extraction unit 103 performs feature vector extraction processing on the face area image obtained in step S202, and acquires the feature vector of the face area image.

[0041] In step S204, the image specification unit 107 judges whether the currently input face area image is an inappropriate image. The procedure for judging whether an image is inappropriate has already been described in detail, so a detailed description will be omitted here. The image specification unit 107 acquires the similarity between the feature vector of the face area image obtained by the feature vector extraction unit 103 and the feature vector of each misidentification inducing image stored in the feature vector storage unit 106, and compares the similarity with an inappropriate image judgment threshold value to make the judgment.

[0042] In step S205, the information processing device 100 judges whether the processes for obtaining feature vectors and judging inappropriate images have been performed on the two input images. That is, it judges whether the two input images to be compared are input and the above-mentioned processes have been performed on each of the input images. If it is judged that the processes have been performed on the two input images (YES in step S205), the process proceeds to step S206. On the other hand, if it is judged that only one input image has been processed (NO in step S205), the process returns to step S201, and the above-mentioned processes of steps S201 to S204 are performed on the other input image to be compared.

[0043] In step S206, the similarity obtaining unit 104 obtains the similarity between the two feature vectors based on the feature vectors obtained from the two face area images.

[0044] In step S207, the similarity adjustment unit 108 refers to the inappropriate image determination result performed in step S204 and determines whether the two currently input face area images have been determined to be inappropriate images. The similarity adjustment unit 108 determines whether at least one of the two currently input face area images has been determined to be inappropriate images based on the inappropriate image determination result by the image specification unit 107. If there is a face area image determined to be an inappropriate image, that is, if the similarity adjustment unit 108 determines that at least one of the face area images has been determined to be an inappropriate image (YES in step S207), the process proceeds to step S208 to adjust the similarity score. If there is no face area image determined to be an inappropriate image, that is, if the similarity adjustment unit 108 determines that neither of the two face area images has been determined to be an inappropriate image (NO in step S207), the process proceeds to step S209 since adjustment of the similarity score is not necessary.

[0045] In step S208, the similarity adjustment unit 108 adjusts the similarity acquired in step S206. In the example of the present embodiment, if any one of the two face area images is determined to be an inappropriate image, the similarity adjustment unit 106 performs adjustment so as to reduce the value of the similarity score indicating the similarity between the feature vectors of the two face area images.

[0046] In step S209, the threshold processing unit 109 compares the similarity score for the feature vectors of the two face area images with a face authentication judgment threshold to determine whether the similarity score exceeds the face authentication judgment threshold. In the process in step S209, if at least one of the two face area images is determined to be an inappropriate image, the similarity score adjusted in step S208 is compared with the face authentication judgment threshold. In addition, if neither of the two face area images is determined to be an inappropriate image, the similarity score acquired in step S206 is compared with the face authentication judgment threshold.

[0047] If it is determined that the similarity score exceeds the face authentication determination threshold (YES in step S209), the process proceeds to step SS210, where the threshold processing unit 109 determines that the people in the two face area images are the same person. On the other hand, if it is determined that the similarity score does not exceed the face authentication determination threshold (NO in step S209), the process proceeds to step SS211, where the threshold processing unit 109 determines that the people in the two face area images are different people.

[0048] According to this embodiment, in the evaluation using the evaluation images, evaluation images that give inappropriate results are stored as misidentification-inducing images, and it is possible to judge whether or not a face area image is an inappropriate image using the stored misidentification-inducing images. Furthermore, when it is judged to be an inappropriate image, it is possible to adjust the similarity between the two face area images.

[0049] This makes it possible to adjust the similarity score for face area images identified as inappropriate images to be smaller, and as a result, it becomes more likely that the people in the two face area images are determined to be different people. In other words, it becomes easier to determine that face recognition results between very blurred images or between images with erroneous face detection are different people, improving the reliability of the face recognition system in practical operation.

[0050] There are various methods for determining whether an image is inappropriate, which the image specification unit 107 performs. That is, when determining whether an image is inappropriate based on a plurality of similarities acquired using the feature vectors of a plurality of misidentification-inducing images, there are various methods for determining based on what method (rule) the image is determined to be inappropriate. The method is not limited to the above, and for example, it may be determined whether an image is inappropriate based on the ratio of similarities that exceed an inappropriate image determination threshold among the plurality of similarities.

[0051] Also, various methods are conceivable for adjusting the similarity score performed by the similarity adjustment unit 108. Also, various methods are conceivable for adjusting the degree of the similarity score (reduction rate). This adjustment method is also not limited to the exemplified method. For example, when the input face area image is determined to be an inappropriate image, the similarity score may be adjusted uniformly regardless of the number of images.

[0052] Furthermore, the image specification unit 107 may output the multiple similarities themselves to the similarity adjustment unit 108, and the similarity adjustment unit 108 may determine the reduction rate of the similarity score based on the multiple similarities input. For example, a neural network may be designed that inputs multiple similarities and outputs the reduction rate of the similarity score, and its parameters may be determined by the following method called cross-validation. In cross-validation, the evaluation data used when collecting the misidentification-inducing images is divided into learning data and validation data, optimal parameters are learned by learning using the learning data, and the learning results are evaluated using the validation data. Using cross-validation is preferable because it makes it possible to select the optimal method while suppressing overfitting to the evaluation data to some extent.

[0053] <Embodiment 2> In the second embodiment, the information processing device of the present embodiment is also applied to a system that performs face recognition (personal authentication). In the first embodiment, an example is shown in which the information processing device is applied to face recognition in which two images are input and it is determined whether the faces in the two images are the same person. In the second embodiment, an example is shown in which face images are registered in advance as registered images and it is determined whether the face in the input image is the same person as any of the registered images or is different from any of the registered images. In particular, a detailed description is given of a case in which the information processing device of the present embodiment is applied to a process of registering face images in advance.

[0054] First, the registration process of a face image will be described. The registration process of a face image refers to a process of registering a face image of a person to be identified by face authentication in advance in a system that performs face authentication. The registered face image (registered image) will be repeatedly compared with an input image (an image in which it is desired to know who the person in the image is) in the subsequent face authentication process. Therefore, using a high-quality image (an image without severe blurring, an image without large occlusions, etc.) as a registered image greatly affects the authentication accuracy in face authentication. In this embodiment, an example will be described in which an inappropriate image is determined when performing the registration process of a face image, and an image identified as an inappropriate image is not adopted as a registered image.

[0055] Fig. 3 is a block diagram showing an example of the configuration of an information processing device 300 in the second embodiment. In Fig. 3, components similar to those shown in Fig. 1 are given the same reference numerals, and duplicated explanations will be omitted. The information processing device 300 has an image input unit 101, an object detection unit 102, a feature vector extraction unit 103, an image storage unit 105, a feature vector storage unit 106, an image specification unit 107, a registration processing unit 301, and a display unit 302. In the following explanation, an image input to be used as a registered image is also referred to as a registration candidate image.

[0056] The registration processing unit 301 determines whether or not to register the currently input registration candidate image as a registration image based on the determination result by the image specification unit 107. If the image specification unit 107 determines that the registration candidate image is not an inappropriate image, the registration processing unit 301 performs registration processing of the registration candidate image. On the other hand, if the image specification unit 107 determines that the registration candidate image is an inappropriate image, the registration processing unit 301 does not perform registration processing of the registration candidate image. In addition, the registration processing unit 301 outputs the determination result of whether or not the registration candidate image has been registered as a registration image to the display unit 302.

[0057] In the registration process of the registration candidate image, the registration candidate image may be stored as it is as registration information, or the feature vector of the registration candidate image may be stored as registration information. In the case where an actual image is stored as registration information, it is necessary to obtain the feature vector of the registration image every time face recognition processing is performed, whereas there is an advantage that the registration image can be continued to be used even if the processing of the feature vector extraction unit 103 is updated. On the other hand, in the case where the feature vector is stored as registration information, it is not necessary to obtain the feature vector of the registration image every time face recognition processing is performed, and the amount of processing in face recognition processing can be reduced.

[0058] The display unit 302 displays the result of whether or not the currently input registration candidate image has been registered, according to the result of the determination of whether or not registration is possible from the registration processing unit 301. The purpose of displaying the result is to inform the user (the person who attempted to register an image) whether or not the registration candidate image was successfully registered as a registration image. Since the registration candidate image was determined to be an inappropriate image and could not be registered, it is necessary to carry out the registration process again with a different image as the registration candidate image.

[0059] According to the information processing device 300 configured in this way, when a face image (face region image) similar to an erroneous recognition inducing image is input as a registration candidate image in the face image registration process, it is determined to be an inappropriate image and registration can be rejected. This makes it possible to prevent images of poor quality (such as a severely blurred image or an image with a large occlusion) from becoming a registered image, and to avoid deterioration of authentication accuracy in a system that performs face authentication.

[0060] The operation of the information processing device 300 shown in Fig. 3 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of processing by the information processing device 300 shown in Fig. 3. Fig. 4 shows the flow of face image registration processing.

[0061] In step S401, the image input unit 101 inputs a registration candidate image. In step S402, the object detection unit 102 executes a face detection process on the input registration candidate image. If a face is not detected in the input image, the process proceeds to step S407, and the registration candidate image is not adopted as a registered image. In step S403, the feature vector extraction unit 103 performs feature vector extraction processing on the registration candidate image, and acquires the feature vector of the registration candidate image.

[0062] In step S404, the image specification unit 107 determines whether the currently input registration candidate image is an inappropriate image or not. The determination whether the registration candidate image is an inappropriate image or not is performed in the same manner as in the first embodiment described above. In step S405, the registration processing unit 301 determines whether or not the registration candidate image has been determined to be an inappropriate image based on the determination result in the image specification unit 107. If the registration processing unit 301 determines that the registration candidate image has not been determined to be an inappropriate image (NO in step S405), the process proceeds to step S406. On the other hand, if the registration processing unit 301 determines that the registration candidate image has been determined to be an inappropriate image (YES in step S405), the process proceeds to step S407.

[0063] In step S406, the registration processing unit 301 registers the registration candidate image as a registration image, and outputs to the display unit 302 a processing result indicating that the registration candidate image has been registered as a registration image. In step S407, the registration processing unit 301 does not register the registration candidate image as a registration image, and outputs to the display unit 302 a processing result indicating that the registration candidate image has not been registered as a registration image. In step S408, the display unit 302 displays the result of the registration process of the registration candidate image.

[0064] According to this embodiment, when an image similar to an error-identification inducing image is input as a registration candidate image, it is determined to be an inappropriate image, and registration of the registration candidate image identified as an inappropriate image can be rejected. In addition, if the error-identification inducing image is given a meaning in advance, when the registration of the registration candidate image is rejected, not only the processing result of the registration rejection but also the reason for the registration rejection can be displayed on the display unit 302. For example, if a reason such as "due to severe blurring" or "due to a large hidden area" is given to or associated with the error-identification inducing image, the reason for the registration rejection can be displayed depending on which error-identification inducing image the registration candidate image resembles. When the registration candidate image is determined to be similar to an error-identification inducing image with severe blurring, the reason for the registration rejection such as "the blurring of the registration candidate image seems to be too severe" can also be displayed. If the reason for the registration rejection of the registration candidate image is known, it becomes clear how to improve it, and a system that is easy for the user (the person who tried to register the image) to use can be realized.

[0065] <Embodiment 3> In the third embodiment, an example will be described in which the information processing device in the present embodiment is applied to a management system for learning images for face recognition (personal authentication). In the present embodiment, an example will be described in which inappropriate images are identified from learning images for face recognition (personal authentication) and the images are excluded from the learning images. Generally, a huge number of learning images are required for neural network learning. In addition, the learning images may contain mislabeled images (learning images in which the correspondence between the image and its label is incorrect) or images that are inappropriate for use in learning face recognition (misdetected faces or images that are face images but from which it is difficult to extract information that distinguishes an individual). It is known that the presence of such images in the learning images significantly affects the accuracy of face recognition obtained as a result of learning.

[0066] Therefore, in order to realize highly accurate face recognition, it is necessary to prepare a large number of high-quality training images without mislabeling. However, it is impossible to visually check the huge number of training images to confirm mislabeling or check the image quality, and it is a challenge to efficiently remove mislabeled images and low-quality images from the entire training images. As a method for removing such inappropriate images from the training images, the information processing device in this embodiment is applied. Removing mislabeled images and low-quality images from the training images in this way is also called cleansing of the training images.

[0067] Fig. 5 is a block diagram showing an example of the configuration of an information processing device 500 in embodiment 3. In Fig. 5, components similar to those shown in Fig. 1 are given the same reference numerals, and duplicated explanations will be omitted. The information processing device 500 has a feature vector extraction unit 103, an image storage unit 105, a feature vector storage unit 106, a learning image management unit 501, and an image specification unit 502.

[0068] The learning image management unit 501 manages the learning images in association with their class labels (person IDs). For ease of explanation, it is assumed that the learning image management unit 501 manages face region images normalized to a certain size by a predetermined method based on the results of face detection as learning images. The learning image management unit 501 outputs the managed learning images to the feature vector extraction unit 103. Furthermore, if the image specification unit 502 determines that the output learning image is an inappropriate image, the learning image management unit 501 removes the image from the learning images. Note that if the image specification unit 502 does not determine that the image is an inappropriate image, the image is not removed from the learning images.

[0069] The image specification unit 502 specifies an inappropriate image based on the feature vector of the training image obtained by the feature vector extraction unit 103 and the feature vector of the misidentification inducing image stored in the feature vector storage unit 106. The image specification unit 502 obtains a similarity between the feature vector of the training image obtained by the feature vector extraction unit 103 and the feature vector of the misidentification inducing image stored in the feature vector storage unit 106, and judges whether the training image is an inappropriate image or not based on the obtained similarity. The image specification unit 502 notifies the training image management unit 501 of the judgment result of whether the training image is an inappropriate image or not. Note that the method of obtaining the similarity and the method of judging an inappropriate image from the similarity may be the same as the method in the above-mentioned embodiment, and are not particularly limited.

[0070] The operation of the information processing device 500 shown in Fig. 5 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of processing by the information processing device 500 shown in Fig. 5.

[0071] In step S601, the learning image management unit 501 inputs one unprocessed learning image from among all learning images. In step S602, the feature vector extraction unit 103 performs feature vector extraction processing on the training image input in step S601, and acquires the feature vector of the input training image. In step S603, the image specification unit 502 determines whether the currently input learning image is an inappropriate image. The determination of whether the learning image is an inappropriate image may be performed in the same manner as in the first embodiment, for example.

[0072] In step S604, the training image management unit 501 determines whether the input training image has been determined to be an inappropriate image based on the determination result notified from the image specification unit 502. If the training image management unit 501 determines that the input training image has been determined to be an inappropriate image (YES in step S604), the process proceeds to step S605. On the other hand, if the training image management unit 501 determines that the input training image has not been determined to be an inappropriate image (NO in step S604), the process proceeds to step S606. In step S605, the learning image management unit 501 excludes the currently input learning image from the learning images.

[0073] In step S606, the training image management unit 501 determines whether or not the cleansing process has been performed on all training images. If the training image management unit 501 determines that the cleansing process has not been performed on all training images, that is, that there are unprocessed training images (NO in step S606), the process returns to step S601. Then, the above-described process from step S601 onwards is performed on the unprocessed training images. On the other hand, if the training image management unit 501 determines that the cleansing process has been performed on all training images (YES in step S606), the process shown in FIG. 6 (cleansing process of training images) is terminated.

[0074] According to this embodiment, it is possible to determine whether or not there is an inappropriate image similar to the misidentification-inducing image in the learning images, and to exclude images identified as inappropriate images from the learning images. This makes it possible to efficiently exclude low-quality images from the entire vast amount of learning images, thereby obtaining high-quality learning images. If learning is performed using the learning images from which the inappropriate images have been excluded, it becomes possible to realize highly accurate face recognition.

[0075] <Embodiment 4> In the fourth embodiment, an example will be described in which the information processing device in the present embodiment is applied to a management system for learning images for face recognition (personal authentication). In the fourth embodiment, a representative vector obtained as a result of learning is used to identify inappropriate images from the learning images. Details regarding the representative vector will be described later, but the number of representative vectors obtained as a result of learning is the same as the number of classes (number of person IDs) present in the learning images. In other words, one representative vector exists for one class label (person ID). In the third embodiment described above, an inappropriate image is identified by acquiring the similarity between the feature vector of the learning image itself and the feature vector of the misidentification-inducing image. In the fourth embodiment described below, an inappropriate image is identified by acquiring the similarity between the representative vector and the feature vector of the misidentification-inducing image.

[0076] Generally, a training image is composed of multiple images for each class, so the number of representative vectors is very small compared to the number of feature vectors of the training image. Therefore, this embodiment has the advantage that the number of calculations required to obtain similarities to identify inappropriate images can be reduced compared to the third embodiment.

[0077] FIG. 7 is a diagram illustrating an example of a configuration related to a learning process in a face authentication system to which the information processing device according to this embodiment is applied. The learning data input unit 701 manages learning data for face recognition. The learning data for face recognition is data in which a face image is associated with a class label (personal ID) of the face image. The learning data input unit 701 supplies the face image to a recognition model 702 and outputs the class label of the face image to a loss calculation unit 705.

[0078] The recognition model 702 has a feature vector extraction unit 703 and a class classification unit 704. Learning means adjusting the parameters (which generally have a large number of parameters) of the feature vector extraction unit 703 and the class classification unit 704 so that the loss value calculated by the loss calculation unit 705 becomes small.

[0079] The feature vector extraction unit 703 extracts information (feature vector) for identifying an individual from an input face image. After completing learning, the feature vector extraction unit 703 is used as the feature vector extraction unit 103 shown in the above-described embodiments 1 to 3. The process of the feature vector extraction unit 703 will be described with reference to FIG. 8.

[0080] 8 is a schematic diagram for explaining the processing in the feature vector extraction unit 703. An image 801 is a face image input to the feature vector extraction unit 802 (703). The face image 801 is an output of the learning data input unit 701 described above.

[0081] The feature vector extraction unit 802 extracts feature vectors using a calculation method called Convolutional Neural Networks (CNN), which is a kind of deep net. CNN extracts abstracted information from an input image by repeatedly performing a process consisting of a convolution process, a nonlinear process, and a pooling process on the input image. A processing unit consisting of a convolution process, a nonlinear process, and a pooling process is often called a layer. There are several known methods for the nonlinear process used, and for example, a method called Rectified Linear Unit (ReLU) may be used. There are also several known methods for the pooling process, and for example, a method called maximum pooling may be used.

[0082] In the feature vector extraction unit 802, the first layer processing unit 803 performs convolution processing, nonlinear processing, and pooling processing on the input face image 801. The output of the first layer processing unit 803 is a first intermediate map 804, which becomes the input of the second layer processing unit 805. Furthermore, the second layer processing unit 805 performs convolution processing, nonlinear processing, and pooling processing on the first intermediate map 804 in the same manner as the first layer processing unit 803, and outputs a second intermediate map 806. The feature vector extraction unit 801 extracts a feature vector capable of identifying an individual by repeatedly performing such layer processing on the input face image 801. The final layer processing unit 807 performs a full connection calculation process to extract a feature vector of the face image 801. In this embodiment, as an example, the number of dimensions (length) of the extracted feature vector is set to d.

[0083] Returning to Fig. 7, the class classification unit 704 estimates a class to which the feature vector belongs based on the feature vector extracted by the feature vector extraction unit 703. In the case of face recognition described in this embodiment, the estimation of the class corresponds to the estimation of a person label. The process of the class classification unit 704 will be described with reference to Fig. 9.

[0084] 9 is a schematic diagram for explaining the processing in the classification unit 704. The classification unit 901 (704) performs calculations using a matrix called a weighting matrix to perform classification.

[0085] In the class classification unit 901, the representative vector holding unit 902 calculates the product of the input feature vector and the weighting matrix it holds. The size of the weighting matrix is ​​(d×n) when the input feature vector has d dimensions and the number of class labels in the training data is n. Therefore, the product of the feature vector and the weighting matrix results in an n-dimensional vector. This n-dimensional vector is output from the representative vector holding unit 902. This n-dimensional vector is referred to as a logit. The representative vector holding unit 902 outputs this logit to the probability calculation unit 903.

[0086] The calculation performed by the representative vector holding unit 902 can be expressed by the following formula. If the weighting matrix is ​​represented by W, then

[0087]

number

[0088] In this case, if the feature vector (d-dimensional) input to the representative vector holding unit 902 is represented by f and the logit (n-dimensional) output from the representative vector holding unit 902 is represented by g, then

[0089]

number

[0090] (* means vector multiplication by a matrix). Element W of weight matrix W i,j (1≦i≦d, 1≦j≦n) are obtained by learning. The n column vectors W that make up the weight matrix W obtained after the learning is completed are j (The size of each column vector is d dimension.) will be called a representative vector. For example, the column vector 904 in the j-th column shown in FIG. 9 will be called the representative vector of class label j.

[0091] The probability calculation unit 903 performs softmax calculations as shown in the following (Equation 6) and (Equation 7) on the logit g output from the representative vector holding unit 902 to calculate the class classification probability.

[0092]

number

[0093] The classification unit 901 performs the above-mentioned calculations.

[0094] 7, the loss calculation unit 705 receives the softmax h (n-dimensional vector) calculated by the classification unit 704 as described above, and also receives a class label corresponding to the face image on which the calculation is based from the training data input unit 701. The type of loss calculated by the loss calculation unit 705 in this embodiment is not particularly limited as long as it is a loss for classification. In this embodiment, as an example, it is assumed that a general softmax cross entropy loss is calculated as a loss for classification.

[0095] When a class label c is input from the training data input unit 701 and a corresponding softmax h is input from the classification unit 704, the softmax cross entropy loss L s is expressed by the following Equation 8.

[0096]

number

[0097] In the learning, the parameters constituting the recognition model 702 are adjusted so as to reduce the loss calculated by the loss calculation unit 705. In this embodiment, the learning method is not particularly limited, but for example, learning can be performed by using a general error backpropagation method and successively updating the parameters by a stochastic gradient method.

[0098] According to (Equation 8), the value g c The larger the logit g calculated by (Equation 5), the smaller the loss. c is calculated as (Equation 9) by the inner product of the feature vector of class label c and the representative vector of class label c.

[0099]

number

[0100] In (Equation 9), f c,k denotes the feature vector extracted from the kth training image belonging to class c.

[0101] As mentioned above, the learning is performed by using the value g calculated by (Equation 9). c In (Equation 9), the feature vector f corresponding to the training image with class label c is c,k and the representative vector W of the class label c c Therefore, the learning is done by calculating the inner product of g c If we proceed in the direction of increasing the size of the vectors (f c,k and W c ) are considered to be pointing in the same direction. Generally, there are multiple learning images (face images) that belong to class c, so multiple feature vectors that belong to class c are also extracted. A representative vector W c Therefore, as learning progresses, the representative vector W c It is expected that the orientation of the representative vector W will be roughly the same as that of many feature vectors of class label c. In other words, without looking at multiple feature vectors of class label c, we can determine the representative vector W c By looking at the feature vector of the class label c, we can see the direction of the feature vector. c is a vector that represents class c. In other words, the representative vector W c can be said to be a vector that represents the face image of a person in class c.

[0102] Next, the flow of processing during learning will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of processing during learning. In step S1001 , the learning data input unit 701 inputs a learning image (face image) to the feature vector extraction unit 703 . In step S1002, the feature vector extraction unit 703 acquires the feature vector of the training image input in step S1001.

[0103] In step S1003, the classification unit 704 calculates a logit based on the feature vector acquired in step S1002. In step S1004, the loss calculation unit 705 calculates a loss based on the logit and the class label calculated in step S1003. In step S1005, parameters of the feature vector extraction unit 703 and the class classification unit 704 are updated based on the loss calculated in step S1004.

[0104] In step S1006, it is determined whether or not a learning end condition is satisfied. There is no particular restriction on the learning end condition, and any known learning end condition may be used for the determination. For example, the learning end condition may be that the learning ends when the parameters are updated a certain number of times. If the learning end condition is satisfied (YES in step S1006), the pre-learning ends. If the learning end condition is not satisfied (NO in step S1006), the process returns to step S1001, and learning continues. This concludes the explanation of face recognition learning.

[0105] When learning is performed in the above manner, a matrix W (Equation 1) in which each column is a representative vector of each class is obtained in the classification unit 704. In this embodiment, an example will be described in which this representative vector is used to identify inappropriate images (mislabeled images or low-quality images) in the training images and remove them from the training images.

[0106] Fig. 11 is a block diagram showing an example of the configuration of an information processing device 1100 in embodiment 4. In Fig. 11, components similar to those shown in Fig. 1 are given the same reference numerals, and duplicated explanations will be omitted. The information processing device 1100 has an image storage unit 105, a feature vector storage unit 106, a representative vector holding unit 1101, an image specification unit 1102, and a learning image management unit 1103.

[0107] The representative vector holding unit 1101 holds learned representative vectors obtained by the above-mentioned learning. The representative vector is an example of representative feature information. The main functions of the representative vector holding unit 1101 have been explained in the representative vector holding unit 902 of FIG. 9, and therefore will not be described here. The representative vector holding unit 1101 also has a function of outputting the representative vectors held therein and class labels corresponding to the representative vectors. The representative vector holding unit 1101 holds a weight matrix as expressed by (Equation 1), for example, and outputs it as a representative vector, column by column, and outputs the corresponding column number as a class label.

[0108] The image specification unit 1102 specifies an inappropriate image based on the representative vector input from the representative vector holding unit 1101 and the feature vector of the misidentification inducing image stored in the feature vector storage unit 106. The image specification unit 1102 acquires a similarity between the representative vector input from the representative vector holding unit 1101 and the feature vector of the misidentification inducing image stored in the feature vector storage unit 106, and judges whether or not the training image is an inappropriate image based on the acquired similarity. Note that the method of acquiring the similarity and the method of judging an inappropriate image from the similarity may be the same as the method in the above-mentioned embodiment, and are not particularly limited.

[0109] The training image management unit 1103 manages training images in association with their class labels. Here, as in the third embodiment, the training image management unit 1103 manages face region images normalized to a certain size by a predetermined method based on the results of face detection as training images. For a representative vector determined by the image specification unit 1102 to be an inappropriate image, the training image management unit 1103 excludes all images belonging to a class corresponding to the representative vector from the training images. For example, n representative vectors W constituting the weight matrix W are j Among (1≦j≦n), the representative vector W c is determined to be inappropriate, the learning image management unit 1103 excludes all images to which c is assigned as the class label from the learning images.

[0110] The operation of the information processing device 1100 shown in Fig. 11 will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of processing by the information processing device 1100 shown in Fig. 11.

[0111] In step S1201, the representative vector holding unit 1101 inputs one unprocessed representative vector from among the held representative vectors to the image specification unit 1102. In step S1202, the image specification unit 1102 determines whether a learning image belonging to a class corresponding to the currently input representative vector is an inappropriate image. The determination of whether the representative vector is an inappropriate image may be performed in the same manner as in the first embodiment described above, with the representative vector being treated as one feature vector.

[0112] In step S1203, the training image management unit 1103 determines whether or not the currently input representative vector has been determined to be that of an inappropriate image based on the determination result in the image specification unit 1102. If the training image management unit 1103 determines that the input representative vector has been determined to be that of an inappropriate image (YES in step S1203), the process proceeds to step S1204. On the other hand, if the training image management unit 1103 determines that the input representative vector has not been determined to be that of an inappropriate image (NO in step S1203), the process proceeds to step S1205. In step S1204, the training image management unit 1103 excludes from the training images all images having a class label corresponding to the currently input representative vector.

[0113] In step S1205, the training image management unit 1103 determines whether or not the cleansing process has been performed on all representative vectors. If the training image management unit 1103 determines that the cleansing process has not been performed on all representative vectors, that is, that there are unprocessed representative vectors (NO in step S1205), the process returns to step S1201. Then, the above-described processes from step S1201 onwards are performed on the unprocessed representative vectors. On the other hand, if the training image management unit 1103 determines that the cleansing process has been performed on all representative vectors (YES in step S1205), the process shown in FIG. 12 (learning image cleansing process) is terminated.

[0114] According to this embodiment, it is possible to determine whether there is a representative vector similar to the feature vector of the misidentification inducing image, and to exclude all images belonging to a class corresponding to the representative vector determined to be that of an inappropriate image from the learning images. This makes it possible to efficiently exclude mislabeled images and low-quality images from the entire vast amount of learning images, and obtain high-quality learning images without mislabels. If learning is performed using the learning images from which the inappropriate images have been excluded, it becomes possible to realize highly accurate face recognition. In addition, the number of calculations required for obtaining similarity in the image specification unit 1102 can be reduced compared to the case of the third embodiment. In detail, to obtain similarity, calculations on the order of the number of learning images are required in the third embodiment, but in the fourth embodiment, calculations on the order of the number of class labels (number of person IDs) constituting the learning images are possible. Therefore, in this embodiment, inappropriate images can be more efficiently excluded from the learning images compared to the third embodiment.

[0115] As described above, the representative vector is a vector that represents the feature vectors of the images belonging to the corresponding class. Therefore, for example, if the representative vector of class label c is similar to the feature vector of the misidentification-inducing image, it means that the images to which c is assigned as the class label are similar to the misidentification-inducing image as a whole. In other words, it is considered that the images of class label c are a collection of very blurry images or a collection of very similar simulacra images. Therefore, it is desirable to exclude all images that constitute such classes from the learning images.

[0116] In addition, if there is a class in which face images of multiple people are assigned to a single class label, the representative vector is expected to be similar to the feature vector of the misclassification-inducing image. For example, assume that face images of multiple people are mixed in an image to which class label d is assigned. In other words, assume that the image with class label d is a mislabeled training image. In such a case, it is expected that learning on the mislabeled training data will not proceed well. This is because, although the entire learning is performed so that face images of different people are classified into different class labels, it is unreasonable to have to proceed with learning so that images of clearly different people are classified into the same class label only for class label d.

[0117] When a face image of a single person is assigned to a single class label, it is expected that as learning progresses, the feature vectors of the class will be approximately the same, and the representative vector will be a vector that represents those feature vectors. However, when face images of multiple people are assigned to a single class label (in the case of mislabeling), it is expected that the feature vectors of the class will not be approximately the same, and the representative vector will not be able to represent the class. It is expected that such a representative vector that does not express the class well will be similar to the feature vector of an image that induces misidentification (a feature vector calculated as a result of not being able to perform feature extraction well for an image that induces misidentification). Therefore, it is expected that the representative vector of the mislabeled class will be determined to be an inappropriate image. In such a case, it is possible to exclude the image corresponding to the mislabeled class from the entire learning images.

[0118] <Hardware configuration> 13 is a block diagram showing an example of the hardware configuration of an information processing device in each of the above-mentioned embodiments. The information processing device 1300 has a CPU 1301, a ROM 1302, a RAM 1303, a storage device 1304, an operation unit 1305, a display unit 1306, a communication unit 1307, and a system bus 1308. The CPU 1301, the ROM 1302, the RAM 1303, the storage device 1304, the operation unit 1305, the display unit 1306, and the communication unit 1307 are communicatively connected via the system bus 1308. Note that the information processing device 1300 may further have other configurations.

[0119] A CPU (Central Processing Unit) 1301 controls the entire information processing device 1300 using computer programs and data stored in a ROM 1302 and a RAM 1303, and realizes, for example, each function of the information processing device described above. Note that the information processing device 1300 may have one or more dedicated hardware pieces different from the CPU 1301, and at least a part of the processing by the CPU 1301 may be executed by the dedicated hardware pieces. Examples of the dedicated hardware pieces include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit).

[0120] A ROM (Read Only Memory) 1302 stores programs that do not require modification. A RAM (Random Access Memory) 1303 temporarily stores programs and data supplied from a storage device 1304, and data supplied from the outside via a communication unit 1307. The RAM 1303 also functions as a main memory, work area, etc. for the CPU 1301. The storage device 1304 is composed of, for example, a hard disk drive (HDD) or a solid state drive (SSD), and stores various data. The storage device 1304 stores, for example, various data required when the CPU 1301 performs processing related to the programs, and various data obtained by the CPU 1301 performing processing related to the programs.

[0121] The operation unit 1305 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and receives operations by a user to input various instructions to the CPU 1301. The display unit 1306 is composed of, for example, a liquid crystal display, an LED, etc., and displays a GUI (Graphical User Interface) for the user to operate the information processing device 1300, processing results by the CPU 1301, etc. Note that the operation unit 1305 and the display unit 1306 may exist as separate devices outside the information processing device 1300. The communication unit 1307 connects the information processing device 1300 to a network, and controls communication with other devices, etc.

[0122] <Other embodiments of the present invention> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0123] It should be noted that the above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0124] The disclosure of the present embodiment includes the following configurations, methods, etc. (Configuration 1) An extraction means for extracting feature information from an image; a first storage means for storing feature information of an evaluation image determined to produce an inappropriate result in the evaluation of the evaluation image regarding object recognition performed by the extraction means; and an image identification means for identifying an inappropriate image in object recognition based on the feature information extracted by the extraction means from the object area image to be processed and the feature information stored in the first storage means. (Configuration 2) 2. The information processing apparatus according to configuration 1, wherein the evaluation image determined to produce an inappropriate result is at least one of a blurred image and an erroneous detection image in which a target object is erroneously detected. (Configuration 3) The information processing device according to configuration 1 or 2, characterized in that the image identification means acquires a similarity between the feature information of the object area image and the feature information stored in the first storage means, and determines whether or not the object area image is the inappropriate image based on the acquired similarity. (Configuration 4) a second storage means for storing the evaluation image determined to produce an inappropriate result; 4. The information processing device according to any one of configurations 1 to 3, wherein the first storage means stores the feature information extracted by the extraction means from the evaluation image stored in the second storage means. (Configuration 5) 5. The information processing device according to any one of configurations 1 to 4, further comprising an object detection means for detecting a target object in an input image and extracting the object region image. (Configuration 6) An acquisition means for acquiring a similarity of the feature information of the two object area images; an adjustment means for adjusting the similarity acquired by the acquisition means based on a result of the image specification means; The information processing device according to any one of configurations 1 to 5, further comprising a processing means for determining whether or not the target objects present in the two object area images are the same based on the similarity adjusted by the adjustment means. (Configuration 7) The processing means determines that the target objects present in the two object area images are the same if the similarity adjusted by the adjustment means is higher than a threshold value; The information processing device according to configuration 6, wherein the adjustment means adjusts the similarity acquired by the acquisition means to be smaller when the object area image is identified as the inappropriate image by the image identification means. (Configuration 8) The information processing device according to configuration 6 or 7, characterized in that when performing a process of registering an object area image as a registered image for authentication, if the object area image is identified as the inappropriate image by the image identification means, registration of the object area image as the registered image is rejected. (Configuration 9) A management unit for managing a plurality of learning images used for learning the extraction unit, 9. The information processing device according to any one of configurations 1 to 8, wherein the management means excludes from the learning images an image identified as the inappropriate image by the image identification means among the learning images. (Configuration 10) The management means manages the learning images and the class labels of the learning images in association with each other, the image identification means identifies an inappropriate image based on representative feature information representative of the plurality of feature information identified based on feature information of the plurality of learning images having the same class label and based on the feature information stored in the first storage means; The information processing device according to configuration 9, wherein the management means excludes from the learning images all images having the class label corresponding to the representative feature information identified as the inappropriate image among the learning images. (Method 1) An information processing method executed by an information processing device having an extraction means for extracting feature information from an image, and a first storage means for storing feature information of an evaluation image determined to produce an inappropriate result in an evaluation of the evaluation image regarding object recognition performed using the extraction means, comprising: an extraction step of extracting the feature information from an object region image to be processed by the extraction means; An information processing method comprising: an image identification step of identifying an inappropriate image for object recognition based on the feature information of the object area image extracted in the extraction step and the feature information stored in the first storage means. (Program 1) A computer of an information processing device having an extraction means for extracting feature information from an image, and a first storage means for storing feature information of an evaluation image determined to produce an inappropriate result in an evaluation of the evaluation image regarding object recognition performed using the extraction means, an extraction step of extracting the feature information from an object region image to be processed by the extraction means; A program for executing an image identification step of identifying an inappropriate image in object recognition based on the feature information of the object area image extracted in the extraction step and the feature information stored in the first storage means. [Explanation of symbols]

[0125] 100, 300, 500: Information processing device 101: Image input unit 102: Object detection unit 103, 703: Feature vector extraction unit 104: Similarity acquisition unit 105: Image storage unit 106: Feature vector storage unit 107, 502, 1102: Image identification unit 108: Similarity adjustment unit 109: Threshold processing unit 301: Registration processing unit 501, 1103: Training image management unit 701: Training data input unit 704: Classification unit 705: Loss calculation unit 1101: Representative vector storage unit

Claims

1. An extraction means for extracting feature information from an image; a first storage means for storing feature information of an evaluation image determined to produce an inappropriate result in the evaluation of the evaluation image regarding object recognition performed by the extraction means; An information processing apparatus comprising: an image identification means for identifying an inappropriate image in object recognition based on the feature information extracted by the extraction means from the object area image to be processed and the feature information stored in the first storage means.

2. 2 . The information processing apparatus according to claim 1 , wherein the evaluation image determined to produce an inappropriate result is at least one of a blurred image and an erroneously detected image in which a target object is erroneously detected.

3. The information processing device according to claim 1, characterized in that the image identification means acquires a similarity between the feature information of the object area image and the feature information stored in the first storage means, and determines whether or not the object area image is the inappropriate image based on the acquired similarity.

4. a second storage means for storing the evaluation image determined to produce an inappropriate result; 2. The information processing apparatus according to claim 1, wherein said first storage means stores said feature information extracted by said extraction means from said evaluation image stored in said second storage means.

5. 2. The information processing apparatus according to claim 1, further comprising an object detection unit for detecting a target object in an input image and extracting the object region image.

6. An acquisition means for acquiring a similarity of the feature information of the two object area images; an adjustment means for adjusting the similarity acquired by the acquisition means based on a result of the image specification means; 2. The information processing apparatus according to claim 1, further comprising a processing unit that determines whether or not the target objects present in the two object area images are the same, based on the similarity adjusted by the adjustment unit.

7. The processing means determines that the target objects present in the two object area images are the same if the similarity adjusted by the adjustment means is higher than a threshold value; 7. The information processing apparatus according to claim 6, wherein the adjustment means adjusts the similarity acquired by the acquisition means so as to be smaller when the object area image is identified as the inappropriate image by the image identification means.

8. 7. The information processing device according to claim 6, wherein, when performing a process of registering an object area image as a registered image for authentication, if the object area image is identified as the inappropriate image by the image identification means, registration of the object area image as the registered image is rejected.

9. A management unit for managing a plurality of learning images used for learning the extraction unit, The information processing apparatus according to claim 1 , wherein the management means excludes from the learning images images any image identified as the inappropriate image by the image identification means.

10. The management means manages the learning images and the class labels of the learning images in association with each other, the image identification means identifies an inappropriate image based on representative feature information representative of the plurality of feature information identified based on feature information of the plurality of learning images having the same class label and based on the feature information stored in the first storage means; The information processing apparatus according to claim 9 , wherein the management means excludes from the learning images all images having the class label corresponding to the representative feature information identified as the inappropriate image among the learning images.

11. An information processing method executed by an information processing device having an extraction means for extracting feature information from an image, and a first storage means for storing feature information of an evaluation image determined to produce an inappropriate result in an evaluation of the evaluation image regarding object recognition performed using the extraction means, comprising: an extraction step of extracting the feature information from an object region image to be processed by the extraction means; An information processing method comprising: an image identification step of identifying an inappropriate image in object recognition based on the feature information of the object area image extracted in the extraction step and the feature information stored in the first storage means.

12. An information processing device includes a computer having an extraction means for extracting feature information from an image, and a first storage means for storing feature information of an evaluation image that is determined to produce an inappropriate result in an evaluation of the evaluation image regarding object recognition performed using the extraction means, an extraction step of extracting the feature information from an object region image to be processed by the extraction means; A program for executing an image identification step of identifying an inappropriate image in object recognition based on the feature information of the object area image extracted in the extraction step and the feature information stored in the first storage means.

Images