Image processing device, image processing method, and program

Abstract

To provide an image processing apparatus that improves classification accuracy of an object in an image.SOLUTION: In an image processing system including an imaging apparatus, an image processing apparatus, and a network, the image processing apparatus 110 includes: a generation unit 302 that generates, for each class, a feature of an image group belonging to the class based on classification information for classifying each of a plurality of images to one of a plurality of classes depending on an object contained in an image; and a determination unit 303 that determines, using the feature of each class, a possibility that the image group belonging to the class includes an image containing another object different from the object corresponding to the class.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] This invention relates to the classification of subjects in images. [Background technology]

[0002] Traditionally, to obtain training images for machine learning models, a large number of images containing a desired person or other subject are classified into a desired class. However, due to time and labor constraints, the large number of images belonging to each class classified by the image processing device are not thoroughly reviewed by the user. Therefore, images that do not contain the desired person may be included in a class that contains images containing the desired person.

[0003] In light of the above problems, Patent Document 1 proposes collecting a large number of images corresponding to keywords of objects and a large number of images corresponding to related word groups associated with the keywords of objects, classifying these images, and selecting a desired class from each class. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2019-67194 [Non-patent literature]

[0005] [Non-Patent Document 1] "ArcFace: Additive Angular Margin Loss for Deep Face Recognition", URL: https: / / arxiv.org / abs / 1801.07698 [Non-Patent Document 2] "The Star Clustering Algorithm for Static and Dynamic Information Organization", Journal of Graph Algorithms and Applications, vol. 8, no. 1, pp. 95-129 (2004), URL: https: / / citeseerx.ist.psu.edu / viewdoc / download?doi=10.1.1.89.272&rep=rep1&type=pdp [Non-Patent Document 3] "Deep Residual Learning for Image Recognition", in arXiv, URL: https: / / arxiv.org / abs / 1512.03385 [Overview of the project] [Problems that the invention aims to solve]

[0006] However, the method described in Patent Document 1 has the problem that it is difficult to generate related word sets for specific person names that are not general keywords, so images of other people may be included in a class defined to include images of a specific person.

[0007] Therefore, the present invention aims to improve the accuracy of classifying the subject matter in an image. [Means for solving the problem]

[0008] To achieve the object of the present invention, an image processing apparatus according to one embodiment of the present invention has the following configuration. That is, the image processing apparatus includes a generation means for generating characteristics of image groups belonging to a class, based on classification information that classifies each of a plurality of images into one of a plurality of classes according to the subject depicted in the image, and a determination means for determining, using the characteristics of each class, the possibility that the image group belonging to the class includes images depicting subjects other than the subject corresponding to the class. [Effects of the Invention]

[0009] According to the present invention, the classification accuracy of the subject in the image can be improved.

Brief Description of the Drawings

[0010] [Figure 1] The figure which shows an example of a structure of an image processing system which concerns on 1st Embodiment. [Figure 2] The figure which shows an example of a hardware structure of an image processing apparatus which concerns on 1st Embodiment. [Figure 3] The block diagram which shows an example of a functional structure of an image processing apparatus which concerns on 1st Embodiment. [Figure 4] The figure which shows an example of an image group which concerns on 1st Embodiment. [Figure 5] The figure explaining the similarity obtained based on the features of two images which concerns on 1st Embodiment. [Figure 6] The figure which shows the result of classifying the image group 400 into n classes which concerns on 1st Embodiment. [Figure 7] The figure which shows the determination result of each class which concerns on 1st Embodiment. [Figure 8] The figure which shows an example of the screen which displayed the image group of class number C0 which concerns on 1st Embodiment. [Figure 9] The block diagram which shows an example of a functional structure of a generation part which concerns on 1st Embodiment. [Figure 10] The detailed figure of the representative vector generation process by the generation part which concerns on 1st Embodiment. [Figure 11] The flowchart of the representative vector generation process by the generation part which concerns on 1st Embodiment. [Figure 12] The figure which shows the determination result of each class in the case of threshold value NA = 9.0 which concerns on 1st Embodiment. [Figure 13] The figure which shows the determination result of each class in the case of threshold value NB = 3.0 which concerns on 1st Embodiment. [Figure 14] The figure which shows the determination result of each class in the case of threshold value SC = 0.7 which concerns on 1st Embodiment. [Figure 15]A diagram showing the image corresponding to class number C1 in Figure 6, according to the first embodiment. [Figure 16] A flowchart illustrating the image processing flow according to the first embodiment. [Figure 17] A diagram showing the image corresponding to class number C1 in Figure 6, related to a modified example. [Figure 18] A diagram showing the image corresponding to class number C1 in Figure 6, related to a modified example. [Figure 19] A flowchart illustrating the image processing flow according to the second embodiment. [Figure 20] A diagram showing an example of the DNN configuration of the DNN holding unit. [Figure 21] An overview diagram of the representative vector generation process by the generation unit. [Modes for carrying out the invention]

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] (First Embodiment) The image processing device generates characteristics for each class of images based on classification information that classifies each of several images into one of several classes according to the subject depicted in the images. Using the characteristics for each class, the image processing device determines whether the group of images belonging to a class contains images of subjects other than the subject corresponding to the class.

[0013] Furthermore, this embodiment may also be used as an image processing system combining an imaging device and an image processing device.

[0014] Figure 1 shows an example of the configuration of an image processing system according to the first embodiment. The image processing system 10 includes an imaging device 100, an image processing device 110, and a network 120.

[0015] The imaging device 100 is a camera that captures images of a subject, and is, for example, a digital camera or a network camera. The imaging device 100 is a single camera, but may consist of two or more cameras.

[0016] The image processing device 110 is a device that detects classes from each class that contain images that do not show a specific person, and is, for example, a desktop computer and a laptop computer.

[0017] Network 120 connects the imaging device 100 and the image processing device 110. Network 120 is, for example, a wired LAN and a wireless LAN.

[0018] Figure 2 is a block diagram showing an example of the hardware configuration of an image processing device according to the first embodiment.

[0019] The image processing device 110 includes an input unit 201, a display unit 202, an I / F 203, a CPU 204, a RAM 205, a ROM 206, a storage unit 207, and a data bus 208.

[0020] The input unit 201 is a device for the user to input various types of data, and includes, for example, a keyboard, mouse, and touch panel.

[0021] The display unit 202 is a device that displays various types of data, and includes, for example, a liquid crystal display (LCD).

[0022] I / F203 transmits and receives various information between the image processing device 110 and other devices (not shown) via a network 120 such as the Internet.

[0023] The CPU 204 is a processor that comprehensively controls each part of the image processing device 110. The CPU 204 reads the control program from the ROM 206, loads it into the RAM 205, and executes the program to perform various controls. By executing the image processing program in the ROM 206 and the storage unit 207, the CPU 204 performs image processing on the image data.

[0024] RAM205 is a temporary storage area for programs executed by CPU204 and work memory.

[0025] ROM206 stores control programs for controlling each part of the image processing device 110.

[0026] The storage unit 207 is a device for storing various types of data, such as image data, setting parameters, and various programs. The storage unit 207 can also store data from external devices (not shown) via the I / F 203.

[0027] The data bus 208 is a transmission path for transmitting data, and it transmits image data and other data received from external devices via the I / F 203 to the CPU 204, RAM 205, and ROM 206. The data bus 208 also transmits image data and other data from the image processing device 110 to external devices.

[0028] Figure 3 is a block diagram showing an example of the functional configuration of an image processing apparatus according to the first embodiment.

[0029] The image processing device 110 comprises an acquisition unit 300, a classification unit 301, a generation unit 302, a determination unit 303, a display unit 304, and a storage unit 305. Furthermore, since all parts of the image processing device 110 are connected to a network, each part of the image processing device 110 can send and receive data from one another.

[0030] The acquisition unit 300 stores the image group acquired from an external device (for example, the storage unit 207) in the storage unit 305. Here, the image group refers to multiple images in which a specific person or other object is depicted.

[0031] The classification unit 301 performs class classification of the image group acquired from the storage unit 305. The classification unit 301 also stores the class classification results of the image group in the storage unit 305.

[0032] The generation unit 302 generates representative vectors that show typical features of each class based on the image group acquired from the storage unit 305 and the classification results of the image group. The generation unit 302 also stores the representative vectors in the storage unit 305.

[0033] The determination unit 303 determines, based on the representative vector obtained from the storage unit 305, whether or not each class contains images that do not depict a specific person. The determination unit 303 also stores the determination result for each class in the storage unit 305.

[0034] Based on the determination results for each class in the storage unit 305, the display unit 304 displays on the display unit 202 classes that include images that do not show a specific person.

[0035] Figure 4 shows an example of a group of images according to the first embodiment.

[0036] Image group 400 includes images of multiple people's faces captured by the imaging device 100 (for example, a digital camera). Image group 400 includes facial images of six people. However, image group 400 may also include facial images of five or fewer people, or facial images of seven or more people.

[0037] Figure 5 illustrates the similarity score calculated based on the features of two images.

[0038] Figure 5 shows feature 502 extracted from image 500 and feature 503 extracted from image 501. The generation unit 302 calculates the similarity of the features (cosine similarity) based on the angle 504 formed by the vector of feature 502 and the vector of feature 503. Here, the angle 504 is the cosine (cosθ) value. Similarity is a measure of how similar two feature vectors are, and is expressed as a numerical value in the range of -1 to 1. (1) If the similarity is -1 (angle 504 is 180°), it indicates that the two feature vectors are dissimilar. (2) If the similarity is 0 (angle 504 is 90°), it indicates that the two feature vectors are neither similar nor dissimilar. (3) If the similarity is 1 (angle 504 is 0°), it indicates that the two feature vectors are similar.

[0039] Next, the classification unit 301 determines the number of classes and the classes to which images 500 and 501 belong, respectively, based on the similarity. The classification unit 301 determines the number of classes and the classes to which images 500 and 501 belong, respectively, without setting the number of classes in advance (see Non-Patent Literature 2).

[0040] Figure 6 shows the results of classifying the group of 400 images into n classes.

[0041] The class number is a classification number obtained by classifying the image group 400 into n classes, such as C0, C1, and C n-1 This includes each class number, with each image group corresponding to it.

[0042] Figure 7 shows an example of the results for each class.

[0043] The classification unit 301 determines, based on the classification results of the image group, whether each class contains images that do not depict a specific person. For example, if the image group corresponding to class number C0 contains one image, the classification unit 301 determines that class number C0 is a class that contains images depicting a specific person.

[0044] Furthermore, if the image group corresponding to class number C0 has two or more images, the classification unit 301 calculates the similarity of features between any two images selected from the image group and selects the smallest similarity M among all the calculated similarities of features. Then, based on whether the smallest similarity M is smaller than a predetermined threshold m, the classification unit 301 determines whether class number C0 is a class that includes images that do not depict a specific person.

[0045] Here, we assume the threshold m=0. The classification unit 301 determines that class number C0 is a class that includes images that do not depict a specific person if the minimum similarity M (=-0.5) of class number C0 is less than the threshold m=0. Also, the classification unit 301 determines that class number C1 is a class that includes only images depicting a specific person if the minimum similarity M (=+0.2) of class number C1 is not less than the threshold m=0.

[0046] Furthermore, the classification unit 301 is class number C n-1 If the minimum similarity M (=+0.1) is not less than the threshold m=0, then the class number C n-1 The class determines that an image contains a specific person. The class determination result is represented as "YES" or "NO" in the "Noise Candidate?" column. Here, "Noise Candidate" refers to a class that contains images that do not contain a specific person. "YES" indicates that the class contains only images that contain a specific person. "NO" indicates that the class contains images that do not contain a specific person. In this way, the classification unit 301 can determine whether each class contains images that do not contain a specific person based on the similarity of the features of the images contained in each class. On the other hand, the determination unit 303 can determine whether each class contains images that do not contain a specific person through a different process. Details of the process performed by the determination unit 303 will be described later.

[0047] Figure 8 shows an example of a screen displaying a group of images for class number C0.

[0048] The display unit 202 includes a user interface (UI). Screen 800 is a screen displayed on the user interface (UI). Screen 800 includes screen 801, button 802, button 803, and button 804. The message "The following classes have been extracted as noise candidates" is displayed on screen 800.

[0049] The user views screen 801, which displays a group of images with class number C0 that have been extracted as noise candidates, and selects one of the buttons 802 to 804.

[0050] When button 802 or button 803 is pressed by the user, the display unit 304 registers the class of screen 801 as either a class with noise or a class without noise. Noise refers to an image in which a specific person is not visible. The display unit 304 stores the registration information for class number C0 in the storage unit 207.

[0051] When button 804 is pressed by the user, the display unit 304 performs a detailed analysis of the image set on screen 801.

[0052] Figure 9 is a block diagram showing an example of the functional configuration of the generation unit.

[0053] The generation unit 302 includes a feature extraction unit 900, a logit calculation unit 901, a probability calculation unit 902, a loss calculation unit 903, an update unit 904, and a DNN storage unit 905. Furthermore, the generation unit 302 includes a feature storage unit 906, a logit storage unit 907, a probability storage unit 908, and a loss storage unit 909.

[0054] All parts of the generation unit 302 are connected to a network, allowing them to send and receive data from each other. The generation unit 302 also processes the image group 910 acquired from an external device (such as the storage unit 207) and the class information 911 to which the image group 910 belongs. As a result, the generation unit 302 generates a representative vector 912 that represents the characteristics of the class information 911 and outputs the representative vector 912 to the external device.

[0055] The feature extraction unit 900 acquires the DNN from the DNN storage unit 905 and acquires the image group 910 from an external device (for example, the storage unit 207). The feature extraction unit 900 also stores the features extracted from the image group 910 using the DNN in the feature storage unit 906.

[0056] The logit calculation unit 901 acquires features of the image group 910 from the feature storage unit 906 and acquires the DNN from the DNN storage unit 905. The logit calculation unit 901 also stores the logit calculated from the features of the image group 910 using the DNN in the logit storage unit 907.

[0057] The probability calculation unit 902 calculates the probability of the individual using the logit obtained from the logit storage unit 907. The probability calculation unit 902 also stores the probability of the individual in the probability storage unit 908.

[0058] The loss calculation unit 903 calculates the loss using the personal probability obtained from the probability storage unit 908. The loss calculation unit 903 also stores the calculated loss in the loss storage unit 909.

[0059] The update unit 904 obtains the loss from the loss retention unit 909 and the DNN from the DNN retention unit 905. The update unit 904 also updates the parameters of the DNN based on the loss.

[0060] Here, we will explain the configuration of the DNN. Figure 20 is a diagram illustrating an example of the configuration of the DNN contained in the DNN holder.

[0061] DNN2000 comprises an intermediate layer 2001 and a fully connected layer 2002. DNN2000 includes the intermediate layer and fully connected layer of Residual network (Non-Patent Literature 3), but may also include other intermediate layers and fully connected layers.

[0062] The intermediate layer 2001 has a convolutional layer and a pooling layer for extracting features from image data. The convolutional layer holds parameters that are updated by the update unit 904 in Figure 9.

[0063] The fully connected layer 2002 has parameters for calculating the logit from the features of the image data. These parameters are updated by the update unit 904, similar to the parameters of the convolutional layer of the intermediate layer 2001.

[0064] Figure 21 is a schematic diagram of the representative vector generation process by the generation unit.

[0065] The generation unit 302 is equipped with a DNN 2102 and acquires image group 2100 and class information 2101 from an external device (for example, the storage unit 207).

[0066] First, the generation unit 302 uses the DNN 2102 to process the image group 2100 and class information 2101, thereby calculating the probability 2103 of each image belonging to the image group 2100 being the person in question.

[0067] Next, the generation unit 302 calculates the loss 2104 by comparing the class information 2101 with the probability of being that person 2103.

[0068] Furthermore, the generation unit 302 updates the parameters of DNN2102 using the loss 2104. The generation unit 302 repeatedly calculates the probability 2103, calculates the loss 2104, and updates the parameters of DNN2102 until the value of the loss 2104 satisfies a predetermined condition.

[0069] After the generation unit 302 finishes updating the parameters, it outputs some of the parameters of the DNN2102 to the outside as representative vectors.

[0070] Figure 10 is a detailed diagram of the representative vector generation process by the generation unit. Figure 10 shows the image 1001, the fully connected layer 1002, the hidden layer 1003, the feature vector 1004, the n-dimensional vector 1005, the n-dimensional vector 1006, the individual probability 1007, the class information 1008, and the loss 1009.

[0071] Figure 11 is a flowchart of the representative vector generation process performed by the generation unit. The process shown in Figure 11 will be explained below with reference to Figures 9 and 10.

[0072] In S1100, the feature extraction unit 900 initializes the parameters of the fully connected layer 1002 and the hidden layer 1003. The fully connected layer 1002 is a matrix with 512 rows and n columns, the same as the dimensions of the feature vector 1004. The parameters of the fully connected layer 1002 are the elements of the matrix. The feature extraction unit 900 initializes each element of the matrix of the fully connected layer 1002 using random numbers between 0 and 1. The parameters of the hidden layer 1003 are the weight coefficients of the convolutional layer that the hidden layer 1003 possesses. The feature extraction unit 900 initializes the weight coefficients of the hidden layer 1003 using random numbers between 0 and 1.

[0073] In S1101, the feature extraction unit 900 extracts features from the image 1001 using the intermediate layer 1003 and obtains a 512-dimensional feature vector 1004.

[0074] In S1102, the logit calculation unit 901 calculates an n-dimensional vector 1006 consisting of n logits. First, the logit calculation unit 901 calculates an n-dimensional vector 1005 using the following equation 1. Let the n-dimensional vector be V.

[0075]

number

[0076] Here, W' T This is a matrix in which each column vector w1, w2, ..., wn in matrix W is normalized so that the vector length of each vector is 1. Also, x' i This is a vector normalized so that the vector length of the 512-dimensional feature vector x is 1. Therefore, w k (k=1, 2, ..., n) and x' i The dot product of these two vectors is cosθ, where θ is the angle between the two vectors. Therefore, equation 1 can be rewritten as equation 2 below.

[0077]

number

[0078] Next, the logit calculation unit 901 calculates an n-dimensional vector 1006 consisting of n logits using the n-dimensional vector 1005 and the class information 1008. Here, the class information 1008 is the class number to which the image i currently being processed belongs. Specifically, the logit calculation unit 901 calculates each logit in the n-dimensional vector 1006 using the following equation 3.

[0079]

number

[0080] However, in Math 3, s is a scalar parameter set empirically by the user. For example, s=32, but s can be any other number. k is the class number to which image i belongs, and can be any number from 1 to n. Also, m is a scalar parameter set empirically by the user, and can be any number from m=0.5, but it can be any other number.

[0081] In S1103, the probability calculation unit 902 calculates the probability of the image being the same person, 1007. Here, the probability of the image being the same person refers to the value obtained by calculating the probability that image i belongs to class C_k using the n-dimensional vector 1006 shown in equation 3. Specifically, the probability calculation unit 902 calculates the probability of the image being the same person, Pi, using the following equation 4.

[0082]

number

[0083] In S1104, the loss calculation unit 903 calculates the loss 1009. Here, the loss calculation unit 903 calculates the loss of all pre-prepared images and calculates the average loss 1009 by averaging the calculated losses. First, if the loss of image i is LOSS_i, the loss calculation unit 903 calculates LOSS_i using the following number 5.

[0084]

number

[0085] Therefore, the loss calculation unit 903 calculates the LOSS using the following equation 6, where N is the number of images prepared in advance.

[0086]

number

[0087] In S1105, the update unit 904 determines whether the loss has converged. If the update unit 904 determines that the loss has not converged (No in S1105), the process proceeds to S1106. On the other hand, if the update unit 904 determines that the loss has converged (Yes in S1105), the process proceeds to S1107.

[0088] For example, if the number of times the loss convergence check in S1105 has been performed is 1, the update unit 904 determines that the loss has not converged.

[0089] Furthermore, if the number of times the loss convergence determination in S1105 has been made is the second or later, the update unit 904 determines that the loss has converged if the magnitude of the loss calculated in S1104 is greater than the magnitude of the loss calculated in the previous step.

[0090] On the other hand, the update unit 904 determines that the loss has not converged if the magnitude of the loss is smaller than the magnitude of the loss calculated previously.

[0091] However, the method for determining whether the loss has converged is not limited to the above; it may also be determined based on the ratio of the current loss to the previous loss. Alternatively, the method for determining whether the loss has converged may be based on whether the loss is smaller than an empirically defined threshold, or any other method.

[0092] In S1106, the update unit 904 updates the parameters of the fully connected layer 1002 and the intermediate layer 1003, and the process returns to S1101.

[0093] For example, the update unit 904 sets the learning rate ε to, for example, ε = 0.01. Next, the update unit 904 uses the loss 1009 obtained in equation 6 to update the parameters of the fully connected layer 1002 and the hidden layer 1003 by backpropagation.

[0094] In S1107, the update unit 904 calculates representative vectors corresponding to each class, and the process ends. Specifically, the update unit 904 extracts data from the matrix W of the fully connected layer 1002 column by column, and uses the vector of the kth column as the representative vector for the class with class number k. The update unit 904 then calculates each column vector w1, w2, ...w in the matrix W of the fully connected layer 1002. n Regarding this, let w1 be the representative vector for the class with class number 1, and let w2 be the representative vector for the class with class number 2. This allows the generation unit 302 to generate representative vectors corresponding to each class.

[0095] In this embodiment, based on the representative vectors that represent the characteristics of each calculated class, it is determined whether or not each class contains images that do not depict a specific person. The determination unit 303 makes the determination for each class in the following three ways. Method A: Classes with a large norm on the representative vector are identified as noise candidates. Method B: Classes with small norms in their representative vectors are identified as noise candidates. Method C: Classes with high similarity between their representative vectors are identified as noise candidates.

[0096] (Method A) Figure 12 shows the classification results for each class when the threshold NA = 9.0.

[0097] As a result of updating the parameters to reduce loss, in a class with a large norm of the representative vector (for example, C0), the variation in the features of the image increases. When the variation in the features of the image is large, there may be an image in the class that shows other people different from a specific person. Therefore, the determination unit 303 determines a class with a large norm of the representative vector as a noise candidate in order to detect a class (noise candidate) that includes an image in which two or more other people are shown.

[0098] First, the determination unit 303 calculates the L2 norm of the representative vector corresponding to each class number. Next, the determination unit 303 determines whether each class is a noise candidate based on whether the L2 norm of each class is equal to or greater than a preset threshold value NA. Here, the threshold value NA is set to 9.0. The determination result of each class is represented by "YES" or "NO" in the column of "Noise candidate?". Here, a noise candidate refers to a class that includes an image in which a specific person is not shown. "YES" indicates that the class includes an image in which a specific person is shown. "NO" indicates that the class includes an image in which a specific person is not shown.

[0099] Specifically, when the L2 norm (= 10.5) of the representative vector of class number C0 is equal to or greater than the threshold value NA (= 9.0), the determination unit 303 determines that class number C0 is a noise candidate. Also, when the L2 norm (= 9.8) of the representative vector of class number C1 is equal to or greater than the threshold value NA (= 9.0), the determination unit 303 determines that class number C1 is a noise candidate. Further, when the L2 norm (= 1.9) of the representative vector of class number C k is not equal to or greater than the threshold value NA (= 9.0), the determination unit 303 determines that class number C k is not a noise candidate. Finally, when the L2 norm (= 2.2) of the representative vector of class number C n-1 is not equal to or greater than the threshold value NA (= 9.0), the determination unit 303 determines that class number C n-1 is not a noise candidate.

[0100] (Method B) Figure 13 shows the classification results for each class when the threshold NB = 3.0.

[0101] As a result of updating the parameters to minimize the loss, the class with a small norm for the representative vector (for example, C) k In this case, the variation in image features is small. When the variation in image features is small, the diversity of image features is poor. Therefore, in classes with a small norm of the representative vector, there is a possibility that there are face images of dolls or stone statues that do not change expression. So, in order to detect classes that contain face images other than human faces (for example, doll faces and stone statue faces), the determination unit 303 determines classes with a small norm of the representative vector as noise candidates.

[0102] First, the determination unit 303 calculates the L2 norm of the representative vector corresponding to each class number. Next, the determination unit 303 determines whether each class is a noise candidate based on whether the L2 norm of each class is less than or equal to a predetermined threshold NB. Here, the threshold NB is set to 3.0. The determination result for each class is expressed as "YES" or "NO" in the "Noise Candidate?" column. Here, a noise candidate refers to a class that contains images that do not include a specific person. "YES" indicates that the class contains images that include a specific person. "NO" indicates that the class contains images that do not include a specific person.

[0103] Specifically, the determination unit 303 determines that class number C0 is not a noise candidate if the L2 norm (=10.5) of the representative vector of class number C0 is not less than or equal to the threshold NB (=3.0). Similarly, the determination unit 303 determines that class number C1 is not a noise candidate if the L2 norm (=9.8) of the representative vector of class number C1 is not less than or equal to the threshold NB (=3.0). Furthermore, the determination unit 303 determines that class number C k If the L2 norm (=1.9) of the representative vector is less than or equal to the threshold NB (=3.0), then the class number C k The determination unit 303 determines that it is a noise candidate. Finally, the determination unit 303 determines that class number C n-1If the L2 norm (=2.2) of the representative vector is less than or equal to the threshold NB (=3.0), then the class number C n-1 This is determined to be a candidate for noise.

[0104] (Method C) Figure 14 shows the classification results for each class when the threshold SC = 0.7.

[0105] The determination unit 303 determines pairs of classes as noise candidates based on whether the similarity between the representative vectors of the two classes is above a threshold, in order to detect classes in which images of a specific person exist among two or more different classes.

[0106] First, the determination unit 303 calculates the L2 norm of the representative vector for each class. Next, the determination unit 303 determines two classes as noise candidates based on whether the similarity between the representative vectors of the two classes is greater than or equal to a predetermined threshold SC. Here, the determination unit 303 calculates the similarity between the representative vectors of the two classes using the cosine value of the angle between the two representative vectors. Here, the threshold SC is set to 0.7. The determination result for each class is expressed as "YES" or "NO" in the "Noise Candidate?" column. Here, a noise candidate refers to a class that includes images that do not contain a specific person. "YES" indicates that the class includes images that contain a specific person. "NO" indicates that the class includes images that do not contain a specific person.

[0107] Specifically, the determination unit 303 determines that the class number pair (C0, C1) is not a noise candidate if the similarity (=-0.8) of the class number pair (C0, C1) is not equal to or greater than the threshold SC (=0.7). m-2 ,C m-1 If the similarity of the two (=0.8) is greater than or equal to the threshold SC (=0.7), then the pair of class numbers (C m-2 ,C m-1 The determination unit 303 determines that the pair of class numbers (C n-2 ,C n-1If the similarity of the pair (=-0.6) is not equal to or greater than the threshold SC (=0.7), the class number pair (C n-2 ,C n-1 ) is determined not to be a noise candidate.

[0108] Figure 15 shows the image corresponding to class number C1 in Figure 6.

[0109] Based on the classification results for each class, the display unit 304 displays images belonging to the class of noise candidates on the screen of the display unit 202. The user checks the images displayed on the screen of the display unit 304 and makes various inputs.

[0110] Screen 1500 is a user interface displayed on the display unit 202. Screen 1500 includes buttons 1502 and 1503. The user views a group of images 1501 belonging to a class of noise candidates and selects either button 1502 or button 1503.

[0111] When button 1502 is pressed by the user, the display unit 304 registers that the displayed class (in this case, class number C1) is a class that includes images that do not depict a specific person. The display unit 304 then stores the registration information in the storage unit 207.

[0112] When button 1503 is pressed by the user, the display unit 304 registers that the displayed class (in this case, class number C1) does not contain an image that does not include a specific person. The display unit 304 then registers this registration information in the storage unit 207.

[0113] Figure 16 is a flowchart illustrating an example of image processing according to the first embodiment. The image processing flow in Figure 16 will be explained with reference to Figures 2 to 15.

[0114] In S1601, the acquisition unit 300 collects images from the storage unit 207, etc.

[0115] In S1602, the classification unit 301 performs class classification of the collected image group 400 using a DNN.

[0116] In S1603, the determination unit 303 determines whether each class is a candidate for noise based on the classification results of the image group 400. Based on the classification results, the determination unit 303 displays the images belonging to the class that is a candidate for noise on the display unit 202.

[0117] In S1604, the display unit 304 displays a message on the user interface asking whether or not to perform a detailed analysis to improve the estimation accuracy of the noise candidate class. If the display unit 304 determines that no further work should be done on the noise candidate class (No in S1604), the process ends. If the display unit 304 determines that further work should be done on the noise candidate class (Yes in S1604), the process proceeds to S1605.

[0118] In S1605, the generation unit 302 calculates representative vectors that represent the characteristics of each class based on the image group 400 and the classification results of the image group 400.

[0119] In S1606, the determination unit 303 uses the representative vector of each class to determine whether or not each class is a noise candidate.

[0120] In S1607, the display unit 304 displays the group of images belonging to the class that is a noise candidate on the display unit 202, based on the determination result obtained by the determination unit 303 to determine whether or not each class is a noise candidate.

[0121] In S1608, the user checks the group of images belonging to the noise candidates displayed on the user interface and determines whether the class contains any images that constitute noise (images that do not contain a specific person). After the display unit 304 receives input from the user regarding the class determination result, processing ends.

[0122] Generally, when automatically collecting and clustering a large number of images, the following issues may occur. For example, images that do not contain the desired person may be included in a class, images from another class may be included in one class, or images of the same object or person may exist in different classes. This can lead to problems such as an enormous amount of unnecessary data and the printing of inappropriate images when printing images within a class. Therefore, candidate classes containing noise (images that do not contain a specific person) are extracted, and these extracted candidate classes are displayed in the user interface. Furthermore, the user can check the group of images of the classes displayed in the user interface and determine whether or not the class contains noisy images (images that do not contain a specific person).

[0123] According to the first embodiment, it is possible to detect classes from each class that include images that do not contain a specific subject.

[0124] (modified version) In the first embodiment, the user determines whether a class displayed on the screen (shown in Figure 15) is a class containing noise (images that do not show a specific person) and presses button 1502 or button 1503. In this determination method, noise determination of a class is limited to determination on a class basis. However, a class may contain both images that do not contain noise (i.e., images that show a specific person) and images that contain noise (images that do not show a specific person). Therefore, the user may further determine whether each image within the class contains noise. Specifically, the user uses the user interface (shown in Figures 17 and 18) to determine whether each image within the class contains noise and selects the image that contains noise from among the images.

[0125] Figure 17 shows the image corresponding to class number C1 in Figure 6.

[0126] Screen 1700 is a user interface displayed on the display unit 202. Screen 1700 includes buttons 1702, 1703, and 1704. The user views a group of images 1701 belonging to a class of noise candidates and selects one of buttons 1702 to 1704.

[0127] When button 1702 is pressed by the user, the display unit 304 registers that the displayed class is a class containing noise (images that do not show a specific person). The display unit 304 then registers the class registration information in the storage unit 207.

[0128] When button 1703 is pressed by the user, the display unit 304 registers that the displayed class does not contain noise (images that do not show a specific person). The display unit 304 then registers the class registration information in the storage unit 207.

[0129] When button 1704 is pressed by the user, the display unit 304 displays the user interface shown in Figure 18.

[0130] Figure 18 shows the image corresponding to class number C1 in Figure 6.

[0131] Screen 1800 is the user interface displayed on the display unit 202. The user views each image in the group 1801 belonging to the class of noise candidates and selects the checkbox located below the image that is determined to be noise.

[0132] The user selects the checkbox located below the image they determined to be noisy, and then selects button 1802 or 1803.

[0133] When button 1802 is pressed by the user, the display unit 304 registers that the image corresponding to the selected checkbox is an image containing noise. The display unit 304 stores the registration information of the image containing noise in the storage unit 207.

[0134] When button 1803 is pressed by the user, the display unit 304 does not register that the image corresponding to the selected checkbox is a noisy image, but instead displays the user interface shown in Figure 17 again.

[0135] As described above, it is possible to determine not only whether each class contains noise, but also whether each image within a class contains noise.

[0136] (Second Embodiment) The first embodiment detects classes containing noise from each class obtained by classifying a group of images captured by a user with an imaging device (e.g., a digital camera). The second embodiment removes noisy images from training data for a DNN used in a system for authenticating people using facial images (hereinafter referred to as the facial recognition system). The second embodiment detects noisy classes from each class obtained by classifying a group of images collected by web crawling. Therefore, the second embodiment is considered to have a higher proportion of noisy images in each class compared to the first embodiment. Accordingly, the second embodiment detects more noise from classes by repeatedly calculating a representative vector for each class and removing noisy images from the classes containing noise. The differences between the second embodiment and the first embodiment will be explained below.

[0137] Figure 19 is a flowchart illustrating the image processing flow according to the second embodiment.

[0138] In S1900, the acquisition unit 300 collects training images for the DNN using a web crawling method. Specifically, the acquisition unit 300 creates a list of people's names expressed in various languages. Next, the acquisition unit 300 collects a set of images by performing a web search using each of the names listed.

[0139] In S1901, the classification unit 301 performs class classification of the image group. The classification unit 301 classifies the collected image group into a single class by searching using a single name.

[0140] In S1902, the generation unit 302 calculates the representative vector for each class.

[0141] In S1903, the determination unit 303 determines whether each class is a class that contains noise (images that do not show a specific person).

[0142] In S1904, the display unit 304 displays images belonging to the class candidate containing noise on the display unit 202, based on the determination results of each class by the determination unit 303.

[0143] In S1905, the user checks the image on the user interface of the display unit 202 and inputs the result of determining whether or not a noise-containing image is included in the candidate class containing noise into the user interface. Alternatively, the user may input the result of determining whether or not each image in the class contains noise using the user interface (shown in Figure 18).

[0144] In S1906, the display unit 304 determines, based on user instructions, whether to continue the process of determining which class an image is a candidate for noise. If the display unit 304 determines to continue the class determination process (Yes in S1906), the process returns to S1902. At this time, the display unit 304 removes the image determined to be noise from the group of images in that class. If the display unit 304 determines to continue the class determination process (No in S1906), the process terminates.

[0145] According to the second embodiment, by repeatedly calculating the representative vector for each class and removing images within the class, noisy images can be efficiently removed from each class.

[0146] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0147] The disclosures herein include the following image processing apparatus, image processing methods, and programs. (Item 1) A generation means that generates characteristics of image groups belonging to a class, based on classification information that classifies each of several images into one of several classes according to the subject in the image, A determination means that determines, using the characteristics of each class, the possibility that a group of images belonging to a class may include images of subjects other than the subject corresponding to the class, An image processing apparatus characterized by comprising: (Item 2) The generation means trains a model that estimates the classification information for the images from the images, and uses the parameters of the trained model to generate features of the image group belonging to the class. The image processing apparatus according to item 1, characterized in that... (Claim 3) The model comprises an intermediate layer that extracts image features from the image, and a fully connected layer that estimates the classification information from the image features. The generation means generates features of the image group belonging to the class using the parameters of the fully connected layer after training. The image processing apparatus according to item 2, characterized in that (Item 4) The model outputs the likelihood that the image belongs to each of the classes. The generation means generates features of a group of images belonging to a specific class using the parameters of the fully connected layer used to obtain the likelihood that the images belong to the specific class. The image processing apparatus according to item 3, characterized in that (Item 5) The model outputs the likelihood that the image belongs to the specific class based on the inner product of the features of the image and the parameters of the fully connected layer corresponding to the specific class. The image processing apparatus according to item 4, characterized in that (Item 6) The system includes an update means that updates the parameters of the intermediate layer and the fully connected layer based on the difference between the likelihood that the image belongs to the specific class and the classification information. The image processing apparatus according to item 4 or 5, characterized by the features described herein. (Item 7) The determination means determines, based on the magnitude of the norm of the vector representing the features of the image group belonging to the specific class, whether the image group belonging to the specific class contains images of subjects other than those corresponding to the subject of the specific class. An image processing apparatus according to any one of items 1 to 6, characterized in that (Item 8) The determination means determines, based on whether the similarity between a vector representing the features of an image group of one of the plurality of classes and a vector representing the features of an image group of the other class exceeds a threshold, whether the image groups belonging to one class and the other class contain images of subjects other than those corresponding to the subjects of the one class and the other class. An image processing apparatus according to any one of items 1 to 7, characterized in that (Item 9) The determination means controls the output of a group of images belonging to the class that has been determined to include images of subjects other than the subject corresponding to the class. An image processing apparatus according to any one of items 1 to 8, characterized by the above. (Item 10) The system includes a user interface that displays a group of images belonging to a class that the determination means has determined to include images of subjects other than the subject corresponding to the class, and accepts processing to be performed on the class or the group of images. An image processing apparatus according to any one of items 1 to 9, characterized in that (Item 11) The determination means controls the output of some of the images belonging to the class based on the processing received by the user interface. The image processing apparatus according to item 10, characterized in that (Item 12) The determination means obtains user input specifying an image from the group of images displayed on the user interface that contains a subject other than the subject corresponding to the class. An image processing apparatus according to item 10 or 11, characterized by the features described herein. (Item 13) The generation means repeatedly generates features of image groups belonging to the class for each class, the determination means repeatedly determines the possibility, and removes images that depict subjects other than the subject corresponding to the class, or image groups belonging to the class that are determined to contain the image. An image processing apparatus according to any one of items 1 to 12, characterized in that (Item 14) The acquisition means for acquiring the aforementioned multiple images, A classification means for classifying each of the aforementioned multiple images into one of several classes based on the characteristics of each of the aforementioned multiple images, An image processing apparatus according to any one of items 1 to 13, characterized by comprising: (Item 15) The subject in question is a person's face. An image processing apparatus according to any one of items 1 to 14, characterized in that (Item 16) The subject is a person, and each of the multiple classes corresponds to a different person. An image processing apparatus according to any one of items 1 to 15, characterized by the features described herein. (Item 17) An image processing method performed by an image processing device, The image processing apparatus generation means generates characteristics of image groups belonging to a class, based on classification information that classifies each of a plurality of images into one of a plurality of classes according to the subject depicted in the image, The determination means of the image processing apparatus includes a determination step of determining, using the characteristics of each class, the possibility that the group of images belonging to the class includes images of subjects other than the subject corresponding to the class, An image processing method characterized by comprising: (Item 18) A program for causing a computer to function as one of the means of an image processing apparatus as described in any one of items 1 through 16.

[0148] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0149] 10: Image processing system, 100: Imaging device, 110: Image processing device, 120: Network, 201: Input unit, 202: Display unit, 203: I / F, 204: CPU, 205: RAM, 206: ROM, 207: Storage unit, 208: Data bus

Claims

1. A classification means for classifying each of multiple images into one of several classes according to the subject matter depicted in the image, A generation means for generating features for each of the image groups belonging to each of the aforementioned multiple classes, A determination means that determines, using the characteristics of the image group for each class, the possibility that the image group belonging to the class includes images of subjects other than the subject corresponding to the class, Equipped with, The generation means is an image processing device characterized in that the parameters of a model learned using the image and the class to which the image belongs as input are used as features of the image group for each class.

2. The image processing apparatus according to claim 1, further comprising a learning means for training a model that estimates the class to which an image belongs from the image.

3. The model comprises an intermediate layer that extracts features from the image, and a fully connected layer that estimates information about the class to which the image belongs based on the features of the image. The generation means generates features of the image group belonging to the class using the parameters of the fully connected layer after training. The image processing apparatus according to claim 2.

4. The model outputs the likelihood that each of the multiple images belongs to each of the classes. The generation means generates features of a group of images belonging to a specific class using the parameters of the fully connected layer used to obtain the likelihood that each of the plurality of images belongs to the specific class. The image processing apparatus according to claim 3.

5. The model outputs the likelihood that each of the multiple images belongs to the specific class, based on the inner product of the features of each of the multiple images and the parameters of the fully connected layer corresponding to the specific class. The image processing apparatus according to feature 4.

6. The system includes an update means for updating the parameters of the intermediate layer and the fully connected layer based on the difference between the likelihood that each of the plurality of images belongs to the specific class and the information of the class to which the image belongs. The image processing apparatus according to feature 4.

7. The determination means determines, based on the magnitude of the norm of the vector representing the features of the image group belonging to the specific class, whether the image group belonging to the specific class contains images of subjects other than those corresponding to the subject of the specific class. The image processing apparatus according to feature 4.

8. The determination means determines, based on whether the similarity between a vector representing the features of an image group of one of the plurality of classes and a vector representing the features of an image group of the other class exceeds a threshold, whether the image groups belonging to one class and the other class contain images of subjects other than those corresponding to the subjects of the one class and the other class. The image processing apparatus according to feature 1.

9. The determination means controls the output of a group of images belonging to the class that has been determined to include images of subjects other than the subject corresponding to the class. The image processing apparatus according to feature 1.

10. The system includes a user interface that displays a group of images belonging to a class that the determination means has determined to include images of subjects other than the subject corresponding to the class, and accepts processing to be performed on the class or the group of images. The image processing apparatus according to feature 1.

11. The determination means controls the output of some of the images belonging to the class based on the processing received by the user interface. The image processing apparatus according to feature 10.

12. The determination means obtains user input specifying an image from the group of images displayed on the user interface that contains a subject other than the subject corresponding to the class. The image processing apparatus according to feature 10.

13. The generation means repeatedly generates features of image groups belonging to the class for each class, the determination means repeatedly determines the possibility, and removes images that depict subjects other than the subject corresponding to the class, or image groups belonging to the class that are determined to contain the image. The image processing apparatus according to feature 1.

14. The system includes means for acquiring the plurality of images, The image processing apparatus according to claim 1, characterized in that the classification means classifies each of the multiple images into one of a plurality of classes based on the characteristics of each of the plurality of images.

15. The subject in question is a person's face. The image processing apparatus according to feature 1.

16. The subject is a person, and each of the multiple classes corresponds to a different person. The image processing apparatus according to feature 1.

17. An image processing method performed by an image processing device, The image processing apparatus generation means includes a classification step of classifying each of the multiple images into one of the multiple classes according to the subject depicted in the image, A generation step of generating features for each of the image groups belonging to each of the aforementioned multiple classes, The determination means of the image processing apparatus includes a determination step of determining, using the characteristics of the image group for each class, the possibility that the image group belonging to the class includes images of subjects other than the subject corresponding to the class, Equipped with, The generation step is an image processing method characterized in that the parameters of a model learned using the image and the class to which the image belongs as input are used as features of the image group for each class.

18. A program for causing a computer to function as each of the means of the image processing apparatus described in claim 1.

Images