Data processing device and method

Abstract

To provide a data processing apparatus and method for specifying an error in class information of a dataset.SOLUTION: An image classification dataset includes: a plurality of object images each of which includes image data of an object; and class information for specifying a class of an object in each of the object images. On the basis of feature quantities FT of m pieces of object images included in the plurality of object images and class information for the m pieces of object images, the m pieces of object images are subjected to clustering. With respect to each cluster of one or more clusters (CTB1, CTB2) obtained by the clustering, centroids (GB1, GB2) of a plurality of feature points belonging to the cluster and distances d between the centroids and respective feature quantities, and an average and a variance of the plurality of distances derived for the plurality of feature quantities are derived. On the basis of the average and the variance as derived results, a cluster to which an object image having an error in class information belongs is specified as an erroneous cluster.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] The present invention relates to a data processing apparatus and method. [Background technology]

[0002] In order to build an inference model that performs image recognition (such as object detection or object classification) by supervised learning, which is a type of machine learning, a large number of images (thousands to millions of images, or even more) to which annotation information is added are generally required. The annotation information is created, for example, by manual annotation by humans or automatic annotation by a trained model. Supervised learning can be performed using a dataset that includes image data and annotation information of a large number of images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-515189 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in both manual and automatic annotation, objects in an image may be annotated with incorrect class information (labels). Manual (human eye) checking of each image can identify incorrect class information (labels), but such checking incurs huge costs (time and human effort). If the device could automatically extract data with incorrect class information, it would be beneficial because it would reduce the cost of searching for defects in a dataset.

[0005] An object of the present invention is to provide a data processing apparatus and method that contribute to identifying errors in class information. [Means for solving the problem]

[0006] A data processing device according to the present invention has a configuration (first configuration) comprising: an image classification dataset acquisition unit that acquires an image classification dataset including a plurality of object images, each of which includes image data of an object, and class information that identifies a class of an object in each object image; a feature derivation unit that derives features of each object image in the image classification dataset; a clustering processing unit that performs clustering of m object images (m is an integer equal to or greater than 3) based on the features of m object images included in the plurality of object images and the class information for the m object images; an index derivation unit that derives, for each cluster obtained by the clustering, a center of gravity of a plurality of feature points belonging to the cluster and a distance between the center of gravity and each feature, and derives an average and a variance of the plurality of distances derived for the plurality of feature amounts; and an error identification unit that identifies, as an erroneous cluster, a cluster to which an object image having an error in the class information belongs, based on the results of derivation of the average and the variance.

[0007] In the data processing device according to the first configuration, the error identification unit may be configured (second configuration) to identify, as the error cluster, a cluster whose derived mean is equal to or greater than a predetermined first threshold and whose derived variance is equal to or greater than a predetermined second threshold.

[0008] In the data processing device according to the first or second configuration, the clustering processing unit may be configured to classify the m object images into k clusters in the clustering, where k is equal to the number of object classes in the m object images represented by the class information for the m object images (third configuration).

[0009] In the data processing device according to any one of the above first to third configurations, the error identifying unit may be configured to perform reclustering of n object images belonging to the error cluster based on n feature amounts for the n object images, and identify an object image having an error in the class information from among the n object images based on a result of the reclustering (n is an integer equal to or greater than 3) (fourth configuration).

[0010] In the data processing device according to the fourth configuration, the error identification unit may be configured to classify the n object images into first and second clusters by the reclustering, and, when the total number of object images belonging to the second cluster is smaller than the total number of object images belonging to the first cluster, identify the object image belonging to the second cluster as an object image having an error in the class information (fifth configuration).

[0011] In the data processing device according to the fourth configuration, the error identification unit may be configured to classify the n object images into first and second clusters by the reclustering, and, when the distance between the center of gravity of the n features for the n object images and the second cluster is greater than the distance between the center of gravity of the n features for the n object images and the first cluster, identify an object image belonging to the second cluster as an object image having an error in the class information (sixth configuration).

[0012] In a data processing device according to any one of the above fourth to sixth configurations, the plurality of object images are images cut out from a plurality of input images, and the data processing device is provided with a training dataset generation unit that uses the plurality of input images as a plurality of training images and generates a training dataset including image data of the plurality of training images and annotation information identifying a position and a class of an object in each training image, and the training dataset generation unit may be configured to extract, as an annotation target, an input image including an object image having an error in the class information from the plurality of input images based on a result of identification by the error identification unit, and correct the annotation information for the annotation target using information provided from outside for the annotation target (seventh configuration).

[0013] In a data processing device according to any of the above fourth to sixth configurations, the plurality of object images are images cut out from a plurality of input images, and the data processing device is provided with a training dataset generation unit that extracts a portion of the plurality of input images as a plurality of training images and generates a training dataset including image data of the plurality of training images and annotation information that identifies the position and class of an object in each training image, and the training dataset generation unit may be configured to exclude, based on a result of identification by the error identification unit, an input image including an object image having an error in the class information from the plurality of training images (eighth configuration).

[0014] In a data processing device according to any one of the above first to eighth configurations, the data processing device may further include a machine learning processing unit that creates an inference model for object classification by performing machine learning on a neural network using the image classification dataset, and the feature derivation unit may be configured to derive features of each object image by inputting the image classification dataset to the inference model (ninth configuration).

[0015] A data processing method according to the present invention has an image classification dataset acquisition step of acquiring an image classification dataset including a plurality of object images each including image data of an object and class information identifying a class of an object in each object image; a feature derivation step of deriving feature amounts for each object image in the image classification dataset; a clustering processing step of clustering the m object images based on the feature amounts of the m object images included in the plurality of object images and the class information for the m object images (m is an integer equal to or greater than 3); an index derivation step of deriving, for each of one or more clusters obtained by the clustering, a center of gravity of a plurality of feature points belonging to the cluster and a distance between the center of gravity and each feature amount, and deriving an average and a variance of the plurality of distances derived for the plurality of feature amounts; and an error identification step of identifying, as an erroneous cluster, a cluster to which an object image having an error in the class information belongs based on the results of derivation of the average and the variance. Effect of the Invention

[0016] According to the present invention, it is possible to provide a data processing device and method that contribute to identifying errors in class information. [Brief description of the drawings]

[0017] [Figure 1] 1 shows a configuration diagram of a data processing device according to an embodiment of the present invention. [Diagram 2] FIG. 2 illustrates a number of input images contemplated in an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram showing how a plurality of object images are cut out from one input image according to the embodiment of the present invention. [Figure 4] FIG. 2 illustrates multiple object images contemplated in an embodiment of the present invention. [Diagram 5] FIG. 1 is a diagram showing an evaluation dataset (ES1) including a plurality of object images and corresponding class information according to a first example belonging to an embodiment of the present invention. [Figure 6]FIG. 6 is a diagram showing a distribution of a plurality of feature points derived for the plurality of object images of FIG. 5 according to a first example belonging to an embodiment of the present invention. [Figure 7] FIG. 11 is a diagram showing an evaluation dataset (ES2) including a plurality of object images and corresponding class information according to a first example belonging to an embodiment of the present invention. [Figure 8] FIG. 8 is a diagram showing a distribution of a plurality of feature points derived for the plurality of object images of FIG. 7 according to a first example belonging to an embodiment of the present invention. [Figure 9] FIG. 11 is a diagram showing an evaluation dataset (ES3) including a plurality of object images and corresponding class information according to a first example belonging to an embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing a distribution of a plurality of feature points derived for the plurality of object images in FIG. 9 according to a first example belonging to an embodiment of the present invention. [Figure 11] FIG. 8 is a diagram showing a result of reclustering performed on the evaluation dataset of FIG. 7 according to a second example belonging to an embodiment of the present invention. [Figure 12] FIG. 10 is a diagram showing a result of reclustering performed on the evaluation dataset of FIG. 9 according to a second example belonging to an embodiment of the present invention. [Figure 13] 13 is a partial flowchart showing the operation of a data processing device according to a fourth embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, examples of the embodiments of the present invention will be described in detail with reference to the drawings. In each of the drawings, the same parts are given the same reference numerals, and duplicated descriptions of the same parts are omitted as a general rule. In this specification, for the sake of simplicity, a symbol or a code referring to information, a signal, a physical quantity, a member, or the like may be written, and the name of the information, signal, physical quantity, member, or the like corresponding to the symbol or code may be omitted or abbreviated. For example, the image classification dataset acquisition unit referred to by "10" described below (see FIG. 1) may be written as the image classification dataset acquisition unit 10 or may be abbreviated as the acquisition unit 10, but they all refer to the same thing.

[0019] 1 shows a configuration diagram of a data processing device 1 according to this embodiment. The data processing device 1 includes an image classification dataset acquisition unit 10, a machine learning processing unit 20, an inference unit 30, a clustering processing unit 40, an index derivation unit 50, an error identification unit 60, a learning dataset generation unit 70, and a database 80. The data processing device 1 may be configured as a single computer device, or may be configured as multiple physically separated computer devices. The data processing device 1 may also be configured by utilizing so-called cloud computing.

[0020] The image classification dataset acquisition unit 10 acquires an image classification dataset based on a plurality of input images. Here, as shown in FIG. 2, it is assumed that the image classification dataset is acquired based on L input images. L is an arbitrary integer equal to or greater than 2, and has a value of, for example, several thousand to several million. Each input image includes one or more objects. For example, a large number of still images selected from images captured by a camera mounted on a vehicle such as an automobile may be used as the plurality of input images. The i-th input image of the L input images may be referred to by the symbol "II[i]". i represents an arbitrary integer.

[0021] In this embodiment, the inclusion of image data of an object in an image may be expressed as the object being included or present in the image. Similarly, the inclusion of image data of an object in a focused image area (e.g., an object area described below) in an image may be expressed as the object being included or present in the focused image area. In this embodiment, the object refers to a recognition target object that is a target of image recognition by the data processing device 1.

[0022] An image 610 in Fig. 3 is an example of one input image. The input image 610 includes three objects 611 to 613. The objects 611, 612, and 613 are a vehicle, a human being, and a traffic sign, respectively. In this embodiment, the vehicle is mainly assumed to be an automobile that can travel on a road.

[0023] The acquisition unit 10 may include an object detector generated by machine learning. In this case, the acquisition unit 10 can generate an image classification dataset by performing object detection for each input image using the object detector. In the object detection, position identification for identifying the position of an object in an image that is a target of image recognition and class identification for identifying the class (type) of an object in an image that is a target of image recognition are performed.

[0024] Therefore, in object detection for the input image 610, the object detector can set an object region 611B including an object 611, an object region 612B including an object 612, and an object region 613B including an object 613 in the input image 610. The object regions 611B to 613B are set by identifying the positions of the objects 611 to 613 in the input image 610. The acquisition unit 10 generates object images 621 to 623 by cutting out the object regions 611B to 613B from the input image 610. The object region of a certain object is a rectangular region (preferably the smallest rectangular region) that surrounds the image of the object, and is also called a bounding box. In addition, the object detector identifies the class (type) of the object in each object region in class identification.

[0025] There are a plurality of object classes. For the sake of concreteness and clarity of explanation, any object that is focused on as a target of any processing, such as position specification or class identification, may be referred to as a focused object. In object detection, each object in an input image can be a focused object. In class identification, it is determined to which of the above-mentioned plurality of classes the focused object belongs. The class of an object is synonymous with the type of object. Here, for the sake of concreteness of explanation, it is assumed that the plurality of classes include first to third classes, and the first, second, and third classes are a vehicle, a human, and a traffic sign, respectively. In this case, in class identification, it is determined whether the focused object is a vehicle, a human, or a traffic sign, or whether it is an object of a class that does not belong to the first to third classes (for example, a fourth class).

[0026] In class identification of the input image 610, the objects in object regions 611B, 612B, and 613B are identified as vehicles (i.e., first class objects), humans (i.e., second class objects), and traffic signs (i.e., third class objects), respectively (assuming that there are no errors in the class identification).

[0027] By performing object detection for each input image by the object detector, a large number of object images, each including image data of one object, are generated from the input images II[1] to II[L], and class information is assigned to each object image. The class information assigned to a certain object image is information that specifies the class of the object in the object image. Therefore, assuming that there is no error in the class identification, the class information assigned to the object images 621, 622, and 623 respectively indicate the first, second, and third classes. Note that the class information is also called a label. Also, information that indicates the result of the position specification is called object position information. The object position information for a certain input image specifies the position of the object region of each object in the input image (more specifically, the coordinate values ​​of both ends of one diagonal line in the object region). Information including the class information and object position information is called annotation information. In object detection by the object detector, annotation information is generated for each input image.

[0028] As shown in FIG. 4, in the following, it is assumed that a total of H object images OI[1]-OI[H] are generated from input images II[1]-II[L], and the class information assigned to the object image OI[i] is represented by the symbol "CLS[i]". H is an arbitrary integer of 2 or more, and since it is assumed here that one or more object regions are cut out from each input image, "H≧L". In reality, for example, H has a value of several thousand to several million. The image classification dataset includes image data of the object images OI[1]-OI[H] and class information CLS[1]-CLS[H], and in the image classification dataset, the image data of the object image OI[i] and the class information CLS[i] are associated with each other. In addition, extraction source information indicating which input image each object image is based on is shared and recognized by each part in the data processing device 1. Therefore, for example, if the object images OI[1]-OI[5] are cut out from the input image II[1], this is specified in the extraction source information. The source information may be included in the image classification dataset.

[0029] Although the method for generating an image classification dataset in the case where the acquisition unit 10 has an object detector has been described, it is not essential that the acquisition unit 10 has an object detector. For example, the image classification dataset may be generated by inputting annotation information to the data processing device 1 based on a manual operation by a human. That is, for example, an object region may be set for each input image and class information for each object region (in other words, each object image) may be specified based on operation information provided to the data processing device 1 by an administrator of the data processing device 1 through a predetermined man-machine interface (not shown), thereby generating annotation information for each input image and generating and acquiring an image classification dataset. Alternatively, for example, the acquisition unit 10 may receive an input of a dataset created in advance from an external device (not shown) different from the data processing device 1 through wired or wireless communication. The dataset here includes image data of the input images II[1] to II[L] and annotation information for each input image, and the acquisition unit 10 may generate and acquire the image classification dataset based on the input dataset. In the following description, the image classification dataset refers to the image classification dataset acquired by the acquisition unit 10 unless otherwise specified.

[0030] The machine learning processing unit 20 has a neural network 21 (hereinafter referred to as NN21) and executes machine learning processing. In the machine learning processing, the machine learning processing unit 20 creates an inference model that performs a predetermined inference by executing machine learning of the NN21 using an image classification dataset. The machine learning here may be classified as deep learning, and therefore the NN21 may be a deep neural network. In the machine learning, parameters (weights and biases) of the NN21 are appropriately set. The NN21 after the parameters (weights and biases) have been appropriately set corresponds to the inference model.

[0031] The inference unit 30 performs a predetermined inference using an inference model by the NN21. The NN21 is a neural network for image classification, and therefore the inference in the inference model is for image classification. That is, when an object image including an object is given, the inference unit 30 performs class discrimination in inference to identify (discriminate) the class of the object included in the object image. At this time, the inference unit 30 executes a feature derivation process to derive a feature useful for realizing class discrimination from the object image, and identifies (discriminates) the class of the object included in the object image based on the feature. The feature is a multidimensional vector quantity, and may be read as a feature vector. The feature derivation process is executed by a feature derivation unit 31 included in the inference unit 30. An image classification dataset is input to the inference model of the NN21, and inference is performed for each object image in the image classification dataset using the inference model of the NN21. In this inference, a feature derivation process is executed for each object image in the image classification dataset, thereby deriving the feature of each object image.

[0032] The clustering processing unit 40 executes clustering processing based on the result of the feature derivation processing. That is, in the clustering processing, the clustering processing unit 40 performs clustering of a plurality of object images in the image classification dataset based on the feature derived for each object image in the image classification dataset. The clustering processing unit 40 may perform dimensional reduction of the feature derived in the feature derivation processing, and perform clustering based on the feature after dimensional reduction. For example, the dimension reduction can be performed using principal component analysis. The feature shown below refers to the feature itself derived in the feature derivation processing if dimensional reduction is not performed, and refers to the feature after dimensional reduction if dimensional reduction is performed.

[0033] Clustering is performed based on the distance between multiple feature amounts derived for multiple object images in an image classification dataset. The distance between the feature amounts is a distance in a feature space in which the feature amounts are defined, such as Euclidean distance or cosine similarity. By the clustering process, object images having similar feature amounts are classified into a common cluster in the image classification dataset. Hierarchical clustering may be used as a clustering method. However, clustering may also be performed by non-hierarchical clustering (such as k-means).

[0034] Here, the clustering in the clustering processing unit 40 is not performed collectively on all object images included in the image classification data set. In this embodiment, the object images OI[1] to OI[H] are divided into a plurality of groups, and clustering of the object images belonging to each group is performed for each group. In other words, the input images [1] to [L] are divided into a plurality of groups, and clustering of the object images in the input images belonging to each group is performed for each group. Specific examples and technical significance of such clustering will be described later.

[0035] Creating an image classification dataset requires annotation of object positions and object classes in input images. Whether an image classification dataset is created by an object detector or manually, the object classes may be misannotated. For example, an object of interest may be misclassified as a vehicle (i.e., a first-class object) when in fact it is a human (i.e., a second-class object), resulting in the inclusion of incorrect class information in the image classification dataset.

[0036] The index derivation unit 50 derives an index that is useful for identifying such errors, and the error identification unit 60 identifies erroneous data, etc., based on the derived index (described in detail below).

[0037] The training data set generator 70 generates a training data set including image data of a plurality of training images and annotation information that identifies the position and class of an object in each training image.

[0038] For convenience, a dataset including image data of input images II[1]-II[L] and annotation information of each input image handled by the acquisition unit 10 is referred to as a provisional dataset. The provisional dataset is a dataset that is the source of a learning dataset to be generated by the generation unit 70, and if there are no errors in the annotations in the provisional dataset, the provisional dataset itself becomes the learning dataset. However, as described above, a huge number of annotations may contain errors, and the generation unit 70 corrects or removes the errors, and generates the provisional dataset after the correction or removal of the errors as the learning dataset.

[0039] The database 80 is made of a non-volatile recording medium, and stores the learning dataset generated by the generation unit 70 under the control of the generation unit 70. The provisional dataset may also be stored in the database 80. It may be considered that the provisional dataset stored in the database 80 is rewritten or updated into the learning dataset by the generation unit 70. The database 80 may be configured using a magnetic disk, an optical disk, or a semiconductor memory, or a combination of these. The database 80 may be an external recording device provided outside the data processing device 1.

[0040] Below, several specific operation examples, application techniques, modified techniques, etc. related to the data processing device 1 will be described among the multiple embodiments. The matters described above in this embodiment are applied to each of the following embodiments unless otherwise specified and unless there is a contradiction. If there are matters in each embodiment that contradict the matters described above, the description in each embodiment may take precedence. Furthermore, unless there is a contradiction, matters described in any of the multiple embodiments shown below can also be applied to any other embodiment (i.e., any two or more of the multiple embodiments can also be combined).

[0041] <<First Embodiment>> The first embodiment will be described. As described above, the clustering in the clustering processing unit 40 is not performed collectively on all object images included in the image classification data set, but clustering is performed in units of an evaluation data set including a part of all object images included in the image classification data set.

[0042] First, referring to FIG. 5, clustering for an evaluation data set ES1 including image data of object images OI

[11] to OI

[20] which are a part of object images OI[1] to OI[H] will be described. The evaluation data set ES1 includes class information (that is, CLS

[11] to CLS

[20] ; refer to FIG. 4) of the object images OI

[11] to OI

[20] extracted from the image classification data set. The object images OI

[11] to OI

[20] are assumed to be object images cut out from one or more input images II[J A1 to II[J A2 . It is assumed that "0 < J A1 ≦ J A2 < L", and the difference (J A2 - J A1 ) has an integer value of 0 or 1 or more.

[0043] The objects included in each of the object images OI

[11] to OI

[14] are vehicles, and the objects included in each of the object images OI

[15] to OI

[20] are humans. In the image classification data set and the evaluation data set ES1, each class information of the object images OI

[11] to OI

[14] indicates the first class corresponding to the vehicle, and each class information of the object images OI

[15] to OI

[20] indicates the second class corresponding to the human. There is no error in these class information. By executing the feature quantity derivation process for each object image, the feature quantity of each object image is derived. Hereinafter, the feature quantity of the object image OI[i] will be referred to by the symbol "FT[i]".

[0044] Fig. 6 is a diagram showing the feature quantities FT

[11] to FT

[20] corresponding to the object images OI

[11] to OI

[20] plotted in a feature space. In reality, each feature quantity is a multidimensional vector quantity, but in Fig. 6, each feature quantity is shown as if it were a two-dimensional vector quantity. This is the same in other diagrams described later that show the manner in which feature quantities are plotted in a feature space. However, the number of dimensions of the feature quantities may be reduced to "2" by the above-mentioned dimensionality reduction.

[0045] In the following, for any natural number u, the fact that the object image OI[u] is classified into or belongs to a certain cluster of interest may be expressed as the fact that the feature value FT[u] is classified into or belongs to the cluster of interest. In other words, the fact that the object image OI[u] is classified into or belongs to the cluster of interest is equivalent to the fact that the feature value FT[u] is classified into or belongs to the cluster of interest.

[0046] The clustering processing unit 40 classifies the object images OI

[11] to OI

[20] into k clusters based on the class information of the object images OI

[11] to OI

[20] in the image classification dataset (evaluation dataset ES1). A Classify into k clusters (i.e., the number of clusters is k A (Set it to 0 and then perform the classification.) A is the number of types of classes represented by the class information for the object images OI

[11] to OI

[20] in the image classification dataset (i.e., the number of types of objects in the object images OI

[11] to OI

[20] represented by the class information). In other words, k A is equal to the number of types of classes (total number) indicated by the 10 pieces of class information assigned to the object images OI

[11] to OI

[20] . In the example of FIG. 5, the 10 pieces of class information assigned to the object images OI

[11] to OI

[20] indicate two classes, the first class and the second class, so “k A When non-hierarchical clustering is used as the clustering method, the total number of clusters is set to “k A If you set hierarchical clustering as the clustering method, the total number of clusters is set to “k A= 2" and then stop clustering.

[0047] As shown in Figure 6, two clusters CT for features FT

[11] -FT

[20] A1 and CT A2 The features FT

[11] to FT

[14] are set as cluster CT A1 The features FT

[15] -FT

[20] are classified into clusters CT A2 In Figure 6, point G A1 is cluster CT A1 It represents the center of gravity of the features FT

[11] to FT

[14] belonging to point G A2 is cluster CT A2 This represents the center of gravity of the features FT

[15] to FT

[20] that belong to

[0048] The index derivation unit 50 derives the center of gravity of each cluster generated by the clustering processing unit 40, and derives the distance between the center of gravity and each feature that belongs to the cluster. The distance between the center of gravity and the feature, like the distance between two features, is a distance in the feature space in which the feature is defined, and is, for example, Euclidean distance or cosine similarity. The distances derived for the features FT

[11] to FT

[20] are referred to by the symbols "d

[11] to d

[20] ", respectively. The distances d

[11] to d

[14] are respectively relative to the center of gravity G A1 and the features FT

[11] to FT

[14] . The distances d

[15] to d

[20] are the distances A2 represents the distance between and features FT

[15] ~FT

[20] .

[0049] The index derivation unit 50 further derives the average and variance of the distances derived for each cluster generated by the clustering processing unit 40. The average and variance derived by the index derivation unit 50 are hereinafter referred to as the distance average and distance variance. Cluster CT A1 The distance average and distance variance derived for A1 " and "σ 2 A1 Then, the distance average AVE A1is the average of the distances d

[11] ~d

[14] , and the distance variance σ 2 A1 is the variance of the distances d

[11] ~d

[14] . Similarly, the cluster CT A2 The distance average and distance variance derived for A2 " and "σ 2 A2 Then, the distance average AVE A2 is the average of the distances d

[15] ~d

[20] , and the distance variance σ 2 A2 is the variance of the distance d

[15] ~d

[20] .

[0050] The error identifying unit 60 judges whether each cluster is an error cluster based on the distance average and distance variance derived for each cluster by the index derivation unit 50. An error cluster is a cluster to which an object image with an error in class information belongs. An object image with an error in class information is, in other words, an object image to which erroneous class information is assigned, and is hereinafter also referred to as an object image with an error label. Conversely, an object image to which correct class information is assigned is referred to as an object image with a correct label. In addition, hereinafter, a cluster that is not identified as an error cluster by the error identifying unit 60 may be referred to as a correct cluster. It can also be said that the error identifying unit 60 has a function of judging whether each cluster is a correct cluster or an error cluster.

[0051] The error identifying unit 60 compares the distance average and distance variance with predetermined thresholds TH1 and TH2, respectively, for each cluster generated by the clustering processing unit 40, and identifies a cluster that satisfies a predetermined error condition as an error cluster based on the comparison result. When the corresponding distance average is equal to or greater than the predetermined threshold TH1 and the corresponding distance variance is equal to or greater than the predetermined threshold TH2 for a certain cluster, the error condition is satisfied for that cluster.

[0052] The thresholds TH1 and TH2 may be set as follows. That is, a plurality of object images for threshold setting, each of which includes a recognition target object, may be prepared in advance, and the thresholds TH1 and TH2 may be set based on the clustering results for the plurality of object images for threshold setting. In this case, the clustering processing unit 40 classifies the plurality of object images for threshold setting into a plurality of clusters. Next, the index derivation unit 50 derives the average and variance of the plurality of distances for each cluster. Next, for the plurality of object images for threshold setting, a value obtained by adding an appropriate offset value (which may be 0) to the average of the plurality of derived distances is set as the threshold TH1, and a value obtained by adding an appropriate offset value (which may be 0) to the variance of the plurality of derived distances is set as the threshold TH2. Then, the set thresholds TH1 and TH2 may be used in the above comparison with the corresponding clusters (clusters related to the same type of object). The image data of the plurality of object images for threshold setting may be obtained from an existing and arbitrary dataset, and may be, for example, image data included in the Microsoft COCO dataset.

[0053] Cluster CT A1 For the distance average AVE A1 and distance variance σ 2 A1 are compared with thresholds TH1 and TH2, respectively, and the cluster CT A2 For the distance average AVE A2 and distance variance σ 2 A2 are compared with the thresholds TH1 and TH2, respectively. In the examples of Figs. 5 and 6, there is no error in the class information of the object images OI

[10] to OI

[20] in the image classification dataset. Therefore, the cluster CT A1 Distance average AVE A1 and distance variance σ 2 A1 is smaller than the thresholds TH1 and TH2, and the cluster CT A2 Distance average AVE A2 and distance variance σ 2 A2 is also smaller than the thresholds TH1 and TH2. Therefore, the cluster CT A1 and CT A2None of them are identified as error clusters (in other words, they are determined to be correct clusters).

[0054] Next, referring to FIG. 7, clustering for the evaluation dataset ES2 including the image data of the body images OI

[31] to OI

[40] which are part of the object images OI[1] to OI[H] will be described. The evaluation dataset ES2 includes the class information (that is, CLS

[31] to CLS

[40] ; see FIG. 4) of the object images OI

[31] to OI

[40] extracted from the image classification dataset. The object images OI

[31] to OI

[40] are assumed to be object images cut out from one or more input images II[J B1 ~II[J B2 . “J A2 <J B1 ≦J B2 <L”, and the difference (J B2 -J B1 ) has an integer value of 0 or 1 or more.

[0055] The objects included in each of the object images OI

[31] to OI

[34] are vehicles, the objects included in each of the object images OI

[35] to OI

[38] are humans, and the objects included in each of the object images OI

[39] and OI

[40] are traffic signs. In the image classification dataset and the evaluation dataset ES2, the class information of each of the object images OI

[31] to OI

[34] indicates the first class corresponding to vehicles, and the class information of each of the object images OI

[35] to OI

[38] indicates the second class corresponding to humans. There is no error in this class information. However, in the image classification dataset and the evaluation dataset ES2, the class information of the object images OI

[39] and OI

[40] indicates the second class corresponding to humans, so there is an error in the class information of the object images OI

[39] and OI

[40] .

[0056] 8 is a diagram showing a state where feature quantities FT

[31] to FT

[40] corresponding to the object images OI

[31] to OI

[40] are plotted on a feature space. The clustering processing unit 40 classifies the object images OI

[31] to OI

[40] into k clusters based on the class information of the object images OI

[31] to OI

[40] in the image classification dataset (evaluation dataset ES2). B Classify into k clusters (i.e., the number of clusters is k B (Set it to 0 and then perform the classification.) B is the number of types of classes represented by the class information for the object images OI

[31] to OI

[40] in the image classification dataset (i.e., the number of types of objects in the object images OI

[31] to OI

[40] represented by the class information). In other words, k B is equal to the number of types of classes (total number) indicated by the 10 pieces of class information assigned to the object images OI

[31] to OI

[40] . In the example of FIG. 7, the 10 pieces of class information assigned to the object images OI

[31] to OI

[40] should indicate three classes, the first to third classes, but in reality they are mistakenly indicated as two classes, the first and second classes, resulting in “k B When non-hierarchical clustering is used as the clustering method, the total number of clusters is set to “k B If you set hierarchical clustering as the clustering method, the total number of clusters is set to “k B = 2" and then stop clustering.

[0057] As shown in Figure 8, there are two clusters CT for features FT

[31] -FT

[40] . B1 and CT B2 Here, the features FT

[31] to FT

[34] are set as the cluster CT B1 The features FT

[35] -FT

[40] are classified into clusters CT B2 In Figure 8, point G B1 is cluster CT B1 It represents the center of gravity of the features FT

[31] to FT

[34] belonging to point G B2 is cluster CT B2This represents the center of gravity of the features FT

[35] to FT

[40] that belong to

[0058] As described above, the index derivation unit 50 derives the center of gravity of each cluster generated by the clustering processing unit 40, and derives the distance between the center of gravity and each feature value belonging to the cluster. The distances derived for the features FT

[31] to FT

[40] are referred to by the symbols “d

[31] to d

[40] ”, respectively. The distances d

[31] to d

[34] are respectively relative to the center of gravity G B1 and the features FT

[31] to FT

[34] . The distances d

[35] to d

[40] are the distances B2 represents the distance between and features FT

[35] ~FT

[40] .

[0059] The index deriving unit 50 further derives the distance mean and distance variance for each cluster as described above. B1 The distance average and distance variance derived for B1 " and "σ 2 B1 Then, the distance average AVE B1 is the average of the distances d

[31] ~d

[34] , and the distance variance σ 2 B1 is the variance of the distances d

[31] ~d

[34] . Similarly, the cluster CT B2 The distance average and distance variance derived for B2 " and "σ 2 B2 Then, the distance average AVE B2 is the average of the distances d

[35] ~d

[40] , and the distance variance σ 2 B2 is the variance of the distance d

[35] ~d

[40] .

[0060] The error identification unit 60 identifies the cluster CT B1 and CT B2 For each of the clusters, determine whether the error condition is true or false. B1 For the distance average AVE B1 and distance variance σ 2 B1are respectively compared with threshold values TH1 and TH2, and for cluster CT B2 the distance average AVE B2 and the distance variance σ 2 B2 are respectively compared with threshold values TH1 and TH2. In the examples of FIGS. 7 and 8, for cluster CT B1 the distance average AVE B1 and the distance variance σ 2 B1 are respectively smaller than threshold values TH1 and TH2, and as a result, cluster CT B1 is not specified as an error cluster (in other words, it is determined to be a correct cluster). However, for cluster CT B2 the distance average AVE B2 and the distance variance σ 2 B2 are respectively larger than threshold values TH1 and TH2. Therefore, cluster CT B2 satisfies the error condition, and as a result, cluster CT B2 is specified as an error cluster.

[0061] Next, referring to FIG. 9, clustering for evaluation data set ES3 including image data of body images OI

[51] to OI

[60] which are part of object images OI[1] to OI[H] will be described. Evaluation data set ES3 includes class information (i.e., CLS

[51] to CLS

[60] ; see FIG. 4) of object images OI

[51] to OI

[60] extracted from the image classification data set. Object images OI

[51] to OI

[60] are assumed to be object images cut out from one or more input images II[J C1 to II[J C2 . Let it be “J B2 <J C1 ≦J C2 <L”, and the difference (J C2 -J C1 ) has an integer value of 0 or 1 or more.

[0062] The object included in each of the object images OI

[51] to OI

[58] is a vehicle, and the object included in each of the object images OI

[59] and OI

[60] is a human. In the image classification dataset and the evaluation dataset ES3, the class information of each of the object images OI

[51] to OI

[58] indicates the first class corresponding to a vehicle. There is no error in this class information. However, in the image classification dataset and the evaluation dataset ES3, the class information of the object images OI

[59] and OI

[60] also indicates the first class corresponding to a vehicle, and therefore the class information of the object images OI

[59] and OI

[60] contains an error.

[0063] 10 is a diagram showing a state where feature quantities FT

[51] to FT

[60] corresponding to the object images OI

[51] to OI

[60] are plotted in a feature space. The clustering processing unit 40 classifies the object images OI

[51] to OI

[60] into k clusters based on the class information of the object images OI

[51] to OI

[60] in the image classification dataset (evaluation dataset ES3). C Classify into k clusters (i.e., the number of clusters is k C (Set it to 0 and then perform the classification.) C is the number of types of classes represented by the class information for the object images OI

[51] to OI

[60] in the image classification dataset (i.e., the number of types of objects in the object images OI

[51] to OI

[60] represented by the class information). In other words, k C is equal to the number of types of classes (total number) indicated by the 10 pieces of class information assigned to the object images OI

[51] to OI

[60] . In the example of FIG. 9, the 10 pieces of class information assigned to the object images OI

[51] to OI

[60] should indicate two classes, the first and second classes, but the first class is mistakenly indicated as the only class, resulting in “k C = 1”. When non-hierarchical clustering is used as the clustering method, the total number of clusters is “k C If you want to perform hierarchical clustering, you can set the number of clusters to “k C Clustering can be continued until k = 1. C= 1”, no actual clustering calculation is required; it is sufficient to simply set a single cluster that includes all of the features FT

[51] to FT

[60] .

[0064] As shown in Figure 10, one cluster CT is generated for features FT

[51] -FT

[60] . C1 In FIG. 10, point G C1 is a cluster CT C1 This represents the center of gravity of the features FT

[51] to FT

[60] that belong to

[0065] As described above, the index derivation unit 50 derives the center of gravity of each cluster generated by the clustering processing unit 40, and derives the distance between the center of gravity and each feature amount belonging to the cluster. C1 is set, so the cluster CT C1 The distances derived for the features FT

[51] to FT

[60] are referred to as “d

[51] to d

[60] ”, respectively. The distances d

[51] to d

[60] are the distances C1 represents the distance between and features FT

[51] ~FT

[60] .

[0066] The index derivation unit 50 further C1 We derive the distance mean and distance variance for the cluster CT. C1 The distance average and distance variance derived for C1 " and "σ 2 C1 Then, the distance average AVE C1 is the average of the distances d

[51] ~d

[60] , and the distance variance σ 2 C1 is the variance of the distance d

[51] ~d

[60] .

[0067] The error identification unit 60 identifies the cluster CT C1 Determine whether the error condition is true or false for cluster CT. C1 About the distance average AVE C1 and distance variance σ 2C1 are compared with thresholds TH1 and TH2, respectively. In the examples of FIGS. 9 and 10, the cluster CT C1 Distance average AVE C1 and distance variance σ 2 C1 are greater than the thresholds TH1 and TH2, respectively. Therefore, the cluster CT C1 satisfies the error condition, and as a result, the cluster CT C1 is identified as an error cluster.

[0068] Since the features in the correct cluster are truly features for objects of the same class, the variation in the features in the correct cluster is relatively small. In contrast, since the features in the error cluster include features for objects of different classes, the variation in the features in the error cluster is relatively large. Taking these factors into consideration, as described above, the distance average and distance variance are derived for each cluster, and the error cluster is identified based on the derived results. This method makes it possible for the data processing device 1 itself to detect the error cluster well without relying on human eyes, and the cost (time and human effort) of searching for defects in the data set can be reduced compared to the case of relying on human eyes. If the error cluster can be identified, it becomes possible to take measures such as correcting the class information of the object image belonging to the error cluster to generate a learning data set or excluding an input image including the object image from the learning data set, and as a result, it becomes possible to improve the quality of the learning data set. By improving the quality of the learning data set, the performance of an inference model for image recognition constructed by machine learning using the learning data set can be improved.

[0069] <<Second Example>> A second embodiment will be described. The second embodiment is implemented in combination with the above-described first embodiment. In the second embodiment, an error label identification process that can be performed by the error identification unit 60 when an error cluster is identified will be described.

[0070] In the error label identification process, the error identification unit 60 performs clustering to group object images belonging to an error cluster into a first and a second cluster within the error cluster. The clustering in the error label identification process is called reclustering to clearly distinguish it from the above-mentioned clustering performed by the clustering process 40. The reclustering is performed based on the feature amounts of each object image belonging to the error cluster. That is, if n object images belong to a certain error cluster, reclustering is performed based on the n feature amounts derived in the feature derivation process for the n object images. However, this reclustering is performed on the premise that "n ≧ 3" is satisfied. Therefore, the error label identification process can be performed on an error cluster that satisfies "n ≧ 3". The reclustering can be realized by non-hierarchical clustering (such as k-means) with the number of clusters set to 2.

[0071] The error identifying unit 60 involved in the error label identifying process identifies an error-labeled object image (i.e., an object image with an error in class information) from among the n object images based on the result of reclustering. It can also be said that the error identifying unit 60 has a function of determining whether each object image belonging to an error cluster is an object image with a correct label or an object image with an error label. As the error label identifying process, the following first or second error label identifying process can be adopted.

[0072] The first erroneous label identification process will be described with reference to the evaluation data set ES2 corresponding to Figs. 7 and 8. See Fig. 11. For the evaluation set ES2, the cluster CT B2 is identified as an error cluster. Error cluster CT B2 Object images OI

[35] to OI

[40] belong to the error cluster CT. B2 The features FT

[35] to FT

[40] belong to the error cluster CT. B2In this case, n=6. The error identifying unit 60 classifies the object images OI

[35] to OI

[40] into a first and a second cluster by re-clustering the object images OI

[35] to OI

[40] based on the feature quantities FT

[35] to FT

[40] . B2_1 and CT B2_2 One of these corresponds to the first cluster, and the other corresponds to the second cluster. The features FT

[35] to FT

[38] are cluster CT B2_1 The features FT

[39] and FT

[40] are classified into cluster CT B2_2 Point G in Figure 11 is classified as B2 is the center of gravity G in Figure 8 B2 Matches.

[0073] In the first error label identification process, of the first and second clusters formed by reclustering each object image in the error cluster, the object image in the cluster with fewer object images is identified as an error-labeled object image. That is, in the first error label identification process, after forming the first and second clusters by reclustering each object image in the error cluster, if the total number of object images belonging to the second cluster is smaller than the total number of object images belonging to the first cluster, the object image belonging to the second cluster is identified as an error-labeled object image (i.e., an object image with an error in class information), and if the total number of object images belonging to the first cluster is smaller than the total number of object images belonging to the second cluster, the object image belonging to the first cluster is identified as an error-labeled object image.

[0074] In the example of Figure 11, the cluster CT B2_1 The total number of object images belonging to cluster CT is 4. B2_2 The total number of object images belonging to is 2. Therefore, when the first erroneous label identification process is applied to the erroneous clusters of the evaluation data set ES2, the cluster CT B2_2 Each object image belonging to is identified as an erroneously labeled object image.

[0075] The second erroneous label identification process will be described with reference to the evaluation dataset ES2. B2_1 and CT B2_2 The process up to classifying the object images into the first and second error label identification processes is common to both the first and second error label identification processes. In the second error label identification process, of the first and second clusters formed by reclustering each object image in the error cluster, the object image in the cluster farther from the center of gravity of the error cluster is identified as an object image with an error label. That is, in the second error label identification process, the first and second clusters are formed by reclustering each object image in the error cluster. Then, if the distance between the center of gravity of the error cluster and the second cluster is greater than the distance between the center of gravity of the error cluster and the first cluster, the object image belonging to the second cluster is identified as an object image with an error label (i.e., an object image with an error in class information). Conversely, if the distance between the center of gravity of the error cluster and the first cluster is greater than the distance between the center of gravity of the error cluster and the second cluster, the object image belonging to the first cluster is identified as an object image with an error label. The distance between the center of gravity of the error cluster and the first cluster may be the distance between the center of gravity of the error cluster and the center of gravity of all features belonging to the first cluster, and the distance between the center of gravity of the error cluster and the second cluster may be the distance between the center of gravity of the error cluster and the center of gravity of all features belonging to the second cluster.

[0076] In the example of FIG. 11 (see also FIG. 8), the error cluster CT B2 Center of gravity G B2 and Cluster CT B2_2 Distance from (for details, see Cluster CT B2_2 The centroids of the features FT

[39] and FT

[40] and the centroid G B2 The distance between the error clusters CT B2 Center of gravity G B2 and Cluster CT B2_1 Distance from (for details, see Cluster CT B2_1 The centroids of features FT

[35] to FT

[38] and the centroid G B2Therefore, when the second error label identification process is applied to the error clusters in the evaluation data set ES2, the cluster CT B2_2 Each object image belonging to is identified as an erroneously labeled object image.

[0077] The first erroneous label identification process will be described with reference to the evaluation data set ES3 corresponding to Figs. 9 and 10. See Fig. 12. For the evaluation data set ES3, the cluster CT C1 is identified as an error cluster. Error cluster CT C1 Object images OI

[51] to OI

[60] belong to the error cluster CT. C1 The features FT

[51] to FT

[60] belong to the error cluster CT. C1 In this case, n=10. The error identifying unit 60 classifies the object images OI

[51] to OI

[60] into a first and a second cluster by re-clustering the object images OI

[51] to OI

[60] based on the feature quantities FT

[51] to FT

[60] . C1_1 and CT C1_2 One of these corresponds to the first cluster, and the other corresponds to the second cluster. The features FT

[51] to FT

[58] are cluster CT C1_1 The features FT

[59] and FT

[60] are classified into cluster CT C1_2 Point G in Figure 12 C1 is the center of gravity G in Figure 10 C1 Matches.

[0078] In the example of Figure 12, the cluster CT C1_1 The total number of object images belonging to cluster CT is 8. C1_2 The total number of object images belonging to is 2. Therefore, when the first erroneous label identification process is applied to the erroneous clusters of the evaluation data set ES3, the cluster CT C1_2 Each object image belonging to is identified as an erroneously labeled object image.

[0079] In the example of FIG. 12 (see also FIG. 10), the error cluster CT C1 Center of gravity G C1 and Cluster CT C1_2 Distance from (for details, see Cluster CT C1_2 The centroids of the features FT

[59] and FT

[60] and the centroid G C1 The distance between the error clusters CT C1 Center of gravity G C1 and Cluster CT C1_1 Distance from (for details, see Cluster CT C1_1 The centroids of features FT

[51] to FT

[58] and the centroid G C1 Therefore, when the second error label identification process is applied to the error clusters in the evaluation dataset ES3, the cluster CT C1_2 Each object image belonging to is identified as an erroneously labeled object image.

[0080] As shown in this embodiment, if reclustering is performed on the erroneous clusters, the erroneous clusters are expected to be decomposed into a cluster to which the correctly labeled object images belong and a cluster to which the erroneously labeled object images belong. Therefore, by referring to the result of reclustering, it becomes possible to easily identify the erroneously labeled object images.

[0081] More specifically, when the erroneous clusters are re-clustered to form the first and second clusters, it is expected that the object images belonging to one of the first and second clusters are correctly labeled object images and the object images belonging to the other cluster are incorrectly labeled object images. On the other hand, it is expected that most of the annotation information in the image classification dataset is basically correct. Therefore, it is highly likely that the object images in the cluster that has fewer object images among the first and second clusters are incorrectly labeled object images. By using the first erroneous label identification process based on this knowledge, it is possible to accurately identify incorrectly labeled object images.

[0082] In addition, since most of the annotation information in the image classification dataset is expected to be correct, the object image in the cluster farther from the center of gravity of the erroneous cluster out of the first and second clusters formed by reclustering each object image in the erroneous cluster is more likely to be an erroneously labeled object image. By using the second erroneous label identification process based on this knowledge, it is possible to accurately identify erroneously labeled object images.

[0083] <<Third Example>> A third embodiment will be described. The third embodiment is implemented in combination with the first and second embodiments. As described above, the learning dataset generating unit 70 generates a learning dataset including image data of a plurality of learning images and annotation information that identifies the position and class of an object in each learning image. Each of the input images II[1] to II[L] is a candidate for a learning image, and if there is no erroneously labeled object image in a dataset (provisional dataset) including image data of the input images II[1] to II[L] and annotation information of each input image handled by the acquiring unit 10, the provisional dataset becomes the learning dataset as it is. However, in reality, there are often erroneously labeled object images to a certain extent. In the third embodiment, an erroneous label handling process that can be executed by the learning dataset generating unit 70 when an erroneously labeled object image is identified will be described.

[0084] First, attention is paid to the evaluation data set ES1 corresponding to Fig. 5 and Fig. 6. Regarding the evaluation data set ES1, the cluster CT generated by the clustering processing unit 40 is A1 and CT A2No error cluster is identified in any of the evaluation data sets, and therefore, none of the object images OI

[11] to OI

[20] are identified as object images with an error label. The generation unit 70 does not execute the error label handling process for the evaluation data set (ES1 in this case) in which no error-labeled object images are identified. The generation unit 70 regards each object image in the evaluation data set in which no error cluster is identified as an object image with a correct answer label, and sets each input image including an object image with a correct answer label as a learning image. The image data of each input image set as a learning image is included in the learning data set.

[0085] For example, if the object images OI

[11] to OI

[14] are A1 ], and object images OI

[15] to OI

[17] are included in input image II[J A1 +1], and the object images OI

[18] to OI

[20] are included in the input image II[J A1 +2] (here, “J A1 +2=J A2 "), three input images II[J A1 ]~II[J A1 +2] as the three learning images, and input image II[J A1 ]~II[J A1 +2] and the input image II[J A1 ]~II[J A1 +2] will be included as is in the training dataset.

[0086] Next, attention will be given to the evaluation data set ES2 corresponding to Figs. 7, 8, and 11. Regarding the evaluation data set ES2, the cluster CT generated by the clustering processing unit 40 B1 and CT B2 Of these, cluster CT B2Only the first error cluster is identified, and among the object images OI

[31] -

[40] , only the object images OI

[39] and

[40] are identified as erroneously labeled object images. The generating unit 70 executes an erroneous label correspondence process on the evaluation data set (ES2 in this case) in which the erroneously labeled object images have been identified. As the erroneous label correspondence process, the following first erroneous label correspondence process can be adopted.

[0087] Now, the object images OI

[31] ~OI

[34] are the input images II[J B1 ], and object images OI

[35] to OI

[37] are included in input image II[J B1 +2], and the object images OI

[38] to OI

[40] are included in the input image II[J B1 Let us consider the case CASE1 included in [+2] (in this case, “J B1 +2=J B2 ").

[0088] In the first erroneous label handling process for case CASE1, the generation unit 70 generates an input image II[J B1 +2] as the annotation target, and input image II[J B1 The administrator of the data processing device 1 is notified that there is a possibility that an error is included in the annotation information of input image II[J+2] (this notification is referred to as notification NT1). The notification NT1 may be performed through a display on a display screen (not shown) provided in or connected to the data processing device 1. B1 The administrator who received the notification NT1 may request a correct answer for the annotation information of the input image II[J B1 Correct answer information indicating the correctness of the annotation information of the input image II[J+2] is input to the data processing device 1 through a predetermined man-machine interface (not shown) such as a pointing device. The correct answer information here indicates that the objects in the object images OI

[39] and

[40] are traffic signs (i.e., objects of the third class). The generation unit 70 generates a correct answer information indicating the correctness of the annotation information of the input image II[J+2] based on the correct answer information. B1+2] (annotation information in the provisional dataset) and input image II[J B1 The image data of [image 1+2] and the corrected annotation information are included in the learning dataset as a set of image data of one teacher image and annotation information of the teacher image.

[0089] In case CASE1, input image II [J B1 ] and II[J B1 +1] is set to two training images, and the input image II[J B1 ] and II[J B1 +1] and the input image II[J B1 ] and II[J B1 +1] will be included in the training dataset as is.

[0090] The object images OI

[31] to OI

[40] are all from a single input image II[J B1 ] Case CASE2 is also possible (in this case, "J B1 =J B2 ").

[0091] In the first erroneous label handling process for case CASE2, the generation unit 70 generates an input image II[J B1 ] is extracted as an annotation target, and the input image II[J B1 The administrator of the data processing device 1 is notified that there is a possibility that an error is included in the annotation information of the input image II[J] (this notification is referred to as notification NT2). The notification NT2 may be performed through a display on a display screen (not shown) provided in or connected to the data processing device 1. B1 The administrator who received the notification NT2 may request a correct answer for the annotation information of the input image II[J B1Correct answer information indicating the correctness of the annotation information of the input image II[J] is input to the data processing device 1 through a predetermined man-machine interface (not shown) such as a pointing device. The correct answer information here indicates that the objects in the object images OI

[39] and

[40] are traffic signs (i.e., objects of the third class). The generation unit 70 generates a correct answer information indicating the correctness of the annotation information of the input image II[J] based on the correct answer information. B1 ] (annotation information in the provisional dataset) and input image II[J B1 The image data of [ ] and the corrected annotation information are included in the learning dataset as a set of image data of one teacher image and annotation information of the teacher image.

[0092] As the error label handling process, the following second error label handling process may be adopted. In the second error label handling process according to the above-mentioned case CASE1, the generating unit 70 generates an input image II[J B1 +2] is not set as a learning image. That is, input image II[J B1 +2] is excluded from the learning dataset (not included in the learning dataset). When the second error label correspondence process is adopted, in case CASE1, the input image II[J B1 ] and II[J B2 ] is set to two training images, and input image II[J B1 ] and II[J B1 +1] and the input image II[J B1 ] and II[J B1 In the second erroneous label handling process for the above-mentioned case CASE2, the generation unit 70 generates an input image II[J B1 ] is not set as a learning image. That is, input image II[J B1The image data of [ ] is excluded from the learning dataset (is not included in the learning dataset). Note that even when the second erroneous label handling process is adopted, the above notifications NT1 and NT2 may be issued in cases CASE1 and CASE2 (however, the correct answer request notification is not executed).

[0093] Although the first and second erroneous label handling processes have been described with a focus on the evaluation data set ES2, similar processes are also performed on other evaluation data sets including the evaluation data set ES3.

[0094] In this way, the generating unit 70 employing the first error label handling process sets a plurality of input images (II[1] to II[L]) as a plurality of learning images, and generates a learning dataset including image data of the plurality of learning images as the plurality of input images (II[1] to II[L]) and annotation information for identifying the position and class of an object in each learning image. At this time, the generating unit 70 employing the first error label handling process extracts an input image including an erroneously labeled object image as an annotation target based on the identification result (identification result of an erroneous cluster and an erroneously labeled object image) of the error identifying unit 60, and corrects the annotation information for the annotation target using information (correct answer information) provided from outside for the annotation target. The corrected annotation information is included in the learning dataset.

[0095] In contrast, the generating unit 70 that employs the second error label handling process extracts some of the multiple input images (II[1] to II[L]) as multiple training images, and generates a training data set that includes image data of the multiple training images obtained by the extraction and annotation information that identifies the position and class of an object in each training image. At this time, the generating unit 70 that employs the second error label handling process excludes input images that include an erroneously labeled object image from the multiple training images, based on the identification results (identification results of error clusters and erroneously labeled object images) of the error identifying unit 60. For example, if there are input images II[1] to II

[10000] , and none of the input images II[1] to II

[9800] contains an erroneously labeled object image, and each of the input images II

[9801] to II

[10000] contains an erroneously labeled object image, then only the input images II[1] to II

[9800] are extracted as a total of 9800 training images, and a training dataset containing image data of the 9800 training images is generated.

[0096] Whether the first or second label correspondence process is adopted, it is possible to generate a high-quality training dataset, i.e., a training dataset with few errors in annotation information (ideally no errors). By improving the quality of the training dataset, it is possible to improve the performance of an inference model for image recognition constructed by machine learning using the training dataset. The second label correspondence process has the advantage that manual input of correct answer information is not required. On the other hand, the first label correspondence process has the advantage that the number of training images can be increased compared to when the second label correspondence process is adopted.

[0097] <<Fourth Example>> A fourth embodiment will now be described. Fig. 13 shows a partial flowchart of the operation of the data processing device 1. It is assumed that the image classification dataset acquisition process, the machine learning process, and the feature derivation process have been executed before reaching step S11. However, the feature derivation process can be executed at any time before reaching step S12 described later.

[0098] In the operation of Fig. 13, in order to perform clustering processing in units of evaluation data sets, first, in step S11, the clustering processing unit 40 sets an evaluation data set including image data of m object images and class information of the m object images. The m object images are object images cut out from M input images in the image classification data set. M is an integer equal to or greater than 1.

[0099] m is an integer of 3 or more. However, m may be an integer of 4 or more, an integer of 5 or more, or an integer of even more. m is a fixed value m CNST (For example, “m=m CNST =10) is also acceptable. CNST ", the value of M is indefinite and a fixed value m CNST The evaluation dataset is set by extracting M input images containing object images of the number m=m from the image classification dataset. CNST ", but "M=M CNST " is also acceptable. CNST is a fixed value, which is basically an integer of 2 or more, but may be 1. “M=M CNST ", then M CNST The total number of object regions (bounding boxes) set in each input image is the value of m, so the value of m can change each time an evaluation dataset is set. “M=M CNST " is the general rule, but if the total number of object regions (bounding boxes) set in the M input images is less than the required number (at least 3), the value of M may be increased until the total number is equal to or greater than the required number.

[0100] When the evaluation data set is set in step S11, the processes of steps S12 to S15 or steps S12 to S16 are sequentially performed on the set evaluation data set. That is, in step S12, the clustering processing unit 40 performs clustering of the m object images in the evaluation data set based on the feature amounts and class information of the m object images, thereby classifying the m object images into one or more clusters. In the following step S13, the index derivation unit 50 derives the center of gravity, the distance average, and the distance variance of the feature amounts for each cluster. Then, in step S14, the error identification unit 60 performs a process for identifying an error cluster based on the distance average and the distance variance. In step S15, it is checked whether or not an error cluster has been identified in this process. If an error cluster has been identified (Y in step S15), the process of step S16 is performed and then the process proceeds to step S17, but if an error cluster has not been identified (N in step S15), the process proceeds directly to step S17 without going through the process of step S16. In step S16, the error identifying unit 60 identifies erroneously labeled object images through reclustering of the error clusters.

[0101] In step S17, the data processing device 1 (for example, the clustering processing unit 40) determines whether or not the evaluation of all object images included in the image classification dataset has been completed. If all object images included in the image classification dataset have been included in any one of the evaluation datasets and the processes of steps S12 to S15 or steps S12 to S16 have been performed, it is determined that the evaluation has been completed (Y in step S17) and the process proceeds to step S18. If one or more object images included in the image classification dataset have not yet been included in the evaluation dataset, the process returns to step S11 and repeats the processes from step S11 onwards.

[0102] In step S18, the training dataset generation unit 70 generates a training dataset based on the provisional dataset while performing the erroneous label handling process described in the third embodiment as necessary. The generated training dataset is stored in the database 80.

[0103] <<Fifth Example>> A fifth embodiment will be described. In the fifth embodiment, applied techniques, modified techniques, or supplementary matters to the above-mentioned matters will be described.

[0104] In the above-described embodiment, a specific example was given in which the number of erroneously labeled object images belonging to an error cluster is two (see Figures 7 to 12), but the number of erroneously labeled object images belonging to one error cluster may be one, or may be three or more.

[0105] The data processing device 1 includes, as hardware, a central processing unit (CPU), which is an arithmetic processing device, a graphics processing unit (GPU), a read only memory (ROM), and a random access memory (RAM), etc. The data processing device 1 may realize the functions of each part shown in Fig. 1 by executing a program stored in the ROM or a program acquired from another device through communication with the CPU, and thus may realize each process of steps S11 to S18 in Fig. 13.

[0106] An inference model for image recognition can be constructed by machine learning an arbitrary neural network using the learning data set generated by the data processing device 1. For example, a learning device including a DNN (Deep Neural Network) may be formed in the in-vehicle device, and the learning data set may be used as teacher data to cause the learning device of the in-vehicle device to perform machine learning. After machine learning, an inference model capable of object detection is formed by the learning device of the in-vehicle device. Then, object detection using the inference model may be performed in the in-vehicle device, and the result of the object detection may be used for automatic driving or driving assistance that may be implemented in the vehicle. The in-vehicle device here refers to a device installed in a vehicle such as an automobile.

[0107] The data processing device 1 itself may be an in-vehicle device. Depending on the vehicle (e.g., a broadcasting vehicle), an in-vehicle device having abundant computing resources may be installed, and in this case, the data processing device 1 itself may be an in-vehicle device.

[0108] A part or all of the processing executed by the data processing device 1 may be realized by a mixture of software and hardware. A computer program for causing a computer to execute the above-mentioned method and a computer-readable recording medium having the program recorded thereon are included in the scope of this embodiment. Here, the computer-readable recording medium is, for example, a flexible disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a large-capacity DVD, a next-generation DVD, or a semiconductor memory.

[0109] The embodiments of the present invention can be modified in various ways as appropriate within the scope of the technical ideas set forth in the claims. The above embodiments are merely examples of the present invention, and the meanings of the terms of the present invention and each component are not limited to those described in the above embodiments. The specific numerical values ​​shown in the above description are merely examples, and can be changed to various numerical values ​​as a matter of course. [Explanation of symbols]

[0110] 1 Data Processing Device 10 Image classification dataset acquisition section 20 Machine learning processing section 21 Neural Networks 30 Reasoning section 31 Feature extraction unit 40 Clustering Processing Unit 50 Indicator derivation part 60 Error Identification Section 70 Learning Dataset Generation Unit 80 Databases

Claims

1. A data processing device having an arithmetic processing unit, The arithmetic processing unit is creating an inference model for object classification by performing machine learning on a neural network using an image classification dataset including a plurality of object images, each of which includes image data of an object, and class information that identifies a class of an object in each of the object images; Derive features of each object image by inputting the image classification dataset into the inference model; performing clustering to classify the m object images into k clusters based on feature amounts of the m object images included in the plurality of object images and the class information for the m object images; deriving a center of gravity of a plurality of feature points belonging to each cluster obtained by the clustering and a distance between the center of gravity and each feature amount, and deriving an average and a variance of the distances derived for the plurality of feature amounts; identifying a cluster to which an object image having an error in the class information belongs as an error cluster based on the derived results of the mean and the variance; performing reclustering of the n object images belonging to the erroneous cluster based on the n feature amounts for the n object images, and identifying an object image having an error in the class information from among the n object images based on a result of the reclustering; m represents an integer of 3 or more, n represents an integer of 3 or more, k is equal to the number of object classes in the m object images represented by the class information for the m object images. , data processing device.

2. The arithmetic processing unit identifies, as the erroneous cluster, a cluster whose derived mean is equal to or greater than a predetermined first threshold and whose derived variance is equal to or greater than a predetermined second threshold.

2. A data processing apparatus according to claim 1.

3. The arithmetic processing unit classifies the n object images into first and second clusters by the reclustering, and when a total number of object images belonging to the second cluster is smaller than a total number of object images belonging to the first cluster, identifies the object image belonging to the second cluster as an object image having an error in the class information.

3. A data processing device according to claim 1 or 2.

4. The arithmetic processing unit classifies the n object images into first and second clusters by the reclustering, and when a distance between a center of gravity of the n feature amounts for the n object images and the second cluster is greater than a distance between a center of gravity of the n feature amounts for the n object images and the first cluster, identifies an object image belonging to the second cluster as an object image having an error in the class information.

3. A data processing device according to claim 1 or 2.

5. the plurality of object images are images cut out from a plurality of input images, the arithmetic processing unit uses the plurality of input images as a plurality of learning images, and generates a learning dataset including image data of the plurality of learning images and annotation information that identifies a position and a class of an object in each learning image; The arithmetic processing unit extracts, as an annotation target, an input image including an object image having an error in the class information from the plurality of input images based on a result of identifying the erroneous cluster, and corrects the annotation information for the annotation target using information provided from outside for the annotation target.

5. A data processing device according to claim 1.

6. the plurality of object images are images cut out from a plurality of input images, the arithmetic processing unit extracts a portion of the plurality of input images as a plurality of learning images, and generates a learning dataset including image data of the plurality of learning images and annotation information that identifies a position and a class of an object in each learning image; The arithmetic processing unit excludes an input image including an object image having an error in the class information from the plurality of learning images based on a result of identifying the erroneous cluster.

5. A data processing device according to claim 1.

7. a feature derivation step of creating an inference model for object classification by executing machine learning of a neural network using an image classification dataset including a plurality of object images each including image data of an object and class information identifying a class of an object in each object image, and deriving features of each object image by inputting the image classification dataset to the inference model; a clustering processing step of performing clustering to classify the m object images into k clusters based on feature amounts of the m object images included in the plurality of object images and the class information for the m object images; an index deriving step of deriving, for each cluster obtained by the clustering, a center of gravity of a plurality of feature points belonging to the cluster and a distance between the center of gravity and each feature amount, and deriving an average and a variance of the distances derived for the plurality of feature amounts; and an error identifying step of identifying, as an error cluster, a cluster to which an object image having an error in the class information belongs, based on the derived results of the mean and the variance, In the error identifying step, reclustering is performed on the n object images belonging to the error cluster based on the n feature amounts for the n object images, and an object image having an error in the class information is identified from among the n object images based on a result of the reclustering; m represents an integer of 3 or more, n represents an integer of 3 or more, k is equal to the number of object classes in the m object images represented by the class information for the m object images. ,Data Processing Methods.

Images