Label correction device, label correction method, and program

Abstract

Correct the incorrect labels in the training data. [Solution] The label correction device 1X mainly comprises a clustering means 32X, a subclass identification means 33X, and a label correction means 34X. The clustering means 32X classifies a set of pairs of images and class labels representing the class to which the images belong into subclasses by clustering the features of the images. The subclass identification means 33X identifies a noise-contaminated subclass, which is the subclass to which pairs with erroneous class labels belong, based on an index of the variability of features for each subclass. The label correction means 34X selects the subclass most similar to the pair belonging to the noise-contaminated subclass from all subclasses and replaces the class label of the pair belonging to the noise-contaminated subclass with the class label representing the class to which the selected subclass belongs.

Description

Technical Field

[0001] This disclosure relates to the technical field of a label correction device, a label correction method, and a program for training data of a machine learning model.

Background Art

[0002] Conventionally, a technique of performing machine learning of a classification model using, as training data, a set of pairs of an image and a label indicating a class to which the image belongs has been known. For example, Patent Document 1 discloses a technique of classifying an object image into a predetermined number of clusters using an evaluation data set in which an object image and class information of the object image are associated, and identifying a cluster having an error in a label from the classified clusters.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When unbalanced data with a bias in the number of samples for each class is obtained as training data, overfitting is likely to occur because samples belonging to the minority class are frequently used. To prevent such overfitting, for example, a method of performing clustering to form subclasses having the same number of samples as the minority class and performing contrast learning between the clusters can be considered. However, even in this case, if the label group of the training data contains noise, the accuracy of clustering decreases, and there is a problem that high-precision model learning cannot be achieved.

[0005] One of the purposes of this disclosure is to provide a label correction device, a label correction method, and a program that can correct erroneous labels contained in training data, in light of the above-mentioned problems. [Means for solving the problem]

[0006] One embodiment of a label correction device is: A clustering means that classifies a set of pairs of images and class labels representing the class to which the images belong into subclasses by clustering the features of the images, A subclass identification means for identifying a noise-infected subclass, which is the subclass to which a pair with an error in the class label belongs, based on an index of the variability of the feature quantities for each subclass, A label modification means that selects from all the subclasses the subclass that is most similar to the pair belonging to the noise-infusing subclass, and replaces the class label of the pair with the class label representing the class to which the selected subclass belongs; This is a label correction device that has the following features.

[0007] One aspect of the label correction method is: Computers The set of pairs of an image and a class label representing the class to which the image belongs is classified into subclasses by clustering the features of the image. Based on the variability index of the feature quantities for each subclass, a noise-infected subclass is identified, which is the subclass to which the pair with the erroneous class label belongs. Select the subclass that is most similar to the pair belonging to the noise-inducing subclass from all the subclasses, and replace the class label of the pair with the class label representing the class to which the selected subclass belongs. This is a method for correcting labels.

[0008] One aspect of the program is: The set of pairs of an image and a class label representing the class to which the image belongs is classified into subclasses by clustering the features of the image. Based on the variability index of the feature quantities for each subclass, a noise-infected subclass is identified, which is the subclass to which the pair with the erroneous class label belongs. This program causes a computer to perform the following operations: select the subclass that is most similar to the pair belonging to the noise-infusing subclass from all the subclasses, and replace the class label of the pair with the class label representing the class to which the selected subclass belongs. [Effects of the Invention]

[0009] One example of the effects of this disclosure is that erroneous labels contained in the training data can be suitably corrected. [Brief explanation of the drawing]

[0010] [Figure 1] The general configuration of the learning system is shown below. [Figure 2] This is a diagram illustrating the overview of the classification model. [Figure 3] This shows the hardware configuration of the learning device. [Figure 4] This is a functional block diagram of the learning device for training classification models. [Figure 5] This figure schematically shows the distribution of subclasses in the feature space. [Figure 6] This diagram shows an overview of inter-subclass learning. [Figure 7] This diagram illustrates an overview of inter-class learning. [Figure 8] This is an example of a flowchart illustrating the process performed by a learning device. [Figure 9] This shows the general configuration of an endoscopic examination system. [Figure 10] This shows the hardware configuration of the image processing unit. [Figure 11] This shows an example of a display screen shown by a display device during an endoscopic examination. [Figure 12] It is a block diagram of a label correction device. [Figure 13] It is an example of a flowchart executed by the label correction device.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of a label correction device, a label correction method, and a program will be described with reference to the drawings.

[0012] <First Embodiment> (1) System Configuration FIG. 1 shows a schematic configuration of a learning system 100. As shown in FIG. 1, the learning system 100 is a system that performs learning of a model (also referred to as "classification model M") for classifying images, and uses training data D1, which has a bias in the number of samples for each class and in which errors are mixed in the labels, for learning. The learning system 100 mainly includes a learning device 1 and a storage device 2 that stores the training data D1 and the model information D2.

[0013] For example, when an endoscopic image taken in an examination using an endoscope for a human organ is input, the classification model M is a classification model that performs classification regarding at least any one of the presence or absence of a lesion, the type of the lesion, or the degree of the lesion in the input endoscopic image. Examples of the endoscope targeted in the present disclosure include, for example, a pharyngeal endoscope, a bronchoscope, an upper gastrointestinal endoscope, a duodenal endoscope, a small intestine endoscope, a large intestine endoscope, a capsule endoscope, a thoracoscope, a laparoscope, a cystoscope, a choledochoscope, an arthroscope, a spinal endoscope, an endovascular endoscope, an epidural endoscope, and the like.

[0014] Furthermore, the classification model M to be trained is not limited to the classification model that classifies lesions in endoscopic images as described above. For example, classification model M may be a classification model that classifies lesions in any medical image other than endoscopy (e.g., images obtained from ultrasound, PET, CT, or MRI scans). In another example, classification model M may be a classification model that, when an image taken during a visual inspection of an object other than a person (visual inspection image) is input, classifies at least one of the following in the input image: whether or not there is an abnormality, the type of abnormality, or the degree of the abnormality.

[0015] The learning device 1 performs machine learning (training) of the classification model M based on the training data D1 stored in the memory device 2, and updates the model information D2 stored in the memory device 2. Furthermore, as will be described later, the learning device 1 modifies the labels included in the training data D1.

[0016] The storage device 2 is a memory that stores information necessary for processing the learning device 1. The storage device 2 may be an external storage device such as a hard disk connected to or built into the learning device 1, a storage medium such as flash memory, or a server device that communicates data with the learning device 1. Furthermore, the storage device 2 may be composed of multiple storage devices, with each of the above-mentioned storage units distributed among them. The storage device 2 contains training data D1 and model information D2.

[0017] Training data D1 is the data used for machine learning of the classification model M by the learning device 1. Training data D1 has multiple records, and each record contains a training image, which is an image to be input to the classification model M in machine learning, and a class label that indicates the correct class that the classification model M should output when the training image is input to the classification model M. In other words, training data D1 contains a set of pairs of training images and class labels that indicate the class of the training image. Hereafter, each record of training data D1 used for training the classification model M will also be called a "sample".

[0018] Here, in the case of classifying lesions in medical images, the class may be classified according to the type of lesion (lesion type), or it may be classified considering the shape of the lesion (e.g., elevated lesion, flat lesion) and the degree of malignancy. In the case of classifying abnormalities other than lesions, for example, the class may be classified considering the presence or absence of an abnormality and the degree of the abnormality. Class labels are generated in advance, for example, through annotation work.

[0019] Here, the training data D1 is imbalanced data with a bias in the number of samples per class, and the class labels that indicate the correct class are mixed with noise (i.e., labels that indicate the wrong class). For example, if the classification model M is a model that classifies lesion types in the input training images, the number of samples will be small for classes corresponding to lesion types that have a low prevalence, look similar to other types of lesions and are difficult to diagnose, and errors will occur in the class labels. For example, flat lesions (surface lesions) do not have distinctive global color or shape compared to the surrounding area, and have a low prevalence and few cases, so the number of samples will be small. Hereafter, class labels that indicate the wrong class will also be called "noise labels".

[0020] Thus, in the classification of lesions in medical images, the number of malignant lesion samples (i.e., the number of cases) is small, making identification difficult. Furthermore, when using training data that is unbalanced and contains such noisy labels, it is difficult to obtain a classification model M that outputs highly accurate classification results through machine learning. In addition, machine learning with unbalanced training data is prone to overfitting because samples from minority classes are frequently used. Taking the above into consideration, the learning device 1 in this embodiment corrects the noisy labels and performs controlled learning that can handle unbalanced data, as will be described later.

[0021] Model information D2 is the information necessary to construct the classification model M, and includes the parameters of the classification model M that are updated by the machine learning performed by the learning device 1. The classification model M is a machine learning model (including statistical models, hereinafter the same) having any architecture used for multi-task learning, such as a neural network or a support vector machine. If the classification model M is constructed by a neural network, the model information D2 includes various parameters (including hyperparameters), such as the layer structure, the neuron structure of each layer, the number and size of filters in each layer, and the weights of each element of each filter.

[0022] Figure 2 shows an overview of the classification model M. As shown in Figure 2, the classification model M is a model (engine) that outputs a classification result of the class to which an input image belongs when an image is input to the classification model M. In other words, the classification model M is a model that has learned the relationship between an input image and the class to which that image belongs. The classification model M includes a feature extractor (backbone) Mb that performs feature extraction, and a head Mh that outputs a class classification result based on the features obtained by the feature extractor Mb. "Features" are vector-form data representing features (feature vectors). Note that features may also be tensor-form data of a predetermined rank other than vector form. "Class classification result" may be a score indicating the likelihood of each candidate class, or it may be a pair of the most likely class and the score indicating its likelihood.

[0023] In this embodiment, the classification model M is assumed to have undergone machine learning in advance using a predetermined number of pairs of training images and class labels, and the parameters obtained from this advance machine learning are stored in the model information D2. This advance machine learning may be performed by the learning device 1 using training data D1, or it may be performed by a device other than the learning device 1 using arbitrary training data.

[0024] The configuration of the learning system 100 shown in Figure 1 is an example, and various modifications may be made. For example, the learning device 1 and the storage device 2 may be implemented by the same device. In another example, the learning device 1 may be composed of multiple devices. In this case, the multiple devices constituting the learning device 1 exchange information necessary to execute pre-assigned processes between the devices by direct wired or wireless communication or by communication via a network.

[0025] (2) Hardware configuration Figure 3 shows an example of the hardware configuration of the learning device 1. The learning device 1 includes a processor 11, memory 12, and interface 13 as hardware. The processor 11, memory 12, and interface 13 are connected via a data bus 19.

[0026] The processor 11 functions as a controller (arithmetic unit) that controls the entire learning device 1 by executing programs stored in memory 12. The processor 11 is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may consist of multiple processors. The processor 11 is an example of a computer.

[0027] Memory 12 is composed of various volatile and non-volatile memories, such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. Memory 12 also stores the program executed by the learning device 1. Some of the information stored in memory 12 may be stored in an external storage device, such as a storage device 2 that can communicate with the learning device 1, or in a storage medium that is detachable from the learning device 1. Furthermore, memory 12 may store information that would otherwise be stored in storage device 2.

[0028] Interface 13 is an interface for electrically connecting the learning device 1 with other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data with other devices, or they may be hardware interfaces for connecting with other devices via cables, etc.

[0029] (3) Learning a classification model Next, we will explain in detail how to train the classification model M. In general, the learning device 1 clusters samples sampled from the training data D1 in the feature space to form subclasses, which are groups that further classify the classes. Then, based on the variability of the samples within the subclass, the learning device 1 identifies subclasses that contain noise labels. The learning device 1 then modifies the labels of each sample within the identified subclasses according to the subclass with the highest similarity to that sample, and performs comparative learning between the modified classes and subclasses. As a result, the learning device 1 accurately modifies noise labels and performs machine learning of the classification model M using imbalanced data without overfitting.

[0030] (3-1) Functional Blocks Figure 4 is a functional block diagram of the learning device 1 related to the learning of the classification model M. Functionally, the processor 11 of the learning device 1 has a feature extraction unit 31, a clustering unit 32, a noise-injecting subclass identification unit 33, a label correction unit 34, and a learning unit 35. In Figure 4, blocks where data is exchanged are connected by solid lines, but the combination of blocks where data is exchanged is not limited to Figure 4. The same applies to the diagrams of other functional blocks described later.

[0031] The feature extraction unit 31 samples "N" samples (i.e., pairs of training images and class labels) from the training data D1 and extracts features from the N training images contained in the sampled N samples. "N" is a positive integer and is predetermined to be a number that allows for the generation of subclasses by clustering, which will be described later. In this case, the feature extraction unit 31 constructs a classification model M by referring to the model information D2 and obtains the features output by the feature extractor Mb by inputting each training image into the feature extractor Mb of the classification model M. The feature extraction unit 31 supplies the N feature quantities and class label pairs to the clustering unit 32. The N samples extracted from the training data D1 are an example of a "set of pairs".

[0032] The clustering unit 32 classifies N samples into classes based on class labels, and further classifies the class-classified samples into subclasses by feature clustering. In this case, the clustering unit 32 clusters the samples for each class based on the similarity of the features. "Similarity" can be any index representing the degree of similarity, for example, the cosine similarity of vector features, or the distance between features in the feature space representing the features. The clustering algorithm can be a non-hierarchical clustering method such as k-means or a mixed normal distribution, or a hierarchical clustering method such as Ward's method, centroid method, or group average method. In this case, for example, the clustering unit 32 may perform clustering such that the smallest class with the fewest number of samples has a granularity (number of samples or variability) similar to that of one subclass in the other classes. In this case, for example, the smallest class becomes one subclass overall, and subclasses with a granularity similar to that of the smallest class are formed in the other classes. The clustering unit 32 supplies feature quantities corresponding to N samples, class labels, and subclass labels indicating subclasses to the noise-infected subclass identification unit 33.

[0033] The noise-infected subclass identification unit 33 identifies a subclass containing a sample with a noise label (also called a "noise-infected subclass") based on the feature quantities, class labels, and subclass labels corresponding to the N samples supplied from the clustering unit 32. In this case, the noise-infected subclass identification unit 33 calculates the degree of variation in the feature quantities of the samples belonging to each subclass (also called "intra-subclass variance") for each subclass. Then, the noise-infected subclass identification unit 33 identifies a noise-infected subclass based on the intra-subclass variance. The noise-infected subclass identification unit 33 then supplies the feature quantities, class labels, subclass labels, and labels indicating whether or not a sample is a noise-infected subclass, corresponding to the N samples, to the label correction unit 34.

[0034] Note that the within-subclass variance is not limited to the statistical indicator value of variance, but may be the value of any indicator that represents the degree of variability. For example, the within-subclass variance may be the value of the determinant of the variance-covariance matrix of the features, or the standard deviation of a specific component of the variance-covariance matrix. In another example, the within-subclass variance may be calculated as the variance of the cosine distance between samples. The same applies to the term "variance" used hereafter.

[0035] The label correction unit 34 calculates the similarity between each sample in the noise-inflicted subclass and all subclasses, and selects the subclass with the highest similarity to each sample (if similarity is distance, the shortest distance; the same applies hereafter). The label correction unit 34 then replaces the class label so that each sample belongs to the selected subclass. In this way, the label correction unit 34 corrects the class label of the sample in the noise-inflicted subclass that has the highest similarity to the subclasses of other classes. The label correction unit 34 also replaces the subclass label of the sample whose class label has been changed with a subclass label that represents the selected subclass. The label correction unit 34 then supplies the noise-inflicted subclass identification unit 33 with the feature quantities corresponding to N samples, the corrected class labels, and the subclass labels to the learning unit 35.

[0036] The learning unit 35 performs controlled learning of the classification model M based on the features of N samples and the class labels and subclass labels of the N samples to which the corrections have been applied, and updates the parameters of the feature extractor Mb of the classification model M. Details of this controlled learning will be described later. The learning unit 35 stores the updated parameters of the feature extractor Mb in the model information D2.

[0037] The feature extraction unit 31, clustering unit 32, noise-injected subclass identification unit 33, label correction unit 34, and learning unit 35 can be implemented, for example, by the processor 11 executing a program. Alternatively, the necessary programs may be recorded on any non-volatile storage medium and installed as needed to implement each component. At least a portion of these components may be implemented not only by software programs, but also by any combination of hardware, firmware, and software. At least a portion of these components may also be implemented using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the program composed of the above components may be implemented using this integrated circuit. At least a portion of each component may also be implemented by an ASSP (Application Specific Standard Produce), ASIC (Application Specific Integrated Circuit), or quantum processor (quantum computer control chip). Thus, each component may be implemented by various hardware. The same applies to other embodiments described later. Furthermore, each of these components may be realized through the collaboration of multiple computers, for example, using cloud computing technology.

[0038] (3-2) Identification of noise intrusion subclasses Next, we will specifically explain the identification of noise-infused subclasses by the noise-infused subclass identification unit 33. Figure 5 is a schematic diagram showing the distribution of subclasses in the feature space of the features extracted by the feature extraction unit 31. For the sake of explanation, the feature space here is assumed to be a two-dimensional space. In this case, there are classes A to C as candidate classes to be classified by the classification model M. The clustering unit 32 forms five subclasses Aa to Ae for class A, three subclasses Ba to Bc for class B, and three subclasses Ca to Cc for class C. Here, the distribution of subclass Ad and the distribution of subclass Ba partially overlap, and the distribution of subclass Bc and the distribution of subclass Cc partially overlap.

[0039] In this case, the noise-infected subclass identification unit 33 calculates the intra-subclass variance for each of the subclasses Aa~Ae, Ba~Bc, and Ca~Cc. The noise-infected subclass identification unit 33 then calculates the average of the intra-subclass variances for each class and identifies subclasses whose intra-subclass variance is greater than the calculated average as noise-infected subclasses. In other words, in this case, the noise-infected subclass identification unit 33 sets the average of the intra-subclass variances for each class as a threshold for the intra-subclass variance to determine which subclasses are noise-infected.

[0040] For example, in the case of class A, the noise-infected subclass identification unit 33 calculates the average of the intraclass variances of subclasses Aa to Ae. The intraclass variances of subclasses Aa to Ac and Ae are less than or equal to the calculated average, and the intraclass variance of subclass Ad is greater than the average, so the noise-infected subclass identification unit 33 identifies subclass Ad as a noise-infected subclass. Similarly, the noise-infected subclass identification unit 33 identifies subclasses Ba and Bc, whose intraclass variances are greater than the average of the intraclass variances of subclasses Ba to Bc, as noise-infected subclasses, respectively. In addition, the noise-infected subclass identification unit 33 identifies subclass Cc, whose intraclass variance is greater than the average of the intraclass variances of subclasses Ca to Cc, as a noise-infected subclass, respectively.

[0041] The noise-infected subclass identification unit 33 may perform a process to normalize the intra-subclass variance in addition to the above process. For example, the noise-infected subclass identification unit 33 calculates the inter-subclass variance, which is the variance between subclasses, for each class. For example, the noise-infected subclass identification unit 33 determines the average of the feature quantities of the samples belonging to each subclass as the centroid (i.e., the center point or centroid) in the feature space of each subclass. The noise-infected subclass identification unit 33 may use an arbitrary representative point that represents the feature quantities of the samples of the subclass in the feature space instead of the centroid.

[0042] The noise-infected subclass identification unit 33 then calculates the centroid variance of each subclass and uses the calculated variance as the inter-subclass variance. The noise-infected subclass identification unit 33 then defines the value obtained by dividing the intra-subclass variance of each subclass by the inter-subclass variance of the corresponding class as the normalized intra-subclass variance. In this case, the noise-infected subclass identification unit 33 may set the threshold value of the normalized intra-subclass variance for determining the noise-infected subclass to a predetermined value.

[0043] (3-3) Label correction Next, the label correction by the label correction unit 34 will be explained in detail with reference to Figure 5.

[0044] The label correction unit 34 calculates the similarity between each sample of the noise-infected subclass identified by the noise-infected subclass identification unit 33 and all subclasses. In the example in Figure 5, the label correction unit 34 calculates the similarity for each sample of subclasses Ad, Ba, Bc, and Cc, which have been identified as noise-infected subclasses. For example, in the case of subclass Ad, which is a noise-infected subclass, each sample of subclass Ad is designated as the sample to be processed. The label correction unit 34 then calculates the distance (e.g., cosine distance) between the feature quantities of the sample to be processed and the representative points of each of the subclasses Aa~Ae, Ba~Bc, and Ca~Cc. The representative points of a subclass are, for example, the centroid of the feature quantities in the feature space of the sample belonging to the subclass, or other points that represent the feature quantities of the sample within the subclass. The label correction unit 34 may also calculate the value of an arbitrary index that serves as an indicator of the similarity between vectors instead of the distance described above.

[0045] The label correction unit 34 then selects the subclass with the shortest distance from the features of the sample to be processed and determines that the sample to be processed belongs to the class to which the selected subtask belongs. In this case, if the class label of the sample to be processed is different from the class to which the selected subclass belongs, the label correction unit 34 corrects the class label of the sample to be processed according to the determination result. The label correction unit 34 also replaces the subclass label of the sample to be processed whose class label has been changed with the subclass label that indicates the subclass closest in distance to the sample.

[0046] For example, if the label correction unit 34 determines that a sample of subclass Ad is the shortest distance from subclass Ba across all subclasses, it replaces the class label of that sample from "Class A" to "Class B". The label correction unit 34 also replaces the subclass label of that sample from "Ad" to "Ba". Then, the label correction unit 34 sets the class labels and subclass labels of samples of subclasses Ba, Ba, Bc, and Cc, which have been identified as noise-infected subclasses, to match the subclass with which they are shortest distance.

[0047] In this way, the label correction unit 34 improves clustering accuracy by correcting labels, enabling effective comparative learning of the classification model M described later, and thereby improving the classification accuracy of the classification model M.

[0048] (3-4) Contrastive Learning The learning unit 35 performs controlled learning of the feature extractor Mb of the classification model M based on the feature quantities for N samples and the class labels and subclass labels for N samples that reflect the corrections. In this case, the learning unit 35 performs inter-subclass learning, which trains the feature extractor Mb to repel different subclasses, and inter-class learning, which trains the feature extractor Mb to repel different classes and attract subclasses of the same class.

[0049] Figure 6 shows an overview of inter-subclass learning. Here, an overview of inter-subclass learning for samples belonging to class A and class B is shown.

[0050] In inter-subclass learning, the learning unit 35 inputs training images of samples belonging to different subclasses, regardless of whether they belong to the same class or not, into the feature extractor Mb. By doing so, the parameters of the feature extractor Mb are updated so that the features output by the feature extractor Mb move further apart from each other. Inter-subclass learning corresponds to first-order controlled learning, in which the feature extractor Mb is trained to bring together the features of any two pairs of identical subclasses and to move apart the features of any two pairs of different subclasses.

[0051] In the example shown in Figure 6, the learning unit 35 updates the parameters of the feature extractor Mb so that the features of samples from subclass Aa of class A are separated from the features of samples from subclass Ab of the same class A. The learning unit 35 also updates the parameters of the feature extractor Mb so that the features of samples from subclass Aa of class A are separated from the features of samples from subclass Ba of a different class B. Similarly, the learning unit 35 updates the parameters of the feature extractor Mb so that the features of samples from subclass Ab of class A are separated from the features of samples from subclass Ba of a different class B. On the other hand, the learning unit 35 updates the parameters of the feature extractor Mb so that the features of any two samples within the same subclass are brought closer together.

[0052] Figure 7 shows an overview of inter-class learning. In inter-class learning, the learning unit 35 inputs training images of samples belonging to different classes into the feature extractor Mb, updating the parameters of the feature extractor Mb so that the features output by the feature extractor Mb are further apart from each other. Inter-class learning corresponds to a second type of controlled learning in which the feature extractor Mb is trained to bring together the features of any two pairs of identical classes and to move apart the features of any two pairs of different classes.

[0053] In the example shown in Figure 7, the learning unit 35 updates the parameters of the feature extractor Mb so that the feature quantities of the samples from subclass Aa of class A are brought closer to the feature quantities of the samples from subclass Ab of the same class A. On the other hand, the learning unit 35 updates the parameters of the feature extractor Mb so that the feature quantities of the samples from subclass Aa of class A are brought further apart to the feature quantities of the samples from subclass Ba of a different class B. Similarly, the learning unit 35 updates the parameters of the feature extractor Mb so that the feature quantities of the samples from subclass Ab of class A are brought further apart to the feature quantities of the samples from subclass Ba of a different class B.

[0054] Alternatively, the learning unit 35 may repeatedly perform controlled learning using N samples, and then perform machine learning on the entire classification model M, including the head Mh, using the training data D1. In this case, the learning unit 35 updates the parameters of the classification model M to minimize the error (loss) between the classification result of the class output by the classification model M when a training image is input to the classification model M and the class label associated with the input training image. The algorithm used to determine the parameters to minimize the error may be any learning algorithm used in machine learning, such as gradient descent or backpropagation. For training the entire classification model M, the learning unit 35 may use class labels that reflect the label corrections made in the label correction unit 34. The learning unit 35 may also perform machine learning on the entire classification model M, including the head Mh, using the training data D1 instead of the controlled learning described above. Even in this case, the learning unit 35 can train the classification model M based on the training data D1 with corrected class labels.

[0055] (4) Processing flow Figure 8 is an example of a flowchart showing an overview of the processes performed by the learning device 1.

[0056] First, the learning device 1 samples N pairs of training images and class labels from the training data D1 stored in the memory device 2, which are N samples to be used for training the classification model M (step S11). In this case, for example, the learning device 1 samples N pairs from the training data D1 by random sampling.

[0057] Next, the learning device 1 extracts the features of the training images acquired in step S11 using the feature extractor Mb (step S12). In this case, the learning device 1 inputs the training images into the feature extractor Mb, which is configured by referring to the model information D2 of the classification model M that has been trained, and obtains the features output by the feature extractor Mb. As a result, the learning device 1 obtains N features corresponding to N training images.

[0058] Next, the learning device 1 performs clustering on N samples based on the similarity of their features (e.g., distance in the feature space), classifying the samples into subclasses that are more detailed than the main class (step S13). This allows the learning device 1 to generate a subclass label for each sample. These subclass labels are then used for controlled learning.

[0059] Next, the learning device 1 calculates the intra-subclass variance for each subclass and identifies the noise-infected subclass from the subclasses identified in step S13 based on the subclass variance (step S14). Then, for each sample within the identified noise-infected subclass, the learning device 1 calculates the similarity to each subclass and selects the subclass with the highest similarity. Then, the learning device 1 modifies the class label and subclass label according to the selected subclass (step S15). In this case, the learning device 1 sets the class label and subclass label for each sample so that it belongs to the selected subclass.

[0060] Next, the learning device 1 refers to the modified class labels and subclass labels and performs comparative learning between subclasses and between classes (step S16). As a result, the learning device 1 updates the parameters of the feature extractor Mb and reflects the updated parameters in the model information D2.

[0061] Then, the learning device 1 determines whether or not learning has finished (step S17). For example, the learning device 1 determines that learning should be terminated if it has trained the classification model M using all the samples in the training data D1, if steps S11 to S16 have been repeated a predetermined number of times, or if other predetermined learning termination conditions are met. If the learning device 1 determines that learning should not be terminated (step S17; No), it returns to step S11. In this case, the learning device 1 may extract samples from the training data D1 that were not used to train the classification model M in step S11.

[0062] On the other hand, if the learning device 1 determines that learning should be terminated (step S17; Yes), it terminates the flowchart processing. In this case, the learning device 1 may perform machine learning on the entire classification model M, including the head Mh, using the training image and class label pairs contained in the training data D1. The class labels of the training data D1 used in this case reflect the modifications made in step S15.

[0063] (5) Application examples This section describes an application example using a classification model M that has been machine-learned by the learning device 1. Here, as an example, the classification model M is assumed to be a classification model that outputs a classification result regarding lesions in an input endoscopic image when an endoscopic image is input.

[0064] Figure 9 shows a schematic configuration of the endoscopic examination system 200. The endoscopic examination system 200 is a system that classifies lesions in endoscopic images taken with an endoscope and presents the classification results. The endoscopic examination system 200 mainly comprises an image processing device 1A, an endoscope scope 3 connected to the image processing device 1A and handled by the examiner, such as a physician, who performs the examination or treatment, and a display device 4.

[0065] The image processing device 1A acquires images (also called "endoscopic images Ia") taken by the endoscope scope 3 in a time series from the endoscope scope 3 and displays a screen based on the endoscopic images Ia on the display device 4. Endoscopic images Ia are images taken at a predetermined frame period during at least one of the insertion or withdrawal process of the endoscope scope 3 into the patient. In this embodiment, the image processing device 1A classifies each endoscopic image Ia in the time series using a classification model M and presents information regarding the classification results. For example, if there is an endoscopic image Ia classified as containing a lesion, the image processing device 1A notifies the examiner that a lesion is present. The image processing device 1A may also function as the learning device 1 of the first embodiment and perform machine learning processing of the classification model M before the endoscopic examination.

[0066] The endoscope scope 3 mainly comprises an operating section 36 for the examiner to perform predetermined inputs, a flexible shaft 37 that is inserted into the organ to be imaged by the patient, a tip section 38 that incorporates an imaging unit such as a miniature image sensor, and a connection section 39 for connecting to the image processing device 1A.

[0067] The configuration of the endoscopic examination system 200 shown in Figure 9 is an example, and various modifications may be made. For example, the image processing device 1A may be configured integrally with the display device 4. In another example, the image processing device 1A may consist of multiple devices.

[0068] Figure 10 shows the hardware configuration of the image processing device 1A. The image processing device 1A mainly includes a processor 21, memory 22, interface 23, input unit 24, light source unit 25, and sound output unit 26. Each of these elements is connected via a data bus 19.

[0069] The processor 21 executes predetermined processes by running programs and other data stored in memory 22. The processor 21 is a CPU, GPU, TPU, or other type of processor. The processor 21 may be composed of multiple processors. The processor 21 is an example of a computer.

[0070] Memory 22 is composed of various volatile memories used as working memory, such as RAM and ROM, and non-volatile memory that stores information necessary for processing by the image processing device 1A. Memory 22 may also include external storage devices such as hard disks connected to or built into the image processing device 1A, or it may include storage media such as removable flash memory. Programs for the image processing device 1A to perform each of the processes in this embodiment are stored in Memory 22.

[0071] Furthermore, memory 22 stores model information D2, which is information about a classification model M that has been previously machine-learned based on the first embodiment. Model information D2 includes the parameters of the classification model M obtained by machine learning. Memory 22 may also optionally include other information necessary for the image processing device 1A to perform each of the processes in this embodiment.

[0072] Interface 23 performs interface operations between the image processing device 1A and an external device. For example, interface 23 supplies display information "Ib" generated by the processor 21 to the display device 4. Interface 23 also supplies light generated by the light source unit 25 to the endoscope scope 3. Interface 23 also supplies an electrical signal indicating the endoscopic image Ia supplied from the endoscope scope 3 to the processor 21. Interface 23 may be a communication interface such as a network adapter for wired or wireless communication with an external device, or it may be a hardware interface compliant with USB, SATA, etc.

[0073] The input unit 24 generates input signals based on operations performed by the examiner. The input unit 24 can be, for example, a button, touch panel, remote controller, or voice input device. The light source unit 25 generates light to be supplied to the tip 38 of the endoscope scope 3. The light source unit 25 may also incorporate a pump for supplying water or air to the endoscope scope 3. The sound output unit 26 outputs sound based on the control of the processor 21.

[0074] Figure 11 shows an example of a display screen displayed by the display device 4 during an endoscopic examination. The image processing device 1A outputs display information Ib, generated based on the endoscopic image Ia acquired by the endoscopic image acquisition unit 30 and the classification model M configured by referring to the model information D2, to the display device 4. The image processing device 1A displays the above-mentioned display screen on the display device 4 by transmitting the endoscopic image Ia and the display information Ib to the display device 4. In the example display screen shown in Figure 11, the image processing device 1A provides a real-time image display area 71 and a lesion detection result display area 72 on the display screen.

[0075] Here, the image processing device 1A displays a moving image representing the latest endoscopic image Ia in the real-time image display area 71. Furthermore, in the lesion detection result display area 72, the image processing device 1A displays the classification result output by the classification model M when the endoscopic image Ia displayed in the real-time image display area 71 is input to the classification model M. At the time the display screen shown in Figure 11 is displayed, the classification result indicating the presence of lesion type X has been obtained from the classification model M, so the image processing device 1A displays a text message in the lesion detection result display area 72 indicating that there is a high probability that lesion type X is present. Alternatively, instead of displaying the text message in the lesion detection result display area 72, or in addition to this, the image processing device 1A may output a sound (including voice) from the sound output unit 26 to notify that there is a high probability that a lesion is present.

[0076] With the above configuration, the endoscopic examination system 200 can accurately classify endoscopic images Ia of the subject's organs using a classification model M and notify the examiner of the classification results.

[0077] <Second Embodiment> Figure 12 is a block diagram of the label correction device 1X. The label correction device 1X mainly comprises a clustering means 32X, a subclass identification means 33X, and a label correction means 34X. The label correction device 1X can be, for example, the learning device 1 of the first embodiment. The label correction device 1X may be composed of multiple devices.

[0078] The clustering means 32X classifies a set of pairs of images and class labels representing the class to which the images belong into subclasses by clustering the features of the images. The clustering means 32X can be the clustering unit 32 in the first embodiment.

[0079] The subclass identification means 33X identifies a noise-infected subclass, which is the subclass to which a pair with an erroneous class label belongs, based on an index of the variability of feature quantities for each subclass. The subclass identification means 33X can be, for example, the noise-infected subclass identification unit 33 in the first embodiment.

[0080] The label correction means 34X selects the subclass most similar to the pair belonging to the noise-infected subclass from all subclasses, and replaces the class label of the pair belonging to the noise-infected subclass with the class label representing the class to which the selected subclass belongs. If the class labels before and after replacement are the same, the class label is not changed. The label correction means 34X can be, for example, the label correction unit 34 in the first embodiment.

[0081] Figure 13 shows an example of a flowchart executed by the label correction device 1X. The clustering means 32X classifies the set of pairs of images and class labels representing the class to which the images belong into subclasses by clustering the image features (step S21). The subclass identification means 33X identifies the noise-contaminated subclass, which is the subclass to which the pairs with erroneous class labels belong, based on an index of the variability of the features for each subclass (step S22). The label correction means 34X selects the subclass most similar to the pairs belonging to the noise-contaminated subclass from all subclasses and replaces the class labels of the pairs belonging to the noise-contaminated subclass with the class labels representing the class to which the selected subclass belongs (step S23).

[0082] According to the second embodiment, the label correction device 1X can accurately correct class labels even if noise is mixed into the set of image-class label pairs. Then, it becomes possible to train a machine learning model that performs image classification with high accuracy using the set of pairs with corrected class labels.

[0083] In each of the embodiments described above, the program can be stored using various types of non-transitory computer-readable medium and supplied to a computer, such as a processor. Non-transitory computer-readable mediums include various types of tangible storage mediums. Examples of non-transitory computer-readable mediums include magnetic storage mediums (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage mediums (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to the computer using various types of transient computer-readable mediums. Examples of transient computer-readable mediums include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable mediums can supply the program to the computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0084] Furthermore, some or all of the above embodiments (including modified versions, hereinafter the same) may also be described as follows, but are not limited to the following. Also, some or all of the configurations described in Appendices 2 to 8, which are dependent on Appendice 1 above, may also be dependent on Appendices 9 and 10 in the same way as Appendices 2 to 8. Moreover, not limited to the devices, methods, and storage media described in the appendices, some or all of the configurations described in the appendices may also be dependent on methods, various hardware, software, various recording means for recording software (including storage media), or systems, without departing from the above embodiments.

[0085] [Note 1] A clustering means that classifies a set of pairs of images and class labels representing the class to which the images belong into subclasses by clustering the features of the images, A subclass identification means for identifying a noise-infected subclass, which is the subclass to which a pair with an error in the class label belongs, based on an index of the variability of the feature quantities for each subclass, A label modification means that selects from all the subclasses the subclass that is most similar to the pair belonging to the noise-infusing subclass, and replaces the class label of the pair with the class label representing the class to which the selected subclass belongs; A label correction device having the following features. [Note 2] The label correction means determines a representative point for each subclass in the feature space of the feature quantity, selects the subclass that is the representative point most similar to the feature quantity corresponding to the pair belonging to the noise-infected subclass, and replaces the class label of the pair with the class label representing the class to which the selected subclass belongs. [Note 3] The label correction device according to Appendix 1, wherein the subclass identification means normalizes the index of variation of the feature quantities for each subclass based on the index of variation between subclasses belonging to the class to which the subclass belongs, and identifies the noise-infected subclass based on the normalized index of variation of the feature quantities for each subclass. [Note 4] The subclass identification means identifies the noise-infected subclass based on a comparison between an index of the variation of the feature quantity and a predetermined threshold, The label correction device according to Appendix 1, wherein the subclass identification means sets the threshold based on the average of the feature variability index for each class. [Note 5] The label correction device according to Appendix 1, further comprising a learning means for performing comparative learning of a feature extractor that extracts the features from the image based on the replaced class label. [Note 6] The learning means considers the pair belonging to the noise-injecting subclass to belong to the selected subclass, A first comparative learning method involves training the feature extractor to bring together any two pairs of features whose subclasses are the same, and to move apart any two pairs of features whose subclasses are different. A label correction device according to Appendix 5, which performs a second controlled learning, in which the feature extractor is trained to bring together any two pairs of features of the same class and to move apart any two pairs of features of different classes. [Note 7] The label correction device according to Appendix 1, further comprising a learning means for performing machine learning of a classification model that outputs a classification result of an image when an image is input, based on the set of pairs including the replaced class labels. [Note 8] The aforementioned image is a medical image. The label correction device as described in Appendix 1, wherein the class is a class classified based on at least one of the presence or absence of a lesion in the medical image, the type of lesion, or the degree of the lesion. [Note 9] Computers The set of pairs of an image and a class label representing the class to which the image belongs is classified into subclasses by clustering the features of the image. Based on the variability index of the feature quantities for each subclass, a noise-infected subclass is identified, which is the subclass to which the pair with the erroneous class label belongs. Select the subclass that is most similar to the pair belonging to the noise-inducing subclass from all the subclasses, and replace the class label of the pair with the class label representing the class to which the selected subclass belongs. A method for correcting labels. [Note 10] The set of pairs of an image and a class label representing the class to which the image belongs is classified into subclasses by clustering the features of the image. Based on the variability index of the feature quantities for each subclass, a noise-infected subclass is identified, which is the subclass to which the pair with the erroneous class label belongs. A program that causes a computer to perform the process of selecting from all the subclasses the subclass that is most similar to the pair belonging to the noise-introducing subclass, and replacing the class label of the pair with the class label representing the class to which the selected subclass belongs. [Note 11] A storage medium containing the program described in Appendix 10.

[0086] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made that are understandable to those skilled in the art within the scope of the present invention. That is, the present invention naturally includes the full disclosure, including the claims, and various modifications and alterations that those skilled in the art could make in accordance with the technical idea. Furthermore, each disclosure of the above-mentioned patent documents and other references is incorporated herein by reference. [Explanation of Symbols]

[0087] 1. Learning device 1X Label Correction Device 2 Storage device 11, 21 processors 12, 22 memory 13, 23 Interfaces 100 Learning Systems 200 Endoscopy System D1 Training Data D2 Model Information

Claims

1. A clustering means that classifies a set of pairs of images and class labels representing the class to which the images belong into subclasses by clustering the features of the images, A subclass identification means for identifying a noise-infected subclass, which is the subclass to which a pair with an error in the class label belongs, based on an index of the variability of the feature quantities for each subclass, A label modification means that selects from all the subclasses the subclass that is most similar to the pair belonging to the noise-infusing subclass, and replaces the class label of the pair with the class label representing the class to which the selected subclass belongs; A label correction device having the following features.

2. The label correction device according to claim 1, wherein the label correction means determines a representative point for each subclass in the feature space of the feature quantity, selects the subclass that is the representative point most similar to the feature quantity corresponding to the pair belonging to the noise-infected subclass, and replaces the class label of the pair with the class label representing the class to which the selected subclass belongs.

3. The label correction device according to claim 1, wherein the subclass identification means normalizes the index of variation of the feature quantities for each subclass based on the index of variation between subclasses belonging to the class to which the subclass belongs, and identifies the noise-infected subclass based on the normalized index of variation of the feature quantities for each subclass.

4. The subclass identification means identifies the noise-infected subclass based on a comparison between an index of the variation of the feature quantity and a predetermined threshold, The label correction device according to claim 1, wherein the subclass identification means sets the threshold based on the average of the feature variability index for each class.

5. The label correction device according to claim 1, further comprising a learning means for performing comparative learning of a feature extractor that extracts the features from the image based on the replaced class label.

6. The learning means considers the pair belonging to the noise-injecting subclass to belong to the selected subclass, A first comparative learning process involves training the feature extractor to bring together any two pairs of features whose subclasses are the same, and to move apart any two pairs of features whose subclasses are different. The label correction device according to claim 5, further comprising: performing a second comparative learning, which involves training the feature extractor to bring together any two pairs of features belonging to the same class and to move apart any two pairs of features belonging to different classes.

7. The label correction device according to claim 1, further comprising a learning means for performing machine learning of a classification model that outputs a classification result of an image when an image is input, based on the set of pairs including the replaced class labels.

8. The aforementioned image is a medical image. The label correction device according to claim 1, wherein the class is a class classified based on at least one of the presence or absence of a lesion in the medical image, the type of lesion, or the degree of the lesion.

9. Computers The set of pairs of an image and a class label representing the class to which the image belongs is classified into subclasses by clustering the features of the image. Based on the variability index of the feature quantities for each subclass, a noise-infected subclass is identified, which is the subclass to which the pair with the erroneous class label belongs. Select the subclass that is most similar to the pair belonging to the noise-inducing subclass from all the subclasses, and replace the class label of the pair with the class label representing the class to which the selected subclass belongs. A method for correcting labels.

10. The set of pairs of an image and a class label representing the class to which the image belongs is classified into subclasses by clustering the features of the image. Based on the variability index of the feature quantities for each subclass, a noise-infected subclass is identified, which is the subclass to which the pair with the erroneous class label belongs. A program that causes a computer to perform the process of selecting from all the subclasses the subclass that is most similar to the pair belonging to the noise-introducing subclass, and replacing the class label of the pair with the class label representing the class to which the selected subclass belongs.

Images