Category-incremental learning method and system for small sample medical image classification

Abstract

The present invention discloses a category incremental learning method and system suitable for small sample medical image classification. First, a basic classification model is trained based on existing medical image samples, and then medical image samples of different categories are collected, including pathological images and corresponding disease categories, and representative features of new categories are calculated based on image features, and the fully connected layer as a classifier model is replaced; then, a category incremental learning task is constructed on the existing medical image data set, and a feature adjustment module is learned; finally, the feature adjustment module is used to calibrate the model prediction during the medical image test. The present invention can solve the difficulties that medical images cannot be obtained at the same time due to privacy issues, and a new model can only be established using existing models and new medical images, and the present invention occupies less resources during implementation and has strong applicability.

Description

Technical Field

[0001] The present invention relates to a learning method for classifying incremental small-sample medical images in an open dynamic environment scenario, and specifically to a small-sample medical image classification method and system based on category incremental learning, belonging to the field of image processing technology. Background Art

[0002] Deep learning algorithms are currently widely used in medical image analysis and have achieved a level comparable to that of human experts in a variety of disease diagnosis tasks, becoming an important tool for auxiliary treatment. However, most of the current intelligent diagnosis systems are designed solely for the diagnosis of a certain disease, and it is difficult to collect a large number of disease images of all diseases in the world to train an efficient intelligent diagnosis system that can handle the diagnosis of all types of diseases. Therefore, giving the intelligent diagnosis system the ability to continuously learn so that it can diagnose based on a small number of images of new types of diseases provided is of great significance to the current deep learning-assisted medical treatment. Considering factors such as patient privacy and the prohibition of data sharing between hospitals, an ideal intelligent diagnosis system should only learn based on the current model and newly provided pathological images, and will not forget the previously learned disease categories while classifying the newly learned pathological images. Summary of the invention

[0003] Purpose of the invention: Since the number of pathological images of some rare diseases is very small, the number of samples provided to the intelligent diagnosis system for updating the model is often in the form of small samples. Therefore, it is necessary to design an effective algorithm so that the intelligent system can not only have continuous learning capabilities, but also be able to update the model from small samples of new types of pathological images.

[0004] There are similarities between the incremental category learning technology and the small sample incremental category learning technology, but the incremental category learning technology mainly considers how to learn new categories with the assistance of a large number of samples without forgetting the old categories, and mainly uses methods such as model output alignment for assistance. However, the small sample incremental category learning task also needs to consider how to extract the key information required for model learning from the small sample. Directly using the incremental category learning technology to deal with the small sample problem will face serious overfitting, causing the model to further forget the previous information. Therefore, the incremental category learning technology is not suitable for the small sample incremental category learning task. In the medical image classification task, it is necessary to design a suitable small sample incremental category learning method.

[0005] Previous category incremental learning techniques used for medical image recognition problems have focused on aligning model prediction results. This series of methods often encounter model overfitting when dealing with small sample medical image tasks, causing the model to forget previously learned disease categories. Taking into account the characteristics of small sample medical image tasks, the present invention provides a learning paradigm based on meta-learning, which constructs small sample learning tasks in existing data sets to adapt to unseen small sample classification problems. Specifically, first, by sampling small sample tasks in existing data sets, in each small sample learning task, the model calibration module is trained to enable it to simultaneously predict known and new category samples. After that, when a small sample task of a new type of disease is collected, the average features of the small sample image are first replaced by the model using the linear layer, and the model calibration module is used to calibrate the model prediction, so that the model can be continuously updated using small sample medical images and the model's existing knowledge can be maintained.

[0006] Technical solution: A category incremental learning method suitable for small sample medical image classification, including the following contents:

[0007] Data collection: Collect small sample medical images from different categories and the category labels of the corresponding medical images in different time periods. Small samples refer to each category containing only one or a few samples.

[0008] Small-sample category incremental learning: Use meta-learning technology to construct small-sample tasks and train the calibration module. In the real small-sample learning task, use the category average feature to replace the linear layer of the existing classifier model, and use the calibration module to complete incremental learning to obtain the classifier model.

[0009] Medical image classification: Use the classifier model constructed in the small sample category incremental learning stage that is suitable for the classification of multiple categories of medical images to perform classification predictions on the medical images to be tested.

[0010] The implementation of the data collection includes the following steps:

[0011] Step 100, collecting previous medical image data, recorded as old medical image data;

[0012] Step 101, collecting the tags corresponding to the previous medical image data, and recording them as old medical image data tags;

[0013] Step 102: After a period of time, collect newly appeared small sample medical images, and record the collected new type of disease images as new medical image data. Small sample means that each type of data contains only one or a few samples.

[0014] Step 103, collect the labels corresponding to the newly appeared small sample medical images and record them as data labels of new medical images.

[0015] The medical image refers to data in the image format of pathological sections, CT images, etc. that can be used for diagnosis.

[0016] The implementation of incremental learning of small sample categories includes the following steps:

[0017] Step 200, initialize the classifier model and the calibration module. The calibration module is a set-to-set function. The calibration module is implemented by a graph model, whose nodes are the feature means of the old medical image samples collected in step 100. The feature representation of the sample is a low-dimensional representation obtained by the convolutional neural network of the classifier model. The edge weight of the graph model is the similarity between two nodes, and the graph model is represented as a transformation T;

[0018] Step 201, construct a small sample incremental learning task in the old medical image dataset. The small sample incremental learning task can be described as N classes and K pictures, that is, N classes are sampled in each small sample incremental learning task, and each class has K pictures. The categories of the sampled tasks do not overlap, and the sampling process will be performed multiple times to simulate the classifier model update process;

[0019] Step 202: For each sampled small sample incremental learning task, first train the initial model based on the cross entropy loss function. The input of the model is a single medical image, and the output of the model is the predicted disease type. The model is decoupled into a feature embedding module φ and a linear layer W. For each new incremental learning stage, the feature embedding module is used to extract the features φ(x) of the medical image, and the feature mean p belonging to the same class is calculated, and the feature mean p is used to replace the corresponding class weight in the linear layer;

[0020] Step 203, after replacing the classifier linear layer with the feature mean, use the graph model T in step 200 to calibrate the replaced linear layer, record the calibrated linear layer as T(W), and use the calibrated linear layer to replace the original linear layer for image prediction. Make predictions in the sampled small sample incremental learning task, and use the cross entropy loss based on gradient back propagation to optimize the feature embedding module φ, the linear layer W, and the graph model T until the loss converges. The set of optimized feature embedding modules, linear layers, and graph models is recorded as a small sample incremental learning model for testing in subsequent stages;

[0021] Step 204, prepare the collected new medical images and their labels, calculate the feature mean p of the same category in the new medical images, use the feature mean p to replace the corresponding category weight in the linear layer, and end the small sample category incremental learning stage.

[0022] The specific steps of medical image classification are:

[0023] Step 300, preparing medical image data for testing as test data, the test data including old medical images and new medical images;

[0024] Step 301, predicting the test data using the small sample incremental learning model learned in step 203;

[0025] Step 302, using the graphical model T optimized in step 203 to calibrate the prediction probability, the specific steps are: first, using the graphical model T to calibrate the classifier linear layer to obtain the calibrated classifier linear layer T(W), using the calibrated classifier linear layer to replace the original linear layer of the classifier, and predicting the sample;

[0026] Step 303: output the classification result of the classifier model calibrated in step 302 on the test data set.

[0027] A category incremental learning system suitable for small sample medical image classification, comprising a data collection module, a small sample category incremental learning module and a medical image classification module;

[0028] The data collection module collects small sample medical images from different categories and category labels of the corresponding medical images in different time periods, where the small sample refers to each category containing only one or several samples;

[0029] The small sample category incremental learning module: uses meta-learning technology to construct small sample tasks, trains a calibration module, uses category average features to replace the linear layer of the existing classifier model in a real small sample learning task, and uses the calibration module to complete incremental learning to obtain a classifier model.

[0030] The medical image classification module uses the classifier model suitable for classification of multiple categories of medical images constructed in the small sample category incremental learning stage to perform classification prediction on the medical image to be tested.

[0031] The implementation processes of the data collection module, the small sample category incremental learning module and the medical image classification module are respectively the same as the implementation processes of the data collection step, the small sample category incremental learning step and the medical image classification step.

[0032] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the category incremental learning method applicable to small sample medical image classification as described above is implemented.

[0033] A computer-readable storage medium stores a computer program for executing the above-described category incremental learning method suitable for small sample medical image classification.

[0034] The method of the present invention is applicable to scenarios where the types of medical images collected in different time periods change, that is, the types of diseases that need to be classified by the medical intelligence system increase over time.

[0035] The graphical model is used as a calibration module in the implementation step of incremental learning of small-sample categories, so that the classifier model can be trained using the sampled small-sample learning task in the training phase and used directly in the testing phase.

[0036] After replacing the classifier linear layer with the category average feature (feature mean), the model prediction output needs to be further adjusted based on the calibration module to assist the model in predicting the test data.

[0037] Beneficial effects: Compared with the prior art, the category incremental learning method for small sample medical image classification provided by the present invention can solve the problem of increasing disease categories in medical auxiliary models, and it takes up less resources during implementation and has strong applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of medical image data collection according to an embodiment of the present invention;

[0039] Figure 2 This is a flowchart of incremental learning of small sample categories according to an embodiment of the present invention;

[0040] Figure 3 This is a flowchart of medical image classification according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0042] The following embodiments are explained by taking the medical auxiliary system for disease classification as a specific example, and define the new types of disease data collected by the hospital after a period of time as new categories of medical image data, and the known types of disease data collected in an earlier period of time as old categories of medical image data. The classification of diseases mainly includes colds, fevers, fractures, pneumonia, etc. As the number of medical images collected increases, the classification of diseases will also increase; the purpose of medical image classification is to be able to establish a medical auxiliary system to continuously learn new categories of medical image classification tasks, and to be able to give efficient prediction results for any medical image of the learned category. The classifier here is a classification model, the input is a medical image, and the output is the disease type corresponding to the medical image. Medical images mainly include CT images, pathological sections and other image forms of input. Since the collected medical images come from a dynamic environment, a medical classification model covering old categories has been established using previous medical images at an earlier time, but due to the evolution of nature, new types of diseases have emerged. Due to privacy restrictions and information protection constraints between hospitals, medical images of old categories cannot be obtained at this time; at the same time, since training the model takes a lot of time, if medical images of old categories are stored and the model is retrained, additional computing resources are required. Therefore, the present invention proposes to use the existing classifier model and a small sample of medical images of new categories to train a medical image classification model that includes all categories.

[0043] like Figure 1 As shown in FIG. 1 , it is the process of collecting medical image data. First, data collection begins (step 10). Data collection mainly includes two parts: collecting old category medical data and new category medical data:

[0044] Step 11, collecting the currently existing medical image data and recording it as old medical image data;

[0045] Step 12, collecting the tags corresponding to the currently existing medical image data and recording them as old medical image data tags;

[0046] Step 13, after a period of time, collect a small sample of new types of new category medical images, which are recorded as new medical image data;

[0047] Step 14, collect labels corresponding to small sample medical images of new types, and record them as new medical image data labels.

[0048] For the collection of medical image data of old categories, it is necessary to first collect medical images, and then label them based on the doctor's diagnosis results to obtain their corresponding category tags. Then, after a period of time, a small amount of data of medical images of new categories is collected, and the doctor's diagnosis results are similarly used as the corresponding new category tags. Finally, the collected data of medical images of new and old categories and the corresponding tag information are output.

[0049] like Figure 2 As shown, the steps of small sample category incremental learning are as follows: first, start small sample category incremental learning (step 20), then prepare the collected old medical image data and its labels, and initialize the classifier model and model calibration module (step 21). In the old medical image data set, sample small sample incremental learning tasks (step 22), and for each sampled small sample incremental learning task, replace the linear layer W of the classifier by the category feature mean (step 23), then use the graph model to calibrate the output probability of the classifier model, and use all sampled small sample incremental learning tasks to optimize the model (step 24). After that, prepare the collected new medical image data and its labels (step 25), and for each newly added category, replace the linear layer of the model with the category feature mean (step 26). The current model is denoted as M, output this classifier M, and end the small sample category incremental learning stage (step 27).

[0050] like Figure 3 As shown, the medical image data classification test process is as follows: first, start the medical image data classification test (step 30), then prepare the test data to be predicted, including old category medical images and new category medical images (step 31), and then use the model M trained in the previous stage to predict the collected medical image data (step 32), and use the optimized calibration module to calibrate the output probability (step 33), which is used as the classification result of the model on the test data (step 34), and end the medical image classification test process (step 35).

[0051] A category incremental learning system suitable for small sample medical image classification, comprising a data collection module, a small sample category incremental learning module and a medical image classification module;

[0052] Data collection module: collects small sample medical images from different categories and the category labels of the corresponding medical images in different time periods. Small samples refer to each category containing only one or a few samples.

[0053] Small sample category incremental learning module: Use meta-learning technology to construct small sample tasks and train the calibration module. In the real small sample learning task, use the category average feature to replace the linear layer of the existing classifier model, and use the calibration module to complete incremental learning to obtain the classifier model.

[0054] Medical image classification module: Use the classifier model suitable for classification of multiple categories of medical images constructed in the small sample category incremental learning stage to perform classification prediction on the medical images to be tested.

[0055] Obviously, those skilled in the art should understand that the various steps of the incremental learning method for classifying small sample medical images or the various modules of the incremental learning system for classifying small sample medical images of the above-mentioned embodiments of the present invention can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by executable program codes of computing devices, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than that shown here, or they can be made into individual integrated circuit modules respectively, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

Claims

1. A class incremental learning method suitable for small sample medical image classification, characterized in that: It includes the following: Data collection: Collect small sample medical images from different categories and the category labels of the corresponding medical images in different time periods; Small-sample category incremental learning: Use meta-learning technology to construct small-sample tasks and train the calibration module. In real small-sample learning tasks, use the category average feature to replace the linear layer of the existing classifier model, and use the calibration module to complete incremental learning to obtain the classifier model. Medical image classification: Use the classifier model suitable for multi-category medical image classification constructed in the small sample category incremental learning stage to perform classification prediction on the medical images to be tested; The implementation of the small sample category incremental learning includes the following steps: Step 200, initialize the classifier model and the calibration module. The calibration module is a set-to-set function. The calibration module is implemented by a graph model, whose nodes are feature means of the collected old medical image samples. The feature representation of the sample is a low-dimensional representation of the sample obtained by the convolutional neural network of the classifier model. The edge weight of the graph model is the similarity between two nodes. The graph model is represented as a transformation T. Step 201, construct a small sample incremental learning task in the old medical image dataset. The small sample incremental learning task can be described as N classes and K pictures, that is, N classes are sampled in each small sample incremental learning task, and each class has K pictures; the categories of the sampled tasks do not overlap, and the sampling process will be performed multiple times to simulate the classifier model update process; Step 202: for each sampled small sample incremental learning task, firstly train the initial model based on the cross entropy loss function, the input of the model is a single medical image, and the output of the model is the predicted disease type; and decouple the model into a feature embedding module φ and a linear layer W; for each new incremental learning stage, use the feature embedding module to extract the feature φ(x) of the medical image, and calculate the feature mean p belonging to the same class, and use the feature mean p to replace the corresponding class weight in the linear layer; Step 203, after replacing the classifier linear layer with the feature mean, calibrate the replaced linear layer using the graph model T in step 200, record the calibrated linear layer as T(W), and use the calibrated linear layer to replace the original linear layer for image prediction; Predictions are made in the sampled small sample incremental learning tasks, and the feature embedding module φ, the linear layer W, and the graphical model T are optimized based on gradient back propagation using cross entropy loss until the loss converges; the set of optimized feature embedding modules, linear layers, and graphical models is recorded as the small sample incremental learning model for testing in subsequent stages; Step 204, prepare the collected new medical images and their labels, calculate the feature mean p of the same category in the new medical images, use the feature mean p to replace the corresponding category weight in the linear layer, and end the small sample category incremental learning stage.

2. The incremental category learning method for small sample medical image classification according to claim 1, characterized in that: The implementation of the data collection includes the following steps: Step 100, collecting previous medical image data, recorded as old medical image data; Step 101, collecting the tags corresponding to the previous medical image data, and recording them as old medical image data tags; Step 102, after a period of time, collect newly appeared small sample medical images, and record the collected new type of disease images as new medical image data; Step 103, collecting the labels corresponding to the newly appeared small sample medical images and recording them as data labels of new medical images; 3. The incremental category learning method for small sample medical image classification according to claim 1, characterized in that: The medical image refers to data in the image format of pathological sections, CT images, etc. that can be used for diagnosis.

4. The incremental category learning method for small sample medical image classification according to claim 1, characterized in that: The specific steps of medical image classification are: Step 300, preparing medical image data for testing as test data, the test data including old medical images and new medical images; Step 301, predicting the test data using a small sample incremental learning model; Step 302, using the optimized graphical model T to calibrate the prediction probability, specifically, first use the graphical model T to calibrate the classifier linear layer to obtain the calibrated classifier linear layer T(W), use the calibrated classifier linear layer to replace the original linear layer of the classifier, and predict the sample; Step 303: output the classification result of the classifier model calibrated in step 302 on the test data set.

5. A category incremental learning system suitable for small sample medical image classification, characterized by: It includes data collection module, small sample category incremental learning module and medical image classification module; The data collection module is configured to collect small sample medical images from different categories and category labels of the corresponding medical images in different time periods; The small sample category incremental learning module: uses meta-learning technology to construct small sample tasks, trains a calibration module, replaces the linear layer of the existing classifier model with the category average feature in the real small sample learning task, and uses the calibration module to complete incremental learning to obtain a classifier model; The medical image classification module: uses the classifier model suitable for classification of multiple categories of medical images constructed in the small sample category incremental learning stage to perform classification prediction on the medical image to be tested; The implementation of the small sample category incremental learning module includes the following steps: Step 200, initialize the classifier model and the calibration module. The calibration module is a set-to-set function. The calibration module is implemented by a graph model, whose nodes are feature means of the collected old medical image samples. The feature representation of the sample is a low-dimensional representation of the sample obtained by the convolutional neural network of the classifier model. The edge weight of the graph model is the similarity between two nodes. The graph model is represented as a transformation T. Step 201, construct a small sample incremental learning task in the old medical image dataset. The small sample incremental learning task can be described as N classes and K pictures, that is, N classes are sampled in each small sample incremental learning task, and each class has K pictures; the categories of the sampled tasks do not overlap, and the sampling process will be performed multiple times to simulate the classifier model update process; Step 202: for each sampled small sample incremental learning task, firstly train the initial model based on the cross entropy loss function, the input of the model is a single medical image, and the output of the model is the predicted disease type; and decouple the model into a feature embedding module φ and a linear layer W; for each new incremental learning stage, use the feature embedding module to extract the feature φ(x) of the medical image, and calculate the feature mean p belonging to the same class, and use the feature mean p to replace the corresponding class weight in the linear layer; Step 203, after replacing the classifier linear layer with the feature mean, calibrate the replaced linear layer using the graph model T in step 200, record the calibrated linear layer as T(W), and use the calibrated linear layer to replace the original linear layer for image prediction; Predictions are made in the sampled small sample incremental learning tasks, and the feature embedding module φ, the linear layer W, and the graphical model T are optimized based on gradient back propagation using cross entropy loss until the loss converges; the set of optimized feature embedding modules, linear layers, and graphical models is recorded as the small sample incremental learning model for testing in subsequent stages; Step 204, prepare the collected new medical images and their labels, calculate the feature mean p of the same category in the new medical images, use the feature mean p to replace the corresponding category weight in the linear layer, and end the small sample category incremental learning stage.

6. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the category incremental learning method for small sample medical image classification as described in any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the category incremental learning method for small sample medical image classification as described in any one of claims 1 to 4.

Images