Classification model construction method and image classification method based on small sample incremental learning

Abstract

This invention provides a method for constructing a classification model and an image classification method based on few-shot incremental learning. By using initial training samples to train the initial classification model and newly added feature sets corresponding to newly added disease types, the incremental classification model is trained and a preset bias matrix is ​​optimized. When the incremental loss function of the incremental classification model converges, each newly added disease category is bound to a basis vector according to the bias matrix to obtain the incremental classification model. Therefore, this invention overcomes the problem of decreased model recognition accuracy during few-shot incremental learning and the difficulty in obtaining medical image data, as well as the difficulty in training with only a small number of data samples, by using an equiangular basis vector matrix instead of the linear classifier in existing technologies to construct a medical image classification model. This not only avoids the additional computational load brought about by increasing classifier parameters, but also effectively avoids the catastrophic forgetting problem by fixing the basis vectors, thus improving classification accuracy.

Description

Technical Field

[0001] This invention relates to the field of medical image classification technology, and in particular to a classification model construction method and image classification method based on few-sample incremental learning. Background Technology

[0002] In recent years, deep learning has achieved considerable success in the field of medical image analysis. Current researchers hope that models can learn continuously, like humans, without forgetting previously learned types. However, current deep learning methods require large amounts of labeled data to train the model effectively; their performance deteriorates when data is scarce. However, in medical image analysis, some diseases are so rare that only a very limited number of rare disease images are collected, meaning there is insufficient data for training. This easily leads to overfitting and catastrophic forgetting. This has given rise to the research on few-shot incremental learning. Currently, the mainstream method for few-shot incremental learning uses the mean of multiple samples as a prototype to increase robustness and reduce overfitting. Specifically, after the initial model training is complete, the feature extractor in the trained model is frozen, softmax is used as the classifier, and the classifier is updated in subsequent few-shot incremental learning stages to achieve classification. However, as the number of categories increases, the parameters of the classifier also increase, leading to significant resource consumption. Therefore, there is an urgent need for a few-shot incremental learning method to overcome the problem that the classifier parameters of medical image classification models continuously increase during few-shot incremental learning, which leads to a decrease in model recognition accuracy. Summary of the Invention

[0003] This invention provides a method for constructing a classification model and an image classification method based on few-shot incremental learning. It effectively solves the problem in existing technologies where the classifier parameters continuously increase during few-shot incremental learning of medical image classification models, leading to a decrease in model recognition accuracy. This effectively improves the classification accuracy of the medical image classification model after incremental learning.

[0004] One embodiment of the present invention provides a method for constructing a classification model based on few-sample incremental learning, comprising:

[0005] Obtain the network parameters of the first feature extraction network in the initial classification model;

[0006] The initial classification model is a neural network model for identifying disease categories based on medical images, and the classifier in the neural network model is constructed using an isometric basis vector matrix; the isometric basis vector matrix consists of several basis vectors, and each initial disease category identified by the initial classification model is bound to one of the basis vectors in the isometric basis vector matrix;

[0007] Construct an incremental classification model to be trained, and initialize the second feature extraction network of the incremental classification model according to the network parameters;

[0008] The incremental classification model is a neural network model with the same structure as the initial classification model, and it adds the identification of several new disease categories on the basis of the initial classification model.

[0009] Acquire several training data points for each newly added disease category and construct incremental training samples;

[0010] The second feature extraction network is used to extract the feature vector of the training data for each newly added disease category in the incremental training samples, and a new feature set for each newly added disease category is generated.

[0011] Using the initial training samples used to train the initial classification model and the newly added feature set, the incremental classification model is trained and the preset bias matrix is ​​optimized so that the distance between each newly added disease category and an unbound basis vector in the isoangular basis vector matrix is ​​reduced.

[0012] When the incremental loss function of the incremental classification model converges, an optimal bias matrix is ​​generated, and each newly added disease category is bound to a basis vector one by one according to the optimal bias matrix to obtain the trained incremental classification model.

[0013] Furthermore, before obtaining the network parameters of the first feature extraction network of the initial classification model, the following steps are also included:

[0014] Construct the isoangular basis vector matrix;

[0015] Construct a neural network model that uses a linear classifier as the fully connected layer;

[0016] The neural network model is trained using pre-constructed initial training samples to obtain the original classification model; wherein the initial training samples consist of several training data for several initial disease categories;

[0017] Replace the linear classifier in the original classification model with the isogonal basis vector matrix;

[0018] The first feature extraction network in the original classification model is used to extract the feature vector of the training data of each initial disease category in the initial training samples, and an initial feature set for each initial disease category is generated.

[0019] A first classification loss function is constructed based on the cosine similarity between the feature vectors in each of the initial feature sets and the basis vectors in the isoangular basis vector matrix;

[0020] The network parameters of the first feature extraction network are adjusted according to the first classification loss function so that each initial feature set is bound to a basis vector one by one.

[0021] When the first classification loss function converges, the initial classification model is obtained.

[0022] Furthermore, constructing the isoangular basis vector matrix includes:

[0023] Based on the preset number of disease categories to be identified and the feature channel dimension extracted by the first feature extraction network of the initial classification model to be constructed, an isometric basis vector matrix is ​​initialized, along with the basis vectors contained in the isometric basis vector matrix.

[0024] Based on the number of identified disease categories, a cosine similarity threshold is generated, and the following isoangular loss function is constructed based on the cosine similarity threshold:

[0025]

[0026]

[0027]

[0028] in, Let i be the unit vector of the i-th basis vector. Let be the unit vector of the j-th basis vector, and α be the cosine similarity threshold.

[0029] By using the equal angle loss function, the angle of each basis vector is adjusted so that the included angle between any two adjacent basis vectors is equal.

[0030] When the isoangular loss function converges, the constructed isoangular basis vector matrix is ​​obtained.

[0031] Furthermore, during subsequent training, the parameters of the fixed equiangular basis vector matrix remain unchanged. The significance of using the equiangular basis vector matrix instead of the traditional linear classifier is that the fixed basis vectors can effectively alleviate the problem of class prototype shift caused by model training when using parameterized weight vectors, thereby effectively ensuring the classification and recognition performance of the classifier.

[0032] Furthermore, the step of training the incremental classification model using the initial training samples used to train the initial classification model and the newly added feature set, and then optimizing the preset bias matrix, includes:

[0033] Calculate the cosine similarity between each feature vector in the initial feature set of each initial disease category and the bound basis vector;

[0034] Based on the representative feature vector with the highest cosine similarity in each initial feature set, the training sample corresponding to the representative feature vector is obtained from the initial training sample as the representative training sample of the corresponding initial disease category.

[0035] Construct a memory retention set based on representative training samples for each initial disease category;

[0036] Reusing the constructed memory retention set in subsequent training can significantly mitigate the model's catastrophic forgetting of historical information.

[0037] Using the memory retention set and the newly added feature set, the incremental classification model is trained and the preset bias matrix is ​​optimized using the following formula:

[0038]

[0039] Among them, v i v' is the feature vector extracted by the incremental classification model based on the input training samples. i Let γ be the eigenvector adjusted by the bias matrix.

[0040] With the feature extractor and classifier fixed in the incremental stage, the bias matrix is ​​used to change the position of the sample features, making the sample features closer to the basis vector, which alleviates the overfitting problem caused by the limited number of samples to a certain extent.

[0041] Furthermore, the incremental loss function is composed of a second classification loss function and a probability distribution loss function:

[0042] loss = αloss cls +loss KL ;

[0043] Where loss is the incremental loss function, loss cls Let loss be the second classification loss function. KL Let α be the probability distribution loss function, and α be the weight of the classification loss function;

[0044] The second classification loss function is:

[0045]

[0046] Among them, v i w is the feature vector extracted by the incremental classification model based on the input training samples. i The basis vectors bound to the disease categories corresponding to the input training samples;

[0047] By minimizing the loss clsThis allows the sample features to continuously approach the basis vectors, enabling the model to maintain good classification and recognition performance even with only a small number of training samples, thus alleviating the overfitting problem caused by the limited number of samples to some extent.

[0048] The probability distribution loss function is:

[0049]

[0050] Wherein, P(x) is the set of probability distributions predicted by each representative training sample in the memory retention set in the initial classification model, and Q(x) is the set of probability distributions predicted by each representative training sample in the memory retention set in the incremental classification model.

[0051] By minimizing the loss KL This ensures that when new categories are added to the training process, the original parameters in the model are not significantly altered, thus effectively resisting the catastrophic forgetting problem and enabling effective classification and recognition of both the original and newly added categories.

[0052] Furthermore, the step of acquiring several training data points for each newly added disease category and constructing incremental training samples includes:

[0053] Based on a preset proportion of the total number of training data in the initial training samples, training data for several new disease categories that are less than the total number are obtained to generate training samples to be processed.

[0054] Two training data points are sequentially extracted from the training samples to be processed, and several preset image enhancement strategies are randomly adopted to perform image enhancement processing on the two currently extracted training data points to generate one enhanced training data point.

[0055] This data augmentation strategy expands the sample data volume, which can effectively reduce overfitting caused by insufficient sample size.

[0056] After all the training data in the training samples to be processed have undergone image enhancement processing, the incremental training samples are generated based on the generated enhanced training data and the training samples to be processed.

[0057] Furthermore, after obtaining the trained incremental classification model, the following steps are also included:

[0058] Calculate the cosine similarity between each feature vector in the new feature set of each newly added disease category and its bound basis vector;

[0059] Based on the representative feature vector with the highest cosine similarity in each newly added feature set, the training sample corresponding to the representative feature vector is obtained from the incremental training sample as the representative training sample of the newly added disease category.

[0060] The memory retention set is updated based on representative training samples for each newly added disease category, so that when incrementally learning the incremental classification model, the updated memory retention set is used to construct the corresponding probability distribution loss function.

[0061] Furthermore, after obtaining the trained incremental classification model, the following steps are also included:

[0062] From the initial training samples and the incremental training samples, a predetermined proportion of training data is extracted as test data to construct a test dataset;

[0063] The normalized class probability value of the similarity between each test data point in the test dataset and each basis vector in the isoangular basis vector matrix is ​​calculated using the following formula:

[0064]

[0065] Among them, v i The features obtained after the test data has been processed by the feature extractor, y i w is the true label for the test data. i Let P(y = y) be the basis vector. i │v i The normalized class probability value is the similarity between the test data and each basis vector in the isogonal basis vector matrix.

[0066] The classification accuracy of the incremental classification model is calculated based on the normalized class probability values ​​of the similarity between the test data and each basis vector in the isoangular basis vector matrix.

[0067] Another embodiment of the present invention provides an image classification method, comprising:

[0068] Acquire the medical image to be identified;

[0069] The medical image is input into a medical image classification model so that the medical image classification model can identify the medical image and determine the image category corresponding to the medical image; wherein, the medical image classification model is constructed by the classification model construction method based on few-sample incremental learning described in the above embodiments of the invention; the image category is classified according to the disease category to which the medical image belongs.

[0070] Another embodiment of the present invention provides an image classification method apparatus, comprising:

[0071] The data acquisition module is used to acquire the medical images to be identified.

[0072] The category recognition module is used to input the medical image into the medical image classification model, so that the medical image classification model can recognize the medical image and determine the image category corresponding to the medical image; wherein, the medical image classification model is constructed by the classification model construction method based on few-sample incremental learning described in the above embodiments of the invention; the image category is classified according to the disease category to which the medical image belongs.

[0073] The following benefits can be obtained by implementing the present invention:

[0074] This invention provides a method for constructing a classification model and an image classification method based on few-shot incremental learning. The method involves constructing a second feature extraction network for the incremental classification model based on the network parameters of the first feature extraction network of the initial classification model during incremental learning. This second feature extraction network is then used to extract features from training samples of newly added disease types, resulting in a new feature set corresponding to each new disease type. The incremental classification model is then trained using the initial training samples and the new feature sets, and a preset bias matrix is ​​optimized. When the incremental loss function of the incremental classification model converges, each new disease category is individually bound to a basis vector, resulting in a trained incremental classification model. Therefore, this invention overcomes the problem of category prototype shift caused by model training during few-shot incremental learning, which leads to a decrease in model recognition accuracy, by using an isoangular basis vector matrix instead of the linear classifier in existing technologies to construct a medical image classification model. By fixing the parameters of the isoangular basis vector matrix during incremental learning, the additional computational load caused by increasing classifier parameters is avoided, and overfitting is effectively resisted, thus improving classification accuracy. Attached Figure Description

[0075] Figure 1 This is a flowchart illustrating a classification model construction method based on few-sample incremental learning, provided by an embodiment of the present invention.

[0076] Figure 2 This is a schematic diagram of an image enhancement strategy provided in an embodiment of the present invention.

[0077] Figure 3 This is a schematic diagram of adjusting the position of the eigenvector using a bias matrix according to an embodiment of the present invention.

[0078] Figure 4 This is a schematic flowchart of an image classification method provided in an embodiment of the present invention.

[0079] Figure 5 This is a schematic diagram of the structure of an image classification device provided in an embodiment of the present invention.

[0080] Figure 6 This is a flowchart illustrating a category recognition module provided in an embodiment of the present invention.

[0081] Figure 7 This is a schematic diagram of the structure of an incremental classification model provided in an embodiment of the present invention.

[0082] Figure 8 This is a schematic diagram of the structure of a first feature extraction network provided in an embodiment of the present invention. Detailed Implementation

[0083] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0084] See Figure 1 This is a flowchart illustrating a classification model construction method based on few-sample incremental learning according to an embodiment of the present invention, including:

[0085] S1. Obtain the network parameters of the first feature extraction network in the initial classification model;

[0086] The initial classification model is a neural network model for identifying disease categories based on medical images, and the classifier in the neural network model is constructed using an isometric basis vector matrix; the isometric basis vector matrix consists of several basis vectors, and each initial disease category identified by the initial classification model is bound to one of the basis vectors in the isometric basis vector matrix;

[0087] In one embodiment of the present invention, the initial classification model is a neural network model used to identify pneumonia categories based on the morphology of lesions in CT images. Specifically, the initial disease categories identified by the initial classification model are: bronchiolitis, lobar pneumonia, bronchopneumonia, interstitial pneumonia, and lung abscess.

[0088] Furthermore, such as Figure 8 As shown, the first feature extraction network has four convolutional blocks. Each convolutional block contains two convolution (conv) operations, two batch normalization (bn) operations, and two activation (relu) operations. Each convolutional block performs two convolution operations.

[0089] Preferably, before obtaining the network parameters of the first feature extraction network of the initial classification model, the method further includes:

[0090] S01. Construct the isoangular basis vector matrix;

[0091] In a preferred embodiment of the present invention, a classifier is constructed before constructing the initial classification model, which is to be used as the initial classification model to be constructed.

[0092] Preferably, constructing the isoangular basis vector matrix includes:

[0093] S011. Based on the preset number of disease categories to be identified and the feature channel dimension extracted by the first feature extraction network of the initial classification model to be constructed, initialize an isometric basis vector matrix and the basis vectors contained in the isometric basis vector matrix.

[0094] In a preferred embodiment of the present invention, the number of disease categories identified is greater than the number of initial disease categories identified by the initial classification model, reserving the number of newly added disease categories that may be added later by performing small-sample incremental learning on the initial classification model.

[0095] Specifically, if the number of disease categories identified is N, and the number of initial disease categories identified by the initial classification model is M, then the number of new disease categories added by subsequent small-sample incremental learning of the initial classification model should be within the range of (0, NM). That is, in this embodiment, the initial disease categories are bronchiolitis, lobar pneumonia, bronchopneumonia, interstitial pneumonia, and lung abscess, so the number of initial disease categories is 5. Therefore, the number of new disease categories that can be added by subsequent small-sample incremental learning of the initial classification model should be less than N-5.

[0096] Furthermore, if the number of disease categories to be identified is N, and the feature channel dimension extracted by the first feature extraction network of the initial classification model to be constructed is d, then the initialized isogonal basis vector matrix is ​​W∈R. d*N W = {w1, w2, ..., w N}, w i ∈R d , where w i It is a basis vector.

[0097] S012. Based on the number of identified disease categories, generate a cosine similarity threshold, and construct the following isoangular loss function based on the cosine similarity threshold:

[0098]

[0099]

[0100]

[0101] in, Let i be the unit vector of the i-th basis vector. Let be the unit vector of the j-th basis vector, and α be the cosine similarity threshold.

[0102] In a preferred embodiment of the present invention, if the number of identified disease categories is N, then the cosine similarity threshold α is:

[0103] Specifically, the formula for the cosine similarity between basis vectors is:

[0104]

[0105] S013. Adjust the angle of each basis vector using the equal angle loss function so that the included angle between any two adjacent basis vectors is equal.

[0106] S014. When the isoangular loss function converges, the constructed isoangular basis vector matrix is ​​obtained.

[0107] In a preferred embodiment of the present invention, w in the isoangular basis vector matrix is ​​adjusted according to the isoangular loss function. i The parameters are set such that the cosine similarity between any two adjacent basis vectors is not lower than the cosine similarity threshold α. If it is lower than the cosine similarity threshold α, training is interrupted and the matrix is ​​saved. The final goal is to find the optimal separation angle where the angles between basis vectors are all within the threshold.

[0108] It's easy to understand why an isoangular basis vector matrix is ​​pre-constructed before building the initial classification model. Firstly, the isoangular basis vector matrix is ​​independent of the medical image classification model, requiring only the number of categories. It eliminates the need to measure the similarity between different samples, significantly reducing computational cost and effectively addressing the problem of decreased classification accuracy caused by category prototype shifts during model training, thereby improving the model's classification and recognition performance. Secondly, under the constraint of a cosine similarity threshold, it's better to maximize the spacing between the basis vectors initially. This prevents misclassification when binding basis vectors to disease categories later, as the distances between basis vectors may be too close.

[0109] S02. Construct a neural network model that uses a linear classifier as a fully connected layer;

[0110] In a preferred embodiment of the present invention, a neural network model using the softmax function as a classifier is constructed.

[0111] S03. The neural network model is trained using pre-constructed initial training samples to obtain the original classification model; wherein, the initial training samples consist of several training data of several initial disease categories;

[0112] In a preferred embodiment of the present invention, the model is constructed based on several CT images of an initial disease category, namely: bronchiolitis, lobar pneumonia, bronchopneumonia, interstitial pneumonia, and lung abscess, and their corresponding annotations. It should be noted that the dimensions of the CT images all conform to the input image size of the neural network model, i.e., 224×224.

[0113] S04. Replace the linear classifier in the original classification model with the isoangular basis vector matrix;

[0114] In a preferred embodiment of the present invention, the softmax classifier in the original classification model is replaced with the isoangular basis vector matrix.

[0115] S05. Use the first feature extraction network in the original classification model to extract the feature vector of the training data of each initial disease category in the initial training samples, and generate the initial feature set of each initial disease category.

[0116] In a preferred embodiment of the present invention, after generating the original classification model, the initial training samples are input into the first feature extraction network so that the number of channels in the input CT image becomes 64 after the first convolution, and the feature map at this time is denoted as feature1. Then, after a max pooling operation, the number of channels remains unchanged, but the image size becomes half of the original, and the feature map at this time is denoted as feature2. After convolution through convolution block 1, the image dimension remains unchanged, and the feature map at this time is denoted as feature3. After convolution through convolution block 2, the number of channels becomes 128, and the feature map at this time is denoted as feature4. After convolution through convolution block 3, the number of channels becomes 256, and the feature map at this time is denoted as feature5. After convolution through convolution block 4, the number of channels becomes 512, and the feature map at this time is denoted as feature6. Finally, after an average pooling layer, the feature map dimension becomes 1×1×512, denoted as feature7. The feature vectors of feature7 of each initial disease category training sample are combined to form the initial feature set.

[0117] S06. Construct a first classification loss function based on the cosine similarity between the feature vectors in each of the initial feature sets and the basis vectors in the isoangular basis vector matrix;

[0118] In a preferred embodiment of the present invention, the first classification loss function is:

[0119]

[0120] Among them, v i w is the feature vector in the initial feature set. i For the closest to v iBasis vectors.

[0121] Specifically, classification is performed by multiplying the feature vectors in the initial feature set, i.e., the feature vectors of feature7, with the isoangular basis vector matrix.

[0122] S07. Adjust the network parameters of the first feature extraction network according to the first classification loss function, so that each initial feature set is bound to a basis vector one by one;

[0123] S08. When the first classification loss function converges, the initial classification model is obtained.

[0124] In a preferred embodiment of the present invention, when the first classification loss function converges, an initial classification model is obtained that can be used to identify pneumonia categories based on the morphology of lesions in CT images.

[0125] It should be further explained that, in order to improve the efficiency of building the initial classification model, a neural network model based on an isoangular basis vector matrix as a classifier can also be directly constructed. The neural network model is still trained using the pre-constructed initial training samples. When the first classification loss function converges, each initial disease category is bound to a basis vector in the isoangular basis vector matrix to obtain the same initial classification model.

[0126] S2. Construct the incremental classification model to be trained, and initialize the second feature extraction network of the incremental classification model according to the network parameters;

[0127] The incremental classification model is a neural network model with the same structure as the initial classification model, and it adds the identification of several new disease categories on the basis of the initial classification model.

[0128] In one embodiment of the present invention, the initial classification model is subjected to small-sample incremental learning to construct an incremental classification model, so that the trained incremental classification model can identify several new disease categories on the basis of the initial classification model. In this small-sample incremental learning process, the new disease categories added on the basis of the initial classification model are: intervertebral disc degeneration, bone hyperplasia, prostatic calcification, benign prostatic hyperplasia, and pelvic effusion.

[0129] It is easy to understand that when constructing the incremental classification model, the network parameters of the first feature extraction network of the initial classification model are loaded into the second feature extraction network of the incremental classification model. The purpose is to prevent the network parameters of the second feature extraction network from being inconsistent with the network parameters of the first feature extraction network during the training of the incremental classification model, which could lead to catastrophic forgetting.

[0130] S3. Obtain several training data for each newly added disease category and construct incremental training samples;

[0131] In one embodiment of the present invention, a small number of CT images related to intervertebral disc degeneration, bone hyperplasia, prostatic calcification, benign prostatic hyperplasia, and pelvic effusion are acquired and labeled respectively to construct incremental data samples for training the incremental classification model in this small sample incremental learning. It should be noted that the training data in the incremental data samples does not overlap with the training data in the initial training samples.

[0132] Preferably, several training data points are obtained for each newly added disease category to construct incremental training samples, including:

[0133] S31. Based on a preset ratio of the total number of training data in the initial training samples, obtain training data for several newly added disease categories that are less than the total number, and generate training samples to be processed.

[0134] In one embodiment of the present invention, five CT images of intervertebral disc degeneration, bone hyperplasia, prostatic calcification, benign prostatic hyperplasia, and pelvic effusion are acquired and labeled respectively to construct the training samples to be processed for this small sample incremental learning.

[0135] S32. Sequentially extract two training data from the training samples to be processed, randomly adopt several preset image enhancement strategies, perform image enhancement processing on the two currently extracted training data, and generate one enhanced training data.

[0136] In a preferred embodiment of the present invention, such as Figure 2 As shown, there are three preset image enhancement strategies in this embodiment, which are used to perform random grayscale transformation and horizontal inversion on CT images to generate corresponding enhancement training data.

[0137] Specifically, the image enhancement strategy can be represented as:

[0138] x k =λx i +(1-λ)x j ;

[0139] x i x j For two CT images of the same type in the training sample to be processed, x k The image is a CT image generated after image enhancement strategy processing. λ is a manually set hyperparameter. This data enhancement strategy expands the amount of sample data and can effectively reduce overfitting caused by insufficient sample size.

[0140] S33. After all the training data in the training samples to be processed have undergone image enhancement processing, the incremental training samples are generated based on the generated enhanced training data and the training samples to be processed.

[0141] In a preferred embodiment of the present invention, an incremental dataset for this small-shot incremental learning is constructed based on several enhanced training data and training samples to be processed, and several training data are obtained from the incremental dataset to construct 5way-5shot incremental data samples.

[0142] It is easy to understand that, in order to improve the incremental learning efficiency of the medical image classification model and save the time of building the training dataset, and to avoid the problems of overfitting and catastrophic forgetting in the small-sample incremental learning process of the medical image classification model due to the difficulty in obtaining CT images of some rare diseases and the difficulty in building a sufficiently large training dataset, the image enhancement processing of the training samples to be processed is performed to expand the training data and generate the incremental training samples. This can help the incremental classification model obtain more features during the training process, so that the incremental classification model generates more stable feature representations during the training process, and further avoid catastrophic forgetting in the small-sample incremental learning process.

[0143] S4. Use the second feature extraction network to extract the feature vector of the training data for each newly added disease category in the incremental training samples, and generate a new feature set for each newly added disease category.

[0144] In one embodiment of the present invention, the second feature extraction network uses the same network parameters as the first feature extraction network. Therefore, the second feature extraction network performs a series of convolutional pooling operations on the input CT images of the newly added disease categories to generate feature images with dimensions of 1×1×512. Then, based on the feature vectors of the feature images of each newly added disease category, a new feature set corresponding to the newly added disease category is constructed.

[0145] S5. Using the initial training samples used to train the initial classification model and the newly added feature set, train the incremental classification model and optimize the preset bias matrix so that the distance between each newly added disease category and an unbound basis vector in the isoangular basis vector matrix is ​​reduced.

[0146] In one embodiment of the present invention, since the network parameters of the second feature extraction network of the incremental classification model are fixed and the isoangular basis vector matrix used as the classifier is not modified, in order to bind the new disease type to an unbound basis vector in the isoangular basis vector matrix, so that the incremental classification model can identify the new disease type, a bias matrix is ​​preset according to the feature channel dimension extracted by the second feature extraction network. This bias matrix is ​​used to change the position of the feature vectors of each training data in the new feature set, so that the distance between each new disease category and an unbound basis vector in the isoangular basis vector matrix is ​​reduced, thereby effectively reducing the overfitting problem of the model when there is only a small amount of training data for each new category.

[0147] Specifically, such as Figure 3 As shown, the bias matrix changes the position of the feature vectors of each training data in the newly added feature set using the following formula:

[0148]

[0149] v' i The position of the eigenvector after adjustment by the bias matrix, where γ is the bias matrix, and v i This refers to the feature vectors of each training data point in the newly added feature set.

[0150] Preferably, the step of training the incremental classification model and optimizing the preset bias matrix using the initial training samples used to train the initial classification model and the newly added feature set includes:

[0151] S51. Calculate the cosine similarity between each feature vector in the initial feature set of each initial disease category and the bound basis vector.

[0152] In a preferred embodiment of the present invention, the cosine similarity between each feature vector in the initial feature set corresponding to bronchiolitis, lobar pneumonia, bronchopneumonia, interstitial pneumonia, and lung abscess and their respective bound basis vectors is calculated.

[0153] S52. Based on the representative feature vector with the largest cosine similarity in each initial feature set, obtain the training sample corresponding to the representative feature vector from the initial training samples as the representative training sample of the initial disease category.

[0154] In a preferred embodiment of the present invention, representative training samples for each initial disease category are obtained using the following formula:

[0155]

[0156] Among them, g i To represent the training samples, w iFor each initial disease category, the basis vector v is... j For each initial disease category, there is a feature vector in the initial feature set.

[0157] S53. Construct a memory retention set based on the representative training samples of each initial disease category;

[0158] In a preferred embodiment of the present invention, after generating the initial classification model and before performing small-sample incremental learning, the memory retention set contains: representative training samples of bronchiolitis, representative training samples of lobar pneumonia, representative training samples of bronchopneumonia, representative training samples of interstitial pneumonia, and representative training samples of lung abscess.

[0159] S54. Using the memory retention set and the newly added feature set, train the incremental classification model and optimize the preset bias matrix using the following formula:

[0160]

[0161] Among them, v' i Let γ be the feature vector in the newly added feature set after adjustment by the bias matrix, and v be the bias matrix. i The feature vectors in the newly added feature set.

[0162] In a preferred embodiment of the present invention, in order to improve the incremental learning efficiency of the medical image classification model and save the time of constructing the training dataset, and to avoid the difficulty in constructing a sufficiently large training dataset due to the scarcity of CT images for some rare diseases, the memory retention set is constructed to avoid the problem of catastrophic forgetting during incremental learning of the incremental classification model using a small number of training samples. This allows incremental learning of the incremental classification model to be completed with a small number of training samples.

[0163] S6. When the incremental loss function of the incremental classification model converges, the optimal bias matrix is ​​generated, and each newly added disease category is bound to a basis vector one by one according to the optimal bias matrix to obtain the trained incremental classification model.

[0164] In one embodiment of the present invention, such as Figure 7The diagram shows the structure of the incremental classification model. It should be noted that by optimizing the incremental loss function of the incremental classification model, the parameters of the bias matrix are adjusted, and the optimal parameters are obtained when the incremental loss function converges, generating the optimal bias matrix. This means that the distance between each newly added disease category and an unbound basis vector in the isoangular basis vector matrix is ​​minimized. Therefore, each newly added disease category is bound to a basis vector one-to-one according to the optimal bias matrix, enabling the isoangular basis vector matrix, acting as the classifier, to identify the newly added disease type, thus obtaining the trained incremental classification model.

[0165] Preferably, the incremental loss function is composed of a second classification loss function and a probability distribution loss function:

[0166] loss = αloss cls +loss KL ;

[0167] Where loss is the incremental loss function, loss cls Let loss be the second classification loss function. KL Let α be the probability distribution loss function, and α be the weight of the classification loss function;

[0168] The second classification loss function is:

[0169]

[0170] Among them, v i w is the feature vector extracted by the incremental classification model based on the input training samples. i The basis vectors to be bound to the disease categories corresponding to the input training samples;

[0171] By minimizing the loss cls This allows the sample features to continuously approach the basis vectors, enabling the model to maintain good classification and recognition performance even with only a small number of training samples, thus alleviating the overfitting problem caused by the limited number of samples to some extent.

[0172] In a preferred embodiment of the present invention, due to the adjusted feature vector v′ i The position is determined by the bias matrix and the original eigenvector v i The dot product is obtained, and the feature vector v′ after displacement is calculated using the second classification loss function. i The distance between the eigenvectors and the basis vectors guides the adjustment direction of the parameters of the bias matrix, so that the integrated eigenvector v′ i The distance between the vector and the basis vector to be bound gradually decreases.

[0173] The probability distribution loss function is:

[0174]

[0175] Wherein, P(x) is the set of probability distributions predicted by each representative training sample in the memory retention set in the initial classification model, and Q(x) is the set of probability distributions predicted by each representative training sample in the memory retention set in the incremental classification model.

[0176] In a preferred embodiment of the present invention, if the number of initial disease categories identified by the initial classification model is N, and the number of disease categories identified by the incremental classification model is N+5 (i.e., 5 new disease categories are added), then the probability distribution of the predicted labels representing training samples in the prediction memory retention set of the initial classification model is P(x)={p1,p2,p3,…,p N The probability distribution of the predicted labels representing training samples in the memory retention set of the incremental classification model is Q(x)={q1,q2,q3,…,q N ,q N+1 …q N+5}, then only take q1 to q from Q(x) N After normalizing the probability distribution, it is calculated and compared with P(x).

[0177] In this embodiment, the probability distributions of the predicted labels for five representative training samples of bronchiolitis, lobar pneumonia, bronchopneumonia, interstitial pneumonia, and lung abscess, predicted by the initial classification model, are compared with the probability distributions of the predicted labels for five representative training samples of the same diseases, predicted by the incremental classification model.

[0178] It is easy to understand that, as the number of disease categories identified by the incremental classification model increases, the distillation loss is calculated using the KL divergence formula as the loss function based on the probability distribution of each representative training sample in the memory retention set in the incremental classification model and the initial classification model. Regularization is used to constrain the classification, which ensures that the model will not significantly change the original parameters when new categories are added. This effectively resists the catastrophic forgetting problem of the model, and thus can effectively classify and identify data of both the original category and the newly added category.

[0179] Preferably, after obtaining the trained incremental classification model, the following steps are also included:

[0180] S61. Calculate the cosine similarity between each feature vector in the new feature set of each newly added disease category and the basis vector it is bound to.

[0181] In a preferred embodiment of the present invention, the cosine similarity between each feature vector in the newly added feature set corresponding to intervertebral disc degeneration, bone hyperplasia, prostatic calcification, benign prostatic hyperplasia, and pelvic effusion and their respective bound basis vectors is calculated.

[0182] S62. Based on the representative feature vector with the largest cosine similarity in each newly added feature set, obtain the training sample corresponding to the representative feature vector from the incremental training samples as the representative training sample of the newly added disease category.

[0183] In a preferred embodiment of the present invention, representative training samples for each newly added disease category are obtained using the following formula:

[0184]

[0185] Among them, g i To represent the training samples, w i For each newly added disease category, v is the basis vector. j For each newly added disease category, there is a feature vector in the newly added feature set.

[0186] S63. Update the memory retention set according to the representative training samples of each newly added disease category, so that when performing incremental learning on the incremental classification model, the updated memory retention set is used to construct the corresponding probability distribution loss function.

[0187] In a preferred embodiment of the present invention, after the incremental classification model is trained, the memory retention set is updated according to the representative training samples of each newly added disease category, so that when performing few-shot incremental learning on the incremental classification model, the updated memory retention set is used to construct the corresponding probability distribution loss function, thereby avoiding forgetting the newly added disease categories learned in this few-shot incremental learning when performing few-shot incremental learning on the incremental classification model again.

[0188] It's easy to understand that when medical image classification models perform few-shot incremental learning, they always update the memory retention set after obtaining the results of the previous few-shot incremental learning stage to avoid catastrophic forgetting. That is, in subsequent few-shot incremental learning, let N... C The total training samples constructed for subsequent incremental learning, D t Let M be the incremental training samples acquired in the current stage, and M be the memory retention module. Then N C =|D t |+|M|.

[0189] Preferably, after obtaining the trained incremental classification model, the following steps are also included:

[0190] S64. From the initial training samples and the incremental training samples, a preset proportion of training data is extracted as test data to construct a test dataset;

[0191] In a preferred embodiment of the present invention, it is readily understood that if small-sample incremental learning is subsequently performed on the incremental classification model, a preset proportion of training data is extracted from the initial training samples, the small-sample incremental training samples, and the incremental training samples in subsequent stages as test data to construct a test dataset.

[0192] S65. Calculate the normalized class probability value of the similarity between each test data point in the test dataset and each basis vector in the isoangular basis vector matrix using the following formula:

[0193]

[0194] Among them, v i The features obtained after the test data has been processed by the feature extractor, y i w is the true label for the test data. i Let P(y = y) be the basis vector. i │v i The normalized class probability value is the similarity between the test data and each basis vector in the isogonal basis vector matrix.

[0195] In a preferred embodiment of the present invention, P(y=y) i │v i The similarity between the test data and each basis vector in the isoangular basis vector matrix is ​​the similarity between the test data and each basis vector. The category with the highest similarity is the disease category to which the input test data belongs.

[0196] S66. Calculate the classification accuracy of the incremental classification model based on the normalized class probability values ​​of the similarity between the test data and each basis vector in the isoangular basis vector matrix.

[0197] In a preferred embodiment of the present invention, it is readily understood that if the incremental classification model is subsequently subjected to few-shot incremental learning, the classification accuracy of the subsequent incremental classification model is still calculated in the same way, so as to evaluate whether the medical image classification model has overfitting and catastrophic forgetting during the few-shot incremental learning process.

[0198] This invention provides a method for constructing a classification model based on few-shot incremental learning. During few-shot incremental learning of a pre-trained initial classification model, a second feature extraction network for the incremental classification model is constructed based on the network parameters of the first feature extraction network of the initial model. This second feature extraction network is then used to extract features from training samples of newly added disease types, obtaining new feature sets corresponding to each new disease type. Subsequently, the initial training samples used to train the initial classification model and the new feature sets are used to train the incremental classification model and optimize a preset bias matrix. When the incremental loss function of the incremental classification model converges, each new disease category is individually bound to a basis vector, resulting in a fully trained incremental classification model. Therefore, this invention overcomes the problem of category prototype shifting during model training in few-shot incremental learning, which leads to a decrease in model recognition accuracy, by using an isoangular basis vector matrix instead of the linear classifier in existing technologies to construct a medical image classification model. By fixing the parameters of the isoangular basis vector matrix during incremental learning, not only is the additional computational load caused by increasing classifier parameters avoided, but overfitting is also effectively resisted, thus improving classification accuracy.

[0199] See Figure 4 The following is a flowchart illustrating a medical image classification method according to another embodiment of the present invention, comprising:

[0200] Step a: Obtain the medical image to be identified;

[0201] Step b: Input the medical image into the medical image classification model so that the medical image classification model can identify the medical image and determine the image category corresponding to the medical image; wherein, the medical image classification model is constructed by the classification model construction method based on few-sample incremental learning described in the above embodiments of the invention; the image category is classified according to the disease category to which the medical image belongs.

[0202] See Figure 5 as well as Figure 6 The following are schematic diagrams and flowcharts of a medical image classification device according to another embodiment of the present invention, including:

[0203] The data acquisition module is used to acquire the medical images to be identified.

[0204] The category recognition module is used to input the medical image into the medical image classification model, so that the medical image classification model can recognize the medical image and determine the image category corresponding to the medical image; wherein, the medical image classification model is constructed by the classification model construction method based on few-sample incremental learning described in the above embodiments of the invention; the image category is classified according to the disease category to which the medical image belongs.

[0205] This invention provides a medical image classification method and apparatus. The medical image classification model is constructed using the classification model construction method based on few-shot incremental learning described in the above-mentioned embodiments. This method involves constructing a second feature extraction network for the incremental classification model based on the network parameters of the first feature extraction network of the initial classification model during few-shot incremental learning. The second feature extraction network is then used to extract features from the training samples of newly added disease types, obtaining new feature sets corresponding to each new disease type. Subsequently, the initial training samples used to train the initial classification model and the new feature sets are used to train the incremental classification model and optimize a preset bias matrix. When the incremental loss function of the incremental classification model converges, each new disease category is bound to a basis vector, resulting in a trained incremental classification model. Therefore, this invention overcomes the problem of category prototype shifting during model training in the few-shot incremental learning process, which leads to a decrease in model recognition accuracy, by using an equiangular basis vector matrix instead of the linear classifier in the prior art to construct the medical image classification model. By fixing the parameters of the isoangular basis vector matrix during incremental learning, not only is the additional computational cost caused by increasing the classifier parameters avoided, but overfitting can also be effectively resisted, thus improving classification accuracy.

[0206] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for constructing a classification model based on few-sample incremental learning, characterized in that, include: Obtain the network parameters of the first feature extraction network in the initial classification model; The initial classification model is a neural network model for identifying disease categories based on medical images, and the classifier in the neural network model is constructed using an isometric basis vector matrix; the isometric basis vector matrix consists of several basis vectors, and each initial disease category identified by the initial classification model is bound to one of the basis vectors in the isometric basis vector matrix; Construct an incremental classification model to be trained, and initialize the second feature extraction network of the incremental classification model according to the network parameters; The incremental classification model is a neural network model with the same structure as the initial classification model, and it adds the identification of several new disease categories on the basis of the initial classification model. Acquire several training data points for each newly added disease category and construct incremental training samples; The second feature extraction network is used to extract the feature vector of the training data for each newly added disease category in the incremental training samples, and a new feature set for each newly added disease category is generated. Using the initial training samples used to train the initial classification model and the newly added feature set, the incremental classification model is trained and the preset bias matrix is ​​optimized so that the distance between each newly added disease category and an unbound basis vector in the isoangular basis vector matrix is ​​reduced. When the incremental loss function of the incremental classification model converges, an optimal bias matrix is ​​generated, and each newly added disease category is bound to a basis vector one by one according to the optimal bias matrix to obtain the trained incremental classification model.

2. The classification model construction method based on few-sample incremental learning as described in claim 1, characterized in that, Before obtaining the network parameters of the first feature extraction network of the initial classification model, the method further includes: Construct the isoangular basis vector matrix; Construct a neural network model that uses a linear classifier as the fully connected layer; The neural network model is trained using pre-constructed initial training samples to obtain the original classification model; wherein the initial training samples consist of several training data for several initial disease categories; Replace the linear classifier in the original classification model with the isogonal basis vector matrix; The first feature extraction network in the original classification model is used to extract the feature vector of the training data of each initial disease category in the initial training samples, and an initial feature set for each initial disease category is generated. A first classification loss function is constructed based on the cosine similarity between the feature vectors in each of the initial feature sets and the basis vectors in the isoangular basis vector matrix; The network parameters of the first feature extraction network are adjusted according to the first classification loss function so that each initial feature set is bound to a basis vector one by one. When the first classification loss function converges, the initial classification model is obtained.

3. The classification model construction method based on few-sample incremental learning as described in claim 2, characterized in that, The construction of the isoangular basis vector matrix includes: Based on the preset number of disease categories to be identified and the feature channel dimension extracted by the first feature extraction network of the initial classification model to be constructed, an isometric basis vector matrix is ​​initialized, along with the basis vectors contained in the isometric basis vector matrix. Based on the number of identified disease categories, a cosine similarity threshold is generated, and the following isoangular loss function is constructed based on the cosine similarity threshold: ， ， ； in, Let i be the unit vector of the i-th basis vector. Let j be the unit vector of the j-th basis vector. The cosine similarity threshold is used. By using the equal angle loss function, the angle of each basis vector is adjusted so that the included angle between any two adjacent basis vectors is equal. When the isoangular loss function converges, the constructed isoangular basis vector matrix is ​​obtained.

4. The classification model construction method based on few-sample incremental learning as described in claim 3, characterized in that, The step of training the incremental classification model and optimizing the preset bias matrix using the initial training samples used to train the initial classification model and the newly added feature set includes: Calculate the cosine similarity between each feature vector in the initial feature set of each initial disease category and the bound basis vector; Based on the representative feature vector with the highest cosine similarity in each initial feature set, the training sample corresponding to the representative feature vector is obtained from the initial training sample as the representative training sample of the corresponding initial disease category. Construct a memory retention set based on representative training samples for each initial disease category; Using the memory retention set and the newly added feature set, the incremental classification model is trained and the preset bias matrix is ​​optimized using the following formula: ； in, This refers to the feature vector extracted by the incremental classification model based on the input training samples. The eigenvectors are adjusted using the bias matrix. Let be the bias matrix.

5. The classification model construction method based on few-sample incremental learning as described in claim 4, characterized in that, The incremental loss function consists of the second classification loss function and the probability distribution loss function: ; in, Let the incremental loss function be... The second classification loss function is... The probability distribution loss function is... The weights of the classification loss function; The second classification loss function is: ； in, This refers to the feature vector extracted by the incremental classification model based on the input training samples. The basis vectors bound to the disease categories corresponding to the input training samples; The probability distribution loss function is: ； in, The memory retention set is the set of probability distributions predicted by each representative training sample in the initial classification model. The memory retention set represents the set of probability distributions predicted by the training samples in the incremental classification model.

6. The classification model construction method based on few-sample incremental learning as described in claim 5, characterized in that, The step of acquiring several training data points for each newly added disease category and constructing incremental training samples includes: Based on a preset proportion of the total number of training data in the initial training samples, training data for several new disease categories that are less than the total number are obtained to generate training samples to be processed. Two training data points are sequentially extracted from the training samples to be processed, and several preset image enhancement strategies are randomly adopted to perform image enhancement processing on the two currently extracted training data points to generate one enhanced training data point. After all the training data in the training samples to be processed have undergone image enhancement processing, the incremental training samples are generated based on the generated enhanced training data and the training samples to be processed.

7. The classification model construction method based on few-sample incremental learning as described in claim 6, characterized in that, After obtaining the trained incremental classification model, the following is also included: Calculate the cosine similarity between each feature vector in the new feature set of each newly added disease category and its bound basis vector; Based on the representative feature vector with the highest cosine similarity in each newly added feature set, the training sample corresponding to the representative feature vector is obtained from the incremental training sample as the representative training sample of the newly added disease category. The memory retention set is updated based on representative training samples for each newly added disease category, so that when incrementally learning the incremental classification model, the updated memory retention set is used to construct the corresponding probability distribution loss function.

8. The classification model construction method based on few-sample incremental learning as described in claim 7, characterized in that, After obtaining the trained incremental classification model, the following is also included: From the initial training samples and the incremental training samples, a predetermined proportion of training data is extracted as test data to construct a test dataset; The normalized class probability value of the similarity between each test data point in the test dataset and each basis vector in the isoangular basis vector matrix is ​​calculated using the following formula: ； in, The features obtained after the test data has been processed by the feature extractor. For the true labels of the test data, As basis vectors, The normalized class probability value is the similarity between the test data and each basis vector in the isogonal basis vector matrix. The classification accuracy of the incremental classification model is calculated based on the normalized class probability values ​​of the similarity between the test data and each basis vector in the isoangular basis vector matrix.

9. An image classification method, characterized in that, include: Acquire the medical image to be identified; The medical image is input into a medical image classification model so that the medical image classification model can identify the medical image and determine the image category corresponding to the medical image; wherein, the medical image classification model is constructed by the classification model construction method based on few-sample incremental learning as described in claims 1-8; the image category is classified according to the disease category to which the medical image belongs.

10. An image classification device, characterized in that, include: The data acquisition module is used to acquire the medical images to be identified. A category recognition module is used to input the medical image into a medical image classification model, so that the medical image classification model can recognize the medical image and determine the image category corresponding to the medical image; wherein, the medical image classification model is constructed by the classification model construction method based on few-sample incremental learning as described in claims 1-8; the image category is classified according to the disease category to which the medical image belongs.

Images