A cost-sensitive method for small sample classification of rare tumor categories

Abstract

The present invention discloses a method for classifying small samples of rare tumor categories based on cost sensitivity. Most of the existing small sample learning methods ignore the particularity and mobility of the feature space, while the present invention focuses on this. The constructed feature storage and screening module allows the features in the previous stage of pre-training to be fully utilized by the meta-learning in the next stage. First, a traditional deep learning neural network is constructed for pre-training, and a non-rare category data set of similar tasks is learned. The parameters of the pre-training model are frozen and the feature prototypes and feature variances of each category in the pre-training samples are stored. Next, the relationship between the current feature and the category prototype selected in the previous stage is encoded through Transformer, and then new features based on the current feature are generated for classification by the small sample classifier. A learnable cost-sensitive function is introduced in the meta-learning process, so that the network is more sensitive to samples of rare categories.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning image processing, and in particular to a small sample medical image classification algorithm that incorporates an attention method. Background Art

[0002] Deep learning is a research direction in the field of machine learning. It originates from the study of artificial neural networks. It simulates the structure of the human brain to build multiple hidden layers. After training with a large number of data sets, it can learn rich features. Deep learning has made significant progress in recent years and has been applied in many fields. In the image classification, target recognition, image segmentation, image fusion and other tasks in the field of computer vision, deep learning plays an important role.

[0003] Small sample learning is to use past knowledge to correctly distinguish test samples when there are only a few labeled training samples. A core idea to solve the small sample problem is called meta-learning. The key concept of meta-learning is learning-to-learn, which is to summarize the knowledge learned in the past and then show strong discrimination ability when facing new unknown samples. This feature is very close to the way humans learn.

[0004] Transformer was proposed in 2017. It abandons the traditional CNN and RNN. The entire network structure is composed entirely of self-attention mechanism (Self-Attention) and feed-forward neural network (FFN). A trainable neural network based on Transformer can be built by stacking Transformers. Transformer was first used in natural language processing and has also been widely used in the field of computer vision in recent years. Summary of the invention

[0005] The present invention adopts a training process of two stages, namely, pre-training and meta-learning. Among them, the features of medical images are focused on, and a feature selection method based on the Gaussian distribution hypothesis is proposed to solve the problem of feature specificity and migration. The feature distribution learned by the pre-training network is mapped to a Gaussian distribution and stored in a feature bank. In terms of classifier learning, a classifier meta-learning method based on the attention mechanism is proposed to solve the problem of utilizing the features acquired in the first stage in the second stage. Among them, a learnable cost-sensitive function is introduced in the meta-learning process to help the network more effectively solve the classification problem of small samples of rare tumor categories.

[0006] Technical Solution

[0007] The present invention discloses a method for classifying small samples of rare tumor categories based on cost sensitivity. Most of the existing small sample learning methods ignore the particularity and mobility of the feature space, while the present invention focuses on this. The constructed feature storage and screening module allows the features in the previous stage of pre-training to be fully utilized by the meta-learning in the next stage. First, a traditional deep learning neural network is constructed for pre-training, and a data set of similar tasks is learned. The parameters of the pre-training model are frozen and the feature prototypes and feature variances of each category in the pre-training samples are stored. Next, the relationship between the current feature and the category prototype selected in the previous stage is encoded through Transformer, and then new features based on the current feature are generated for classification by the small sample classifier. A cost-sensitive function is introduced in the meta-learning process, so that the network is more sensitive to samples of rare categories.

[0008] The specific steps include:

[0009] S1, construct and preprocess a small sample dataset of rare tumor categories. The dataset is collected based on actual problems;

[0010] S2, build a feature learning pre-training network to learn non-rare category data sets, train a powerful pre-training model, and freeze parameters and store features for use in subsequent tasks. Use the widely used and high-performance Resnet50 as the training network, and use cross entropy combined with softmax classifier for training.

[0011] S3, extract and store features. In the pre-training stage, features are extracted and stored in the feature library. The previously learned features will be distributed and mapped into a Gaussian distribution. The feature prototype and variance of each category are stored in the feature library.

[0012] S4, constructs a feature screening module. By calculating the similarity between the current feature and the feature prototype of the category in the pre-trained feature space, k pre-trained categories that are closest to the current sample can be selected as candidate categories to achieve the migration and transition of subsequent features.

[0013] S5, construct a classifier meta-learning method based on the attention mechanism. Transformer encodes the relationship between the current sample features and the screened category prototype through the feature set of the previous step to complete feature migration.

[0014] S6, introduces a cost-sensitive function in the classification process of meta-learning, so that the network is more sensitive to samples of rare categories. By setting a range, the weights can learn to obtain appropriate values ​​within the interval to achieve different sensitivity to different samples.

[0015] The image preprocessing process S1 of the above dataset specifically includes the following steps:

[0016] S11, read the pictures in the dataset;

[0017] S12, crop the original image with random aspect ratio and resize it to 224*224 size

[0018] S13, perform random horizontal flipping with probability p=0.5

[0019] S14, modify brightness, contrast and saturation, and perform random color fluctuations

[0020] S15, regularization processing

[0021] In step (2), a feature learning pre-training network is constructed to learn the data set of non-rare categories, train a powerful pre-training model, and freeze the parameters and store the features for use in subsequent tasks; the pre-training model uses the widely used and high-performance Resnet50 as the training network, and uses cross entropy combined with softmax classifier for training.

[0022] In the above step S3, after the pre-training phase, an additional feature storage module is introduced: feature bank. The features learned by the pre-training network will be distributed and mapped into a Gaussian distribution and stored in the feature bank. Specifically, the following steps are included:

[0023] S31, the features of each category are distributed and mapped to a Gaussian distribution through power transform:

[0024]

[0025] S32, calculates the feature mean and feature variance for the corresponding category, and the calculation formula is as follows:

[0026]

[0027] S33, assuming that there are n categories in the pre-training process, then the feature means and feature variances of the n corresponding categories are stored in the feature library, and each category occupies a column in the feature library.

[0028] In the above step S4, by calculating the similarity between the current feature and the feature prototype of the category in the pre-trained feature space, k pre-trained categories that are closest to the current sample can be selected as candidate categories to achieve the migration and transition of subsequent features. Specifically, the following steps are included:

[0029] S41, calculate the similarity between the current feature and the feature prototype of the category in the pre-trained feature space. The similarity calculation formula is expressed as:

[0030]

[0031] S42, in order to reduce the network overhead, select the first k feature prototypes and corresponding feature variances (μ i , δ i ), these features are combined into a new feature set for the input of the next stage transformer.

[0032] In the above step S5, a classifier meta-learning method based on the attention mechanism is constructed. The backbone learned by the network during the pre-training process will be fixed. This step only updates the parameters of the Transformer and the classifier. Specifically, the following steps are included:

[0033] S51, select the tasks that actually need to be classified and build the support set and query set.

[0034] S52, through the network trained in the previous pre-training stage, fix the BackBone and extract the features of the current input.

[0035] S53, through the feature screening of step S4, the current input feature and the k most similar features are input into the transformer.

[0036] S54, during the training of transformer, Gaussian sampling is used to simulate the expression of each selected category in the feature space. The newly generated features can be expressed as q i ~N(μ i , δ i ).

[0037] S55, Transformer encodes the relationship between the current feature and the category prototype screened in the previous step, completes feature migration, and then generates new features based on the current feature through the Transformer's Decoder for classification by the small sample classifier. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of a method for classifying small samples of rare tumor categories based on cost sensitivity provided by an embodiment of the present invention.

[0039] Figure 2 This is a complete network structure diagram of the present invention.

[0040] Figure 3 This is a schematic diagram of pre-training and feature storage.

[0041] Figure 4 Schematic diagram of the feature screening process

[0042] Figure 5 This is a schematic diagram of the classifier meta-learning process. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0044] The present invention adopts a training process of two stages, namely, pre-training and meta-learning. Among them, the features of medical images are focused on, and a feature selection method based on the Gaussian distribution hypothesis is proposed to solve the problem of feature specificity and migration. The feature distribution learned by the pre-training network is mapped to a Gaussian distribution and stored in a feature bank. In terms of classifier learning, a classifier meta-learning method based on the attention mechanism is proposed to solve the problem of utilizing the features acquired in the first stage in the second stage. Among them, a learnable cost-sensitive function is introduced in the meta-learning process to help the network more effectively solve the classification problem of small samples of rare tumor categories.

[0045] Figure 2 It is a complete network structure diagram of the present invention, which supports the above two-stage training process.

[0046] Example

[0047] The method for classifying small samples of rare tumor categories based on cost-sensitive classification includes the following steps: Figure 1 As shown:

[0048] Step 1: Construct a dataset. The first is the dataset of the pre-training stage. The primary purpose of the pre-training stage is to learn a feature space with strong enough expression ability. Therefore, a large enough dataset is needed. The dataset used is a dataset of non-rare categories. A dataset similar to the specific image classification task is selected. The second is the dataset of the meta-learning stage, which is the dataset for the actual classification task. The dataset comes from rare category data collected from actual problems, and some pre-processing operations are performed. When reading the pictures in the dataset, the original picture is first cropped with random aspect ratio and resized to 224*224, then randomly flipped horizontally with probability p=0.5, the brightness, contrast and saturation are modified, random color fluctuations are performed, and finally regularization is performed.

[0049] Step 2: Construct a feature learning pre-training network, which is a traditional deep learning neural network. The purpose is to train a powerful pre-training model to obtain better parameters by learning non-rare category data sets, and freeze parameters and store features for subsequent tasks. Use the widely used and high-performance Resnet50 as the training network, and use cross entropy combined with softmax classifier for training.

[0050] Step 3: Extract and store features. After the pre-training phase, an additional feature storage module is introduced: the feature bank. The features learned by the pre-training network will be distributed and mapped into a Gaussian-like distribution. Only the feature prototype (mean μ ) of each category needs to be stored in the feature bank. i ) and characteristic variance (variance δ i ). The mapping process refers to power transform, and the specific formula is as follows:

[0051]

[0052] Where v represents the sample features of each category, ∈ and β are hyperparameters. Through the transformation of f(v), the output features have the characteristics of Gaussian distribution, and the feature distribution of a category can be represented by only two values: mean and variance. Overall, after the pre-training phase, all training samples in the pre-training data set are traversed, their corresponding feature representations are counted, and then the feature mean and feature variance are calculated for the corresponding category. The calculation formula is as follows:

[0053]

[0054] where μ i is the characteristic mean of the i-th class, δ i is the characteristic variance of the i-th class. j represents the sample features belonging to category i, that is, the features of the jth sample, f(x j ) represents the sample features under Gaussian distribution after power transform, n i Indicates the number of samples belonging to category i. Assuming there are n categories in the pre-training process, the feature library will store the feature means and feature variances of n corresponding categories. For the pre-training and feature storage stage, please refer to Figure 3 .

[0055] Step 4: Construct a feature screening module. By calculating the similarity between the current feature and the feature prototype of the category in the pre-trained feature space, the k pre-trained categories that are closest to the current sample can be selected as candidate categories to achieve the migration and transition of subsequent features. The similarity calculation formula is expressed as:

[0056]

[0057] Where a and b are the features of the two inputs. Calculate the distance between two features and find its reciprocal so that the more similar a and b are, the closer the distance is, and the larger s(a,b) is. The purpose of adding a small constant δ to the denominator is to prevent the denominator from being 0.

[0058] The reason for selecting the nearest k is to reduce the network overhead. The feature selection process can be referred to Figure 4 .

[0059] Through feature screening, the current sample feature q and the first k feature prototypes and corresponding feature variances (μ i , δ i ). These features form a feature set, which will be used as the input of the transformer in the next stage.

[0060] Step 5: Construct a classifier meta-learning method based on the attention mechanism. The training set and test set at this time are the training set and test set of the actual classification task. The backbone learned by the network in the previous pre-training process will be fixed. In this stage, only the parameters of the Transformer and the classifier are updated, and other parts of the network are not learned. Transformer is used to combine the current features with the features in the pre-trained feature space. The relationship between the current features and the category prototype selected in the previous step is encoded by Transformer to complete feature migration, and then the new features based on the current features are generated by the Transformer Decoder for classification of small sample classifiers.

[0061] When training Transformer, Gaussian sampling is used to simulate the expression of each selected category in the feature space. The newly generated features can be expressed as q i ~N(μ i , δ i ). Through this sampling method, the features of the pre-training sample categories can be richer, thereby improving the robustness of the results. The classifier meta-learning process can be referred to Figure 5 .

[0062] Step 6: Introduce rare category sensitivity and introduce a cost-sensitive function into the classification process of meta-learning, so that the network is more sensitive to samples of rare categories and helps the network effectively solve the classification problem of small samples of rare tumor categories.

[0063] Taking binary classification as an example, the traditional cross entropy loss function is expressed as:

[0064] L=-y + log(y + )-y - log(y - )

[0065] The two parts introduce weights w1 and w2 respectively:

[0066] L=-w1*y + log(y + )-w2*y - log(y - )

[0067] Where y+ is the probability of positive samples, y- is the probability of negative samples, w1 and w2 are weight hyperparameters respectively; by increasing the weight of rare categories, the network is made more sensitive to rare category samples;

[0068] In the meta-learning process, by setting the maximum and minimum values, the weights w1 and w2 are allowed to learn to obtain appropriate values ​​within the range, so as to achieve different sensitivities of the classifier to different samples, thereby achieving the purpose of paying more attention to rare samples.

Claims

1. A method for classifying small samples of rare tumor categories based on cost sensitivity, characterized in that: The following steps are involved: Step (1) construct and preprocess a small sample data set of rare tumor categories, where the data set is collected based on actual problems; Step (2) constructing a feature learning pre-training network to learn a dataset of non-rare categories and freeze parameters and store features for use in subsequent tasks; Step (3) extracting and storing features: In the pre-training stage, features are extracted and stored in a feature library. The previously learned features are distributed and mapped into a Gaussian-like distribution. The feature prototype and variance of each category are stored in the feature library. Step (4) constructs a feature screening module, by calculating the similarity between the current feature and the feature prototype of the category in the pre-trained feature space, selects k pre-trained categories that are closest to the current sample feature as candidate categories, and realizes the migration and transition of subsequent features; Step (5) constructs a classifier meta-learning method based on the attention mechanism. Transformer encodes the relationship between the current sample features and the screened category prototype through the feature set of the previous step to complete feature transfer; Step (6) introduces a learnable cost-sensitive function into the classification process of meta-learning, so that the network is more sensitive to samples of rare categories. By setting the maximum and minimum values, the weights are learned within the interval to obtain appropriate values, so as to achieve different sensitivity levels to different samples. In step (3), the features obtained in the pre-training phase will be mapped to a Gaussian-like distribution and stored in the feature library, including the following steps: (3.1) The features of each category are distributed and mapped into a Gaussian distribution through power transform: Where v represents the sample features of each category, ∈ and β are hyperparameters, and the distribution of v is mapped to a Gaussian-like distribution through f(v); (3.2) Calculate the feature mean and feature variance for the corresponding category, and the calculation formula is as follows: where μ i is the characteristic mean of the i-th class, δ i is the characteristic variance of the i-th class; x j represents the feature of the jth sample, f(x j ) represents the characteristics of the sample under the Gaussian distribution after Power Transform, n i represents the number of samples belonging to category i; (3.3) Assuming that there are n categories in the pre-training process, the feature means and feature variances of the n corresponding categories are stored in the feature library, and each category occupies one column in the feature library; In step (4), k pre-trained categories that are closest to the current sample features are selected as candidate categories based on similarity to achieve subsequent feature migration and transition, including the following steps: (4.1) Calculate the similarity between the current feature and the feature prototype of the category in the pre-trained feature space. The similarity calculation formula is expressed as: Where a and b are the features of the two inputs. Calculate the distance between two features and find its reciprocal so that the more similar a and b are, the closer the distance is, and the larger s(a,b) is. The purpose of adding a small constant δ to the denominator is to prevent the denominator from being 0. (4.2) In order to reduce the network overhead, we select the first k feature prototypes and corresponding feature variances (μ i , δ i ), these features are combined into a new feature set for the input of the next stage transformer; In step (5), constructing a classifier meta-learning method based on the attention mechanism includes the following steps: (5.1) Select the tasks that actually need to be classified and build the support set and query set; (5.2) Through the network trained in the previous pre-training stage, fix the BackBone and extract the features of the current input; (5.3) Through the feature screening of step (4), the current input feature and the k most similar features are input into the transformer; (5.4) During the training of the transformer, Gaussian sampling is used to simulate the expression of each selected category in the feature space. The newly generated features are shown as q i ~N(μ i , δ i ), which is consistent with μ i is the mean, δ i is a Gaussian distribution with variance; (5.5) Transformer encodes the relationship between the current feature and the category prototype selected in the previous step, completes feature migration, and then generates new features based on the current feature through the Transformer Decoder for classification of small sample classifiers; In step (6), a cost-sensitive function is introduced in the classification process of meta-learning to make the network more sensitive to samples of rare categories. Weights w1 and w2 are introduced on the basis of the traditional binary cross entropy: L=-w1*y + log(y + )-w2*y - log(y - ) Where y+ is the probability of positive samples, y- is the probability of negative samples, w1 and w2 are weight hyperparameters respectively; by increasing the weight of rare categories, the network is made more sensitive to rare category samples; In the meta-learning process, by setting the maximum and minimum values, the weights are allowed to learn by themselves within the range, so as to obtain appropriate weights to achieve different sensitivities of the classifier to different samples, thereby achieving the purpose of paying more attention to rare samples.

2. The method for classifying small samples of rare tumor categories based on cost sensitivity according to claim 1, characterized in that: In step (1), the image preprocessing includes the following steps: (1.1) Read the images in the dataset; (1.2) Crop the original image with random aspect ratio and resize it; (1.3) Perform random horizontal flip; (1.4) Modify brightness, contrast and saturation to perform random color fluctuations; (1.5) is subjected to regularization.

Images