Small sample pathological image classification method based on representation learning

Abstract

The invention relates to a small sample pathological image classification method based on representation learning, and the current main technology challenges how to construct a robust deep learning model under the condition of limited samples due to the scarcity of medical image data. According to the method, a small sample sub-packet representation learning method is adopted to train a deep learning model, and a pseudo packet is constructed from the same feature distribution, so that the expansion of a training sample is realized, and the problem of small sample deep learning modeling is effectively solved. And learning a feature prototype by using a momentum dictionary contrast learning method so as to capture image features in different pathological states. The model trained through the method shows good performance in experiments, and the application value of the model in pathological image classification is verified.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a small-sample pathological image classification method based on representation learning. Background Art

[0002] Pathological image classification is an important research direction in the field of medical image analysis. Traditional pathological image analysis relies on the visual inspection and experience judgment of pathologists. This process is not only time-consuming and laborious, but also easily affected by subjective factors, limiting its accuracy and efficiency.

[0003] With the progress of technology, computer vision and machine learning technologies have been widely applied to the automated analysis of pathological images, greatly improving the accuracy and efficiency of classification. In particular, deep learning models, such as convolutional neural networks (CNNs), have become a research hotspot due to their excellent performance in image recognition tasks. However, deep learning models usually require a large amount of labeled data for training, and in actual medical research, obtaining a large-scale high-quality pathological image dataset is often limited by problems such as cost, privacy, and data scarcity.

[0004] How to alleviate the learning difficulties of neural networks in the case of limited sample size has attracted the attention of researchers, and existing research still faces multiple challenges. For example, how to extract effective and robust features from a small number of samples, how to design learning strategies that can adapt to different pathological tasks, and how to handle the problem of sample imbalance, etc.

[0005] In summary, there is currently a lack of a pathological image classification method in the case of a small number of samples. Summary of the Invention

[0006] The purpose of the present invention is to overcome the above-mentioned defects existing in the prior art and provide a small-sample pathological image classification method based on representation learning. A distribution prediction vector obtained from the output of the intermediate layer of the first-level model is recombined to form synthetic data and participate in the next round of training, increasing the available training sample size.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] The present invention provides a small-sample pathological image classification method based on representation learning, which uses a first-level model and a second-level model to classify pathological images. Among them, the training process of the first-level model and the second-level model includes the following steps:

[0009] Obtain the real data of multiple pathological tissues. For each real data, segment it into multiple image patches and encode them to construct a feature matrix;

[0010] Use the feature matrix as the input of the first-level model to obtain the predicted probability of the first category of the output and the distribution prediction vector obtained from the output of the intermediate layer of the first-level model;

[0011] Sort the distribution prediction vectors corresponding to each feature matrix, sample them at a preset sampling rate to form sub-data, use the sub-data as the input of the second-level model, and obtain the predicted probability of the second category of the output;

[0012] Based on the predicted probability of the first category and the predicted probability of the second category, perform weighted fusion, calculate the loss function value considering the energy alignment loss and the momentum dictionary loss, and train the first-level model and the second-level model;

[0013] Add the sub-data to the storage pool, extract multiple sub-data of the same classification from the storage pool to form synthetic data and participate in the next round of training.

[0014] As a preferred technical solution, the first-level model and the second-level model are learned in a supervised learning manner, and the loss function of the supervised loss is:

[0015]

[0016]

[0017] where, are the supervised losses of the first-level model and the second-level model respectively, · represents calculating the expected value, X i and Y i represent the model input features and the corresponding labels respectively, represents the first-level model and the second-level model including the classification head, represents the cross-entropy loss function.

[0018] As a preferred technical solution, the loss function of the energy alignment loss is:

[0019]

[0020]

[0021] where, is the loss function of the alignment loss, T is the temperature parameter, K is the number of classification categories, f1 and f2 respectively represent the first-level model and the second-level model without the classification head, and E represents the anchored energy value.

[0022] As a preferred technical solution, after constructing the feature matrix, it further includes:

[0023] Divide according to the method of five-fold cross-validation.

[0024] As a preferred technical solution, the calculation process of the momentum dictionary loss includes:

[0025] Construct a momentum dictionary for providing positive samples and negative samples;

[0026] For the current training sample, calculate the cosine similarity with all positive samples in the momentum dictionary and sort them, select the target positive sample, calculate the cosine similarity between the current training sample and all negative samples and sort them, and select the target negative sample based on the training progress;

[0027] Calculate the momentum dictionary loss based on the target positive sample and the target negative sample.

[0028] As a preferred technical solution, the loss function of the momentum dictionary loss is:

[0029]

[0030] where, is the loss function of the momentum dictionary loss, v i , v gi are the feature vectors generated by the first-level model and the second-level model respectively, v + and v - are the target positive sample and the target negative sample respectively, ‖·‖2 represents taking the second norm, and n represents the size of the training set.

[0031] As a preferred technical solution, in the process of weighted fusion, the weights are updated as learnable parameters during training.

[0032] As a preferred technical solution, each of the image patches formed by segmentation includes pathological tissues.

[0033] As a preferred technical solution, a pre-trained ResNet50 model is used to encode the image patches.

[0034] As a preferred technical solution, the first-level model and the second-level model are trained using the Adam optimizer.

[0035] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0036] (1) Expand the available training sample size: In this application, the distribution prediction vector obtained from the output of the intermediate layer of the first-level model is used as the prediction of the feature distribution of the current input data. By using the method of splitting based on the feature distribution to obtain sub-data and recombining them into synthetic data, the synthetic data is recombined and participates in the next round of training, increasing the available training sample size.

[0037] (2) Improve the consistency of pathological image recognition: This application uses the loss function value considering the energy alignment loss for training, and improves the recognition of the consistency of pathological image data through the feature alignment of the dual models.

[0038] (3) Improve the prediction ability of gene phenotypes with small samples: This application adopts a representation learning method based on a momentum dictionary, effectively improving the prediction ability of gene phenotypes with small samples. Brief Description of the Drawings

[0039] Figure 1 It is a schematic flowchart of the small-sample pathological image classification method based on representation learning in the embodiment;

[0040] Figure 2 It is a schematic diagram of the model structure in the embodiment. Detailed Embodiments

[0041] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0042] The terms "first" and "second" are only used for descriptive purposes, and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of this application, "a plurality of" means two or more, unless otherwise specifically defined.

[0043] Embodiment 1

[0044] In view of the problems existing in the foregoing prior art, this embodiment provides a small-sample pathological image classification method based on representation learning, and uses a small-sample sub-package representation learning method to train a deep learning model. This method constructs pseudo-packages from the same feature distribution, thereby realizing the expansion of training samples and effectively solving the problem of small-sample deep learning modeling. The purpose of this method is to solve or partially solve the problem in the classification research of pathological images that due to the small number of samples and too much supervision information provided by pathological images, it is difficult to support the deep learning model for effective learning.

[0045] S1. First, perform tissue segmentation on the whole-slide digital slices in the database to obtain a large number of image patches and ensure that all image patches contain tissues.

[0046] Specifically, at a magnification of 20 times, the color segmentation algorithm is used to scan the field of view according to a 256*256 window. When the tissue area in the window is higher than the threshold, the window position is recorded until the entire whole-slide digital section is scanned.

[0047] S2. Refer to Figure 2 On the left side, use the ResNet50 model pre-trained on ImageNet to encode all the image patches in Step 1, so that the length of the encoded image features is 1024 dimensions. Then, organize the encoded features according to the corresponding whole-slide sections, so that each section finally corresponds to a feature matrix. The value of dimension 1 of the matrix is equal to the number of image patches on the section, and the value of dimension 2 is equal to the feature length, that is, 1024 dimensions.

[0048] Specifically, given the pre-trained ResNet model, crop the images according to the statistically recorded window positions, and then perform feature embedding through the pre-trained model. When cropping the images, image transformations are adopted with a certain probability. At this time, for the whole-slide digital section, it can be converted into multiple feature matrices, thereby realizing offline feature enhancement. The corresponding ID and label are consistent with the original whole-slide section.

[0049] S3. Define a neural network with the input being the feature matrix, the middle layer generating latent vectors of the size of the number of image patches, and the output being the probability of each category as the first-level model. Input the encoded whole-slide sections in S2, that is, the feature matrix, into the first-level model, and the model will output the predicted probability of each category (i.e., the first-category prediction probability); provide the label of the whole-slide digital section for the training of the first-level model, and use the cross-entropy function to train the model; during the training process of the model, extract the latent vectors generated by the middle layer as the prediction of the feature distribution of the current input data by the model; sort the distribution prediction vectors, that is, the latent vectors, in order, and then sample them. The sampled data is called sub-data, and the sampling rate can be set according to the experiment. The sub-data is mapped by the second-level model to obtain the second-category prediction probability, and the mapping results are weighted and fused. The weights are updated as learnable parameters during training. The fused features are mapped into category probabilities through the output layer and calculate the cross-entropy with the image-level label to update the model.

[0050] In Steps S2 and S3, for the first-level model and the second-level model, perform five-fold cross-validation. The training objective for each fold is to minimize Use the Adam optimizer with a learning rate of 0.0005, and the number of training times is 200. Select the first-level model with the highest accuracy on the validation set and evaluate it on the test set.

[0051] S4. In step S3, the sub - data are all treated as independent data, and the labels of the whole - field digital slices are assigned to the corresponding sub - data. The second - level model trains this batch of data in the form of direct supervision.

[0052] S5. All the sub - data in step S3 enter the storage pool and retain the labels of the whole - field digital slices to which they belong. During the training process, multiple data with the same classification labels are selected from the storage pool and then packed into a complete new data (referred to as synthetic data). Repeat the process of step S2 so that this new data participates in the training of the two models.

[0053] S6. In steps S3 and S5, the first - level model and the second - level model perform supervised learning according to the labels of the whole - field digital slices. The loss calculation formulas are formula (1) and formula (2) respectively, where represent the first - level and second - level models including the classification head, f1 and f2 represent the first - level and second - level models without the classification head, X i 、Y i represent the input features and labels respectively, represents the cross - entropy loss function; at the same time, energy - based alignment is performed between the mappings of the two models (i.e., considering the energy alignment loss) so that the information differences between different embedding methods can be learned by the models. The definition formula of energy is formula (3), where T is the temperature parameter used to control the "sharpness" of the distribution, and K is the number of classification categories. Define the anchored energy value as E, and align the features output by the two models by minimizing formula (4) so that the alignment information is learned as a kind of supervision information.

[0054]

[0055]

[0056]

[0057]

[0058] S7. Create a momentum dictionary to save the feature vectors generated by each training data through the model during training. For the feature vectors, it refers to the vectors obtained by mapping the input data through the neural network to 512 - dimensional vectors. Different from the latent vectors used to predict the distribution, the feature vectors contain the information most relevant to classification because the feature vectors will ultimately be converted into the prediction probabilities for each category. The key corresponding to the feature vector is the ID of each whole - field digital slice. When the first round of deep - learning training ends, all the IDs already exist in the dictionary, and at this time, the mapping of the data will update the values in the dictionary in the form of momentum. This momentum dictionary provides positive and negative samples for the contrast learning of the model.

[0059] S8. For a certain training sample, based on the Easy-SemiHard-Hard principle, select appropriate positive and negative samples from the momentum dictionary. The SemiHard principle is as follows: when selecting negative examples, the distance between the negative example and the anchor point is greater than the distance between the anchor point and the positive example, but still within a certain boundary. The ESH principle is as follows: according to the progress of training, the discrimination between the selected positive and negative samples gradually decreases, which promotes the model to better learn the ability to distinguish similar samples, and at the same time avoids some potential training problems, such as overly difficult samples may lead to unstable model training. (That is, considering the momentum dictionary loss)

[0060] In steps S7 and S8, control the discrimination between positive and negative samples according to the ESH principle, which involves using the definition of discrimination and the control of discrimination. For the current training sample, calculate the cosine similarity with all positive sample features in the dictionary mentioned in step S7, then randomly select one from the top T positive samples with the highest similarity, and record the corresponding similarity; calculate the cosine similarity between the current sample and all negative samples in the dictionary, then sort them from small to large, and select the corresponding negative samples in the sorting according to the progress of training. For example, when the training progress is 50%, select the negative sample at the median position in the sorting result. Calculate the loss according to the following formula:

[0061]

[0062] where, v i , v gi are the feature vectors generated by the first-level and second-level models respectively, v + and v - are the positive and negative samples respectively, and ‖v‖2 represents the two-norm of the vector.

[0063] The following uses a specific example to illustrate the specific implementation steps:

[0064] 1. The system first performs data preprocessing on the full-field digital slice to obtain all small image patches (size 256*256) containing tissue regions, and then uses the pre-trained model to perform feature embedding on the image patches. After processing all slices in turn, perform data division according to the five-fold cross-validation method;

[0065] 2. Build the first-level model, and use the data processed in step S1 and the existing labels to train it. In addition, the synthetic samples described in step S5 are also used for training;

[0066] 3. Split the data based on the prediction results of the first-level model on the feature distribution, and build a second-level model. This model maps the split data separately, and then uses the fusion mechanism to obtain the vector for classification, and then obtains the probability output through the classification output head, calculates the loss and uses the gradient descent algorithm to update the model. At the same time, each sub-data is also input into the second-level model as a valid sample with the original label for learning;

[0067] 4. The mapping of the first-level and second-level models to the data will be constrained by an energy-based alignment method, using the difference between the mapped energy of the two models and the anchor energy as a kind of supervision information;

[0068] 5. Use the mapping features to build a momentum dictionary, and select positive and negative samples for the current training samples according to the ESH principle, so as to promote the model to better express the data by shortening the distance to the positive samples and alienating the distance to the negative samples;

[0069] 6. Conduct experiments based on five-fold cross-validation, and use the first-level model with the smallest validation error to predict pathological images on the test set. First, follow the steps in step S1 to preprocess the data of the full-view digital slices in the test set one by one, and then use the pre-trained model to embed features in the image blocks. All feature embeddings on a single full-view digital slice are integrated into a feature matrix, which is then input into the model (the first-level model with the smallest validation error in the five-fold cross-validation model) to obtain the predicted probability of each category. The category with the highest probability is the predicted category of this single full-view digital slice.

[0070] Table 1 shows the results of the model training under different dataset sizes and evaluating on an independent test set. The method proposed in the present invention can achieve better performance when trained on 59, 69, 78, 88, and 98 full-field digital pathology slides.

[0071] Table 1 Results under different dataset sizes

[0072]

[0073] This method increases the available number of training samples by splitting based on feature distribution to obtain sub-data and reorganizing it into synthetic data, and improves the recognition of the consistency of pathological image data through feature alignment of dual models. In addition, a method for realizing the selection of positive and negative samples based on the momentum dictionary and the ESH principle is proposed to achieve representation learning with adjustable training difficulty, effectively improving the prediction ability of gene phenotypes under small samples. The momentum dictionary contrast learning method is used to learn feature prototypes to capture image features under different pathological states. The model trained by this method shows good performance in experiments, and its application value in pathological image classification is verified. Considering the generalization ability of the model, the effectiveness of feature extraction, and computational efficiency, the deficiencies of the existing technology are overcome. Generally speaking, the small-sample pathological image classification technology has important application value and broad market prospects in solving the problems of limited medical resources and data privacy. This method aims to further promote the technological progress in this field and provide new solutions for the automated analysis of pathology.

[0074] Example 2

[0075] This embodiment provides an electronic device, including: one or more processors and a memory. The memory stores one or more programs, and the one or more programs include instructions for executing the small-sample pathological image classification method based on representation learning as described in Example 1.

[0076] Example 3

[0077] This embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device. The one or more programs include instructions for executing the small-sample pathological image classification method based on representation learning as described in Example 1.

[0078] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A few-shot pathological image classification method based on representation learning, characterized in that, Classify pathological images using a first-level model and a second-level model. The training processes of the first-level model and the second-level model include the following steps: Obtain the real data of multiple pathological tissues. For each real data, segment it into multiple image patches, encode them, and construct a feature matrix; Use the feature matrix as the input of the first-level model to obtain the predicted probability of the first category as the output and the distribution prediction vector obtained from the output of the intermediate layer of the first-level model; Sort the distribution prediction vectors corresponding to each feature matrix, sample them at a preset sampling rate to form sub-data, use the sub-data as the input of the second-level model, and obtain the predicted probability of the second category as the output; Perform weighted fusion based on the predicted probability of the first category and the predicted probability of the second category, calculate the loss function value considering the energy alignment loss and the momentum dictionary loss, and train the first-level model and the second-level model; Add the sub-data to the storage pool, extract multiple sub-data of the same classification from the storage pool to form synthetic data and participate in the next round of training.

2. The small-sample pathological image classification method based on representation learning according to claim 1, wherein, The first-level model and the second-level model are learned in a supervised learning manner. The loss function of the supervised loss is: Among them, are the supervision losses of the first-level model and the second-level model respectively, represents calculating the expected value, X i , Y i represent the model input features and the corresponding labels respectively, represents the first-level model and the second-level model including the classification head, represents the cross-entropy loss function.

3. A few-shot pathological image classification method based on representation learning according to claim 1, characterized in that The loss function of the energy alignment loss is: Among them, is the loss function of the alignment loss, T is the temperature parameter, K is the number of classification categories, f1 and f2 respectively represent the first-level model and the second-level model without the classification head, and E represents the anchored energy value.

4. A few-shot pathological image classification method based on representation learning according to claim 1, characterized in that After constructing the feature matrix, it also includes: Divide according to the method of five-fold cross-validation.

5. A few-shot pathological image classification method based on representation learning according to claim 1, characterized in that, The calculation process of the momentum dictionary loss includes: Construct a momentum dictionary for providing positive samples and negative samples; For the current training sample, calculate the cosine similarity with all positive samples in the momentum dictionary and sort them, select the target positive sample, calculate the cosine similarity with all negative samples and sort them, and select the target negative sample based on the training progress; Calculate the momentum dictionary loss based on the target positive sample and the target negative sample.

6. The small-sample pathological image classification method based on representation learning according to claim 1, characterized in that The loss function of the momentum dictionary loss is: Among them, is the loss function for the momentum dictionary loss, v i , v gi are the feature vectors generated by the first-level model and the second-level model respectively, v + and v - are the target positive sample and the target negative sample respectively, ‖·‖2 represents the calculation of the second norm, and n represents the size of the training set.

7. A few-shot pathological image classification method based on representation learning according to claim 1, characterized in that, In the process of the weighted fusion, the weights are updated as learnable parameters during the training process.

8. A few-shot pathological image classification method based on representation learning according to claim 1, characterized in that, Each of the image patches formed by segmentation includes pathological tissues.

9. A few-shot pathological image classification method based on representation learning according to claim 1, characterized in that Use a pre-trained ResNet50 model to implement the encoding of the image patches.

10. A few-shot pathological image classification method based on representation learning according to claim 1, characterized in that, The first-level model and the second-level model are trained using the Adam optimizer.

Images