A multi-stained kidney pathological image segmentation method based on self-supervised meta-learning

Abstract

The present application relates to a kind of multi-staining kidney pathological image segmentation methods based on self-supervised meta-learning, belong to artificial intelligence auxiliary medical examination technical field, including the following steps: S1, the full field pathological photo of different staining kidney pathological section is obtained, while randomly selecting 2-5 each kind of dyeing is carried out pixel-level artificial annotation;S2, self-supervised pre-training model is trained by unlabelled sample;S3, with the self-supervised pre-training model of training completion as first-level model, with U-Net structure as backbone network to build second-level model, second-level model is trained by meta training set, and the weight parameters of first-level model are updated by second-level model is tested;The above-mentioned test and training process is repeated by reselecting dyeing sample, and the first-level model obtained is used as meta-learning segmentation model;S4, the pathological section to be measured is segmented by meta-learning segmentation model.The present application can be efficiently segmented to the region of interest of different staining kidney pathological section.

Description

Technical Field

[0001] This invention relates to a multi-stain kidney pathology image segmentation method based on self-supervised meta-learning, belonging to the field of artificial intelligence-assisted medical examination technology. Specifically, it relates to a method that uses full-view images obtained from puncture samples of kidney disease patients in a pathology slide scanner as analysis data, and automatically segments the regions of interest in the images with minimal training cost through meta-learning and semantic segmentation algorithms. Background Technology

[0002] In full-view images of kidney pathology, the glomeruli, sclerotic glomeruli, renal tubules, and small blood vessels are the regions of interest (ROIs), and different ROIs exhibit different morphologies under different staining methods. In the pathological diagnosis of chronic kidney disease, pathologists first need to locate and observe each ROI in high-resolution full-view slides with various staining methods, and then determine whether a lesion has occurred in that ROI. However, the locations of ROIs in kidney pathology slides are not fixed and are numerous. Pathologists need to frequently zoom in and out of the optical microscope under various staining methods to locate each ROI individually. This outdated and cumbersome technology makes it difficult to implement intelligent assisted diagnosis of kidney pathology in a timely manner, resulting in extremely low consultation efficiency for patients with chronic kidney disease, with pathology results taking up to a week to obtain. This inefficient diagnostic process hinders the development of pathological diagnosis of chronic kidney disease.

[0003] Patent application number 202110590511.8 discloses a method and system for identifying renal tubular atrophy regions based on deep learning, which uses instance segmentation to segment atrophied renal tubular regions in full-view renal pathology sections. Patent application number 202010560815.0 discloses an intelligent detection method and system based on pathological images and deep learning, which uses Faster R-CNN to detect glomeruli in full-view renal pathology sections. Patent application number 202011119567.2 discloses an AI-based renal pathology image segmentation model, method, and system, which segments glomeruli in full-view renal pathology sections by constructing a U-net fully convolutional network. However, in pathological diagnosis, pathologists need to simultaneously observe multiple regions of interest under various staining methods to obtain a comprehensive diagnosis of kidney disease patients. Existing AI-based methods for identifying regions of interest in renal pathology sections often identify or segment a single region of interest under a single staining, which is not suitable for clinical use. On the other hand, the size of a full-field section of kidney pathology is usually around 1 billion pixels. It is extremely difficult to perform extensive annotation of every region of interest for every staining type, and as shown in the attached figure... Figure 2As shown, tissues from different staining sources have significant morphological differences, making it difficult to directly transfer models. This makes it very difficult to apply artificial intelligence in the field of kidney pathology. Therefore, it is currently impossible to perform small-sample segmentation of multiple regions of interest in multi-stained kidney pathology, which limits the application of artificial intelligence in medical image recognition. Summary of the Invention

[0004] This invention addresses the urgent need for segmentation of multiple regions of interest (ROIs) in renal pathology under various staining conditions, and the problem that current ROI segmentation algorithms for renal pathology suffer from high annotation costs and thus limited staining methods and ROI segmentation. It provides a multi-stain renal pathology image segmentation method based on self-supervised meta-learning.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a multi-stain kidney pathology image segmentation method based on self-supervised meta-learning, comprising the following steps:

[0006] S1. Obtain full-field pathological images of kidney pathological sections. At the same time, randomly select 2 to 5 digital pathological sections for each staining and manually annotate the shapes of glomeruli, sclerotic glomeruli, renal tubules and blood vessels at the pixel level to obtain annotated samples under different staining conditions.

[0007] S2. Obtain unlabeled samples and train a self-supervised pre-trained model;

[0008] S3. Using the self-supervised pre-trained model as the first-level model, and the U-Net structure as the backbone network to construct the second-level model; randomly select one type of stained labeled samples, and randomly divide them into a meta-training set and a meta-test set. Assign the weights of the first-level model to the second-level model, train the second-level model through the meta-training set and update the weight parameters of the second-level model through backpropagation, test the second-level model through the corresponding stained meta-test set, and update the weight parameters of the first-level model according to the test results using the backpropagation rule; randomly select data under one type of stained data again, randomly divide it into a meta-training set and a meta-test set, and repeat the above training and testing process until the loss of the first-level model converges. The obtained first-level model is used as the meta-learning segmentation model.

[0009] S4. The pathological slides to be tested are segmented using a meta-learning segmentation model to obtain the final segmentation result.

[0010] Preferably, in step S2, the training method for the self-supervised pre-trained model is as follows:

[0011] S201. Cropping the unlabeled sample image into several sub-images of fixed size;

[0012] S202. Randomly select two sub-images with the same coloring, use one sub-image as the support image and the other sub-image as the query image; use a convolutional neural network to extract features from the two selected images and calculate the contrast loss.

[0013] S203. Update the parameters of the self-supervised pre-trained model based on the calculated contrastive loss;

[0014] S204. Repeat S202 to S203 until the comparison converges, completing the training of the self-supervised pre-trained model.

[0015] Preferably, in step S201, the unlabeled pathological image is cropped into several sub-images of a fixed size of 256*256.

[0016] Preferably, in step S202, the formula for calculating the contrast loss is:

[0017]

[0018] Where τ is a settable temperature hyperparameter, v i It supports feature vectors of images. It is a feature that matches the query image, which is from the same coloring as the supporting image. This is a feature that supports images derived from sub-images with different colors, N k Represents a set of subgraphs that have different colors from the supporting image.

[0019] Preferably, the network structure of the self-supervised pre-trained model is a ResNet-50 network.

[0020] Preferably, in step S3, the training method for the meta-learning segmentation model is as follows:

[0021] S301. Divide the labeled samples into multiple sub-images of a defined size; use the self-supervised pre-trained model as the first-level model and the Unet structure as the backbone network to build the second-level model;

[0022] S302. Randomly select multiple sub-images from one of the coloring methods and divide them into a meta-training set and a meta-test set.

[0023] S303. Assign the weights of the first-level model to the second-level model, and train the second-level model using the meta-training set data;

[0024] S304. After training once, the second-level model is tested using data from the meta-test set, and the weight parameters of the first-level model are updated using the backpropagation rule based on the test results.

[0025] S305. Return to step S302 and continue to randomly select a type of stained data. Repeat S302 to S305 until the loss of the first-level model converges, thus completing the training of the meta-learning segmentation model.

[0026] Preferably, in step S302, eight sub-images with one of the following coloring methods are randomly selected, with six of them serving as the meta-training set and two serving as the meta-test set.

[0027] Preferably, in step S301, the labeled sample is cropped into several sub-images of a fixed size of 256*256;

[0028] Preferably, in step S2, the remaining unlabeled samples from step S1 are treated as unlabeled samples.

[0029] Preferably, in step S301, a secondary model is constructed using a ResNet-50-based U-Net structure as the backbone network.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] This invention proposes a few-sample segmentation method for multi-stain kidney pathology images based on self-supervised meta-learning. It only requires a large number of easily accessible unlabeled samples and a small amount of difficult-to-obtain pixel-level annotation cost. With only a small annotation cost, a pre-trained model is constructed through self-supervised contrastive learning. Meta-learning methods are used to mine multi-tissue information in multiple domains and obtain the common features of each tissue in multiple staining domains. That is, the region of interest of different stained kidney pathology sections is efficiently segmented to achieve the purpose of region of interest segmentation in multi-stain kidney pathology images, and finally meet the clinical application needs of artificial intelligence in pathology. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the morphology of different tissues under multiple staining in kidney pathology images;

[0033] Figure 2 This is a flowchart illustrating a few-shot segmentation method for multi-stain kidney pathology images based on self-supervised meta-learning, which is involved in this invention.

[0034] Figure 3 This is a schematic diagram of the training process in the meta-learning training module involved in this invention.

[0035] Figure 4 These are segmentation results of multi-stained kidney tissue using the present invention and without the present invention; Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] like Figure 2 As shown, this invention designs a few-shot segmentation method for multi-stain renal pathology images based on self-supervised meta-learning. After digitizing the pathology slides, a self-supervised pre-trained model is trained with a large amount of unlabeled data. A meta-learning segmentation model is then trained in conjunction with a small amount of pixel-level labeled data using the self-supervised pre-trained model. This ultimately completes the task of segmenting multiple staining tissues in renal pathology images with minimal annotation costs. The specific steps include:

[0038] S1. Obtain full-field pathological images of multi-stained kidney pathological sections. At the same time, randomly select 2 to 5 digital pathological sections for each staining and manually annotate the shapes of glomeruli, sclerotic glomeruli, renal tubules and blood vessels at the pixel level to obtain annotated samples under different staining.

[0039] Specifically, in this embodiment, a KF-PRO-005-EX digital slide scanner is used to scan multi-stained kidney pathology slides to obtain full-field pathology images. During scanning, the slides are placed in the pathology slide scanner and digitized into 400x full-field slide images. The images are named according to the rule "abbreviation of staining method_patient pathology number" and then stored in the database for preservation, completing the digitization of the pathology slides. Subsequently, 2-5 digital pathology slides of each stain are randomly selected, and the pathologist uses LabelMe software to annotate the shape of glomeruli, sclerotic glomeruli, renal tubules, and blood vessels. This small amount of pixel-level labeled data is sent to the meta-learning segmentation training module, while the remaining images are sent to the self-supervised pre-training module as a large amount of unlabeled data. In this embodiment, the collected kidney pathology slide samples include four stains: H&E, PAS, PASM, and MASSON. Two digital pathology slides are randomly selected for each stain.

[0040] S2. Obtain unlabeled samples and train a self-supervised pre-trained model.

[0041] In step S2 of this embodiment, the unlabeled pathological image is cropped into several sub-images of a fixed size. Then, contrast loss is used to maximize the sub-images from different staining groups and minimize the sub-images from the same staining group, thus constructing a self-supervised pre-trained model. The specific training process is as follows:

[0042] S201. The unlabeled pathological image is cropped into several sub-images of a fixed size; in this embodiment, the unlabeled pathological image is cropped into several sub-images of a fixed size of 256*256.

[0043] S202. Randomly select two sub-images with the same coloring, use one sub-image as the support image and the other sub-image as the query image; use a convolutional neural network to extract features from the two selected images and calculate the contrast loss.

[0044] The formula for calculating the contrast loss is as follows:

[0045]

[0046] Where τ is a settable temperature hyperparameter, v i It supports feature vectors of images. It is a feature that matches the query image, which is from the same coloring as the supporting image. This is a feature that supports images derived from sub-images with different colors, N k It is a set of features of a sub-image that is colored differently from the supporting image.

[0047] S203. Update the parameters of the self-supervised pre-trained model based on the calculated contrastive loss.

[0048] S204. Repeat S202 to S203 until the contrastive loss converges to the minimum value and no longer decreases, thus completing the training of the self-supervised pre-trained model.

[0049] The self-supervised pre-trained model uses a ResNet-50 network. In step S2, the remaining unlabeled samples from step S1 are treated as unlabeled samples.

[0050] S3. Using the self-supervised pre-trained model as the first-level model, and the U-Net structure as the backbone network to construct the second-level model; randomly select one type of stained labeled samples, and randomly divide them into a meta-training set and a meta-test set. Assign the weights of the first-level model to the second-level model, train the second-level model through the meta-training set, and update the weight parameters of the second-level model through backpropagation. Test the second-level model through the corresponding stained meta-test set, and update the weight parameters of the first-level model according to the test results using the backpropagation rule. Randomly select data under one type of stained setting again, randomly divide it into a meta-training set and a meta-test set, and repeat the above training and testing process until the loss of the first-level model converges. Use the obtained first-level model as the meta-learning segmentation model.

[0051] In step S3, the self-supervised pre-trained model generated in step S2 is used as the base model. A small number of images with pixel-level annotations of tissues are used as the training set. Stainings are randomly selected and the common information of multiple stainings is extracted using meta-learning methods to train the meta-learning segmentation model.

[0052] Specifically, such as Figure 3 As shown, the training method for the meta-learning segmentation model is as follows:

[0053] S301. Cut the labeled samples into multiple sub-images of a defined size; use the self-supervised pre-trained model as the first-level model, and build the second-level model using the Unet structure as the backbone network.

[0054] In step S301, the labeled samples are cropped into several sub-images of a fixed size of 256*256. In step S301, a secondary model is constructed using a U-Net structure based on ResNet-50 as the backbone network.

[0055] S302. Randomly select multiple sub-images from one of the coloring methods and divide them into a meta-training set and a meta-test set.

[0056] Specifically, in step S302, eight sub-images with one of the following coloring methods are randomly selected, with six of them serving as the meta-training set and two serving as the meta-test set.

[0057] S303. Assign the weights of the first-level model to the second-level model, train the second-level model using the meta-training set data, and update the weight parameters of the second-level model.

[0058] S304. After training once, the secondary model is tested using data from the meta-test set, and the weight parameters of the primary model are updated based on the test results using the backpropagation rule. Specifically, in this embodiment, the parameters of the primary model are updated by comparing the test results with the true values.

[0059] S305. Return to step S302 and continue to randomly select a type of colored data. Repeat S302 to S304 until the loss of the first-level model converges to the minimum value and no longer decreases. The training of the model is then completed, and the weight parameters of the first-level model are used as the weight parameters of the final meta-learning segmentation model. The training of the meta-learning segmentation model is then completed.

[0060] Traditional single-model training methods randomly select images for training in each round. For multi-stained samples, this approach can lead to training confusion. That is, in one round of training, if images of a certain stain are used, the model will be more suited to that stain; however, in the next round, if images of a different stain are used, the model will be suited to that stain, resulting in the model being unable to accurately identify samples with different stains simultaneously. In this invention, a primary and secondary model are set up. The secondary model is responsible for the training process, focusing on the performance on the test set (i.e., the model's learning outcome) rather than the content learned in the current training set. This result is used to update the primary model, ensuring that the primary model focuses on the learning process rather than what was learned in the current round. Ultimately, a model capable of segmenting tissues under multiple staining domains is obtained, overcoming the shortcomings of existing single-model training methods.

[0061] S4. The pathological slides to be tested are segmented using a meta-learning segmentation model to obtain the final segmentation result.

[0062] Figure 4 A qualitative comparison of the segmentation results of this invention and the segmentation results using only the U-Net model is presented when four types of stained images are input. The experimental data were collected by Shanxi Provincial People's Hospital. 25 kidney biopsy images containing four types of staining were used as the training set and 125 kidney biopsy images were used as the test set. The above images were pixel-level annotated by two senior pathologists for the four types of tissues. The U-Net network and the method proposed in this invention were used to train the training set and to test the test set. Figure 4 This indicates that the test results of the present invention are closer to the true values, the segmentation contours are more complete, and the number of missed and false detections of tissues is lower than that of the results using the U-Net network. Table 1 shows a quantitative comparison between the segmentation results of the present invention and the segmentation results using only the U-Net model for four types of stained image inputs. Table 1 shows that the segmentation method of the present invention has a higher DICE coefficient and better quantification results than the method using only the U-Net network, and has a better segmentation effect.

[0063] Table 1

[0064]

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A segmentation method for multi-stain kidney pathological images based on self-supervised meta-learning, characterized in that, Includes the following steps: S1. Obtain full-field pathological images of kidney pathological sections. At the same time, randomly select 2-5 digital pathological sections for each staining and manually annotate the shapes of glomeruli, sclerotic glomeruli, renal tubules and blood vessels at the pixel level to obtain annotated samples under different staining conditions. S2. Obtain unlabeled samples and train a self-supervised pre-trained model; S3. Using the self-supervised pre-trained model as the first-level model, and the U-Net structure as the backbone network to construct the second-level model; randomly select one type of stained labeled samples, and randomly divide them into a meta-training set and a meta-test set. Assign the weights of the first-level model to the second-level model, train the second-level model through the meta-training set and update the weight parameters of the second-level model through backpropagation, test the second-level model through the corresponding stained meta-test set, and update the weight parameters of the first-level model according to the test results using the backpropagation rule; randomly select data under one type of stained data again, randomly divide it into a meta-training set and a meta-test set, and repeat the above training and testing process until the loss of the first-level model converges. The obtained first-level model is used as the meta-learning segmentation model. S4. The pathological slide to be tested is segmented using a meta-learning segmentation model to obtain the final segmentation result; in step S2, the training method of the self-supervised pre-training model is as follows: S201. Cropping the unlabeled sample image into several sub-images of fixed size; S202. Randomly select two sub-images with the same coloring, use one sub-image as the support image and the other sub-image as the query image; use a convolutional neural network to extract features from the two selected images and calculate the contrast loss. S203. Update the parameters of the self-supervised pre-trained model based on the calculated contrastive loss; S204. Repeat S202~S203 until the comparison converges, completing the training of the self-supervised pre-trained model.

2. The method for segmenting multi-stain kidney pathological images based on self-supervised meta-learning according to claim 1, characterized in that, In step S201, the unlabeled pathological image is cropped to a fixed size of 256. Several subgraphs of size 256.

3. The method for segmenting multi-stain kidney pathological images based on self-supervised meta-learning according to claim 1, characterized in that, In step S202, the formula for calculating the contrast loss is: ； in It is a configurable temperature hyperparameter. It supports feature vectors of images. It is a feature that matches the query image, which is from the same coloring as the supporting image. This is a feature that supports images derived from sub-images with different colors. Represents a set of subgraphs that have different colors from the supporting image.

4. The method for segmenting multi-stain kidney pathological images based on self-supervised meta-learning according to claim 1, characterized in that, The network structure of the self-supervised pre-trained model is the ResNet-50 network.

5. The method for segmenting multi-stain kidney pathological images based on self-supervised meta-learning according to claim 1, characterized in that, In step S3, the training method for the meta-learning segmentation model is as follows: S301. Divide the labeled samples into multiple sub-images of a defined size; use the self-supervised pre-trained model as the first-level model and the Unet structure as the backbone network to build the second-level model; S302. Randomly select multiple sub-images from one of the coloring methods and divide them into a meta-training set and a meta-test set. S303. Assign the weights of the first-level model to the second-level model, and train the second-level model using the meta-training set data; S304. After training once, the second-level model is tested using data from the meta-test set, and the weight parameters of the first-level model are updated using the backpropagation rule based on the test results. S305. Return to step S302 and continue to randomly select a type of colored data. Repeat S302 to S305 until the loss of the first-level model converges, thus completing the training of the meta-learning segmentation model.

6. The method for segmenting multi-stain kidney pathological images based on self-supervised meta-learning according to claim 5, characterized in that, In step S302, eight sub-images with one of the following coloring methods are randomly selected, with six of them serving as the meta-training set and two serving as the meta-test set.

7. The method for segmenting multi-stain kidney pathological images based on self-supervised meta-learning according to claim 5, characterized in that, In step S301, the labeled sample is cut to a fixed size of 256. Several subgraphs of size 256; In step S2, the remaining unlabeled samples from step S1 are treated as unlabeled samples.

8. The method for segmenting multi-stain kidney pathological images based on self-supervised meta-learning according to claim 5, characterized in that, In step S301, a secondary model is constructed using a U-Net structure based on ResNet-50 as the backbone network.

Images