A small sample medical image segmentation method, system, device and medium based on self-supervised learning

Abstract

A method, system, device and medium for small-sample medical image segmentation based on self-supervised learning. The method comprises: performing data enhancement on an unlabeled medical image dataset to obtain an enhanced medical image dataset, performing clustering processing using a self-supervised learning maskSLIC algorithm to generate pseudo-labels, performing data enhancement on the labeled medical image dataset and the pseudo-labels to obtain a support set and a query set, constructing a small-sample segmentation model based on a prototype network, inputting the support set and the query set into the above-mentioned model to obtain a prediction result after the medical image dataset is segmented, inputting the prediction result into the above-mentioned model to obtain a fine-tuned small-sample segmentation model, using the model for medical image segmentation, and obtaining a small-sample medical image segmentation result based on self-supervised learning. The system, device and medium are used to implement the small-sample medical image segmentation method based on self-supervised learning. The present invention has the characteristics of strong generalization ability, large amount of medical image data, abundant global diversity and good segmentation performance.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a small sample medical image segmentation method, system, device and medium based on self-supervised learning. Background Art

[0002] Medical image segmentation is the foundation of medical image analysis. Its main purpose is to separate the target region from the background in a medical image. Medical image segmentation often requires manual intervention, which is costly, labor-intensive, and prone to subjective errors. Therefore, research on how to automatically segment medical images using computer algorithms has become a hot research topic in the field of medical image analysis.

[0003] The existing medical image segmentation algorithms mainly include:

[0004] (1) Medical image segmentation based on U-Net

[0005] U-Net employs a symmetric encoder-decoder architecture with skipped layers. The decoder network uses an upsampling-convolution structure, which is simple, efficient, and easy to construct. This architecture has achieved impressive image segmentation results in medical imaging datasets with relatively small data volumes. Residual U-Net is an improvement on U-Net, combining a residual network with a U-Net. This architecture better utilizes contextual spatial information in the image, effectively reducing mispredictions and strengthening the learning of convolutional neural networks. It also prevents gradient degradation at deeper layers, ensuring that the essential image attributes are more clearly identified while extracting features at the network level. Furthermore, R2U-Net utilizes a recurrent residual convolutional network, enabling deeper network training and better feature extraction, achieving superior performance with the same number of parameters. However, such methods typically require large datasets for training. Medical image data is difficult to annotate, making it difficult to achieve good results with small datasets.

[0006] (2) Transformer-based medical image methods

[0007] The UNETR model integrates the visual Transformer model into the U-Net feature extraction framework for liver segmentation in 3D imaging data. Leveraging the Transformer, this model effectively establishes long-range connections and dependencies between features. Compared to the classic U-Net, UNETR can exploit a wider range of contextual semantic information, resulting in stronger segmentation capabilities for liver contours and complex lesions. In recent years, leveraging the Transformer's powerful ability to represent global contextual information and the advantages of CNNs in extracting local features, some researchers have proposed combining Transformers and CNNs to achieve the fusion of global and local features. For example, the TransUNet model retains the encoder-decoder network but introduces a Transformer module in the final decoder layer to compensate for the loss of global information and improve segmentation accuracy compared to pure Transformer methods. However, Transformer-based medical image methods cannot achieve good accuracy with small datasets, resulting in poor noise immunity, global diversity, and generalization.

[0008] (3) Medical image segmentation based on semi-supervision

[0009] The URPC model proposes an uncertainty-corrected pyramid consistency regularization method for semi-supervised medical image segmentation. While using a supervised loss to train the network for labeled data, it also employs a pyramid structure to generate multi-scale predictions for unlabeled data to enforce consistency. However, this method still requires a labeled dataset for training and cannot be trained on unlabeled datasets. Furthermore, this model struggles to address the foreground-background imbalance in medical images, resulting in poor segmentation performance.

[0010] The patent application with patent number [CN202211432379.4] provides "a full-field medical image region segmentation method based on self-supervised learning". First, data preprocessing is performed, and the process includes tissue region segmentation, sampling image blocks, and data enhancement through random cropping, Gaussian blur, color transformation, etc.; then, the image blocks are self-supervised pre-trained and the pre-trained model is saved; finally, a semantic segmentation network is constructed to complete the region segmentation; however, this method only performs well in the training data set, and the effect is poor when using other data sets, and it does not have strong generalization ability. Summary of the Invention

[0011] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a small sample medical image segmentation method, system, device and medium based on self-supervised learning. By adopting data enhancement strategy and medical image data enhancement strategy, integrating the self-supervised learning maskSLIC algorithm, utilizing the characteristics of small sample image segmentation, and adopting a small sample segmentation model based on the prototype network, it can also achieve good results even with a small data set. It is trained on labeled and unlabeled data sets, and has a wide range of applicable data sets, alleviating the pressure of medical data labeling, the problems of foreground-background imbalance and poor noise resistance. Therefore, it has the characteristics of strong generalization ability, abundant global diversity and good segmentation performance.

[0012] A small sample medical image segmentation method based on self-supervised learning includes the following steps:

[0013] Step 1: Obtain a medical image dataset of unlabeled images, and use a data enhancement strategy to perform data enhancement on the medical image dataset of unlabeled images to obtain an enhanced medical image dataset;

[0014] Step 2: Cluster the unlabeled medical image dataset and the enhanced medical image dataset obtained in step 1 using the self-supervised learning maskSLIC algorithm to generate pseudo labels;

[0015] Step 3: Use the medical image data augmentation strategy to perform data augmentation on the medical image dataset with annotated images and the pseudo labels obtained in step 2 to obtain the support set and query set;

[0016] Step 4: Build a small sample segmentation model based on the prototype network;

[0017] Step 5: Input the support set and query set obtained in step 3 into the small sample segmentation model obtained in step 4 to obtain the prediction results after the segmentation of the medical image dataset;

[0018] Step 6: Input the prediction result of the medical image dataset obtained in step 5 after segmentation into the small sample segmentation model obtained in step 4 to obtain a fine-tuned small sample segmentation model, and use the fine-tuned small sample segmentation model for medical image segmentation to obtain a small sample medical image segmentation result based on self-supervised learning.

[0019] The data enhancement strategy in step 1 includes:

[0020] 1) Horizontal flipping: This was performed using data augmentation operations in OpenCV (a cross-platform computer vision and machine learning software library released under the Apache 2.0 license) and the image class PIL method in the Python image processing library. First, a medical image dataset of unlabeled images was imported from PIL. Then, the images were cropped using Image.resize and rotated using Image.rotate.

[0021] 2) Random rotation: random angle rotation strategy is used for data enhancement;

[0022] 3) Random Cropping: Randomly select images from the medical image dataset of unlabeled images and perform random scale cropping operations to achieve data enhancement.

[0023] Step 2 uses the self-supervised learning maskSLIC algorithm to perform clustering on the unlabeled medical image dataset and the enhanced medical image dataset to generate pseudo labels, specifically:

[0024] Step 2.1: Obtain neighborhood cluster centers: Evenly distribute seed points for the medical image dataset after data augmentation in step 1 according to the set number of superpixels. Assume that the medical image dataset has a total of N pixels, which are pre-divided into K superpixels of the same size. Then the size of each superpixel is N / K, and the neighborhood cluster centers are obtained.

[0025] Step 2.2: Reselect a seed point within the neighborhood cluster center of the seed point: Calculate the gradient values of all pixels within the neighborhood cluster center obtained in step 2.1, and move the seed point to the point with the minimum gradient within the neighborhood cluster center.

[0026] Step 2.3: Assign a class label to each pixel in the neighborhood cluster center around each seed point obtained in step 2.2;

[0027] Step 2.4: Based on the assigned class labels obtained in step 2.3, use the Euclidean distance transform to iteratively find the seed point with the maximum distance from the neighborhood boundary and any other seed point;

[0028] The Euclidean distance transformation formula is as follows:

[0029]

[0030] Where D(x) represents the distance transform at position x, n represents the number of spatial dimensions, L is the set containing the background B and the marked point P, and B is the set of background labels without masks;

[0031] Maximum distance p * It can be obtained by the following formula:

[0032] p * =arg x maxD(x)

[0033] When the marked point P becomes P∪{p *}, corresponding to the next iteration, the iteration needs to be repeated N times until the seed point P with the maximum distance is obtained;

[0034] Step 2.5: Substitute the seed point with the maximum distance obtained in step 2.4 into the linear iterative clustering SLIC to obtain superpixels. The obtained superpixels are used to specify the pixels defined inside the label mask to generate pseudo labels.

[0035] The specific process of step 3 is:

[0036] Step 3.1: Using a medical image data enhancement strategy to enhance the image data to obtain a fused image; the medical image data enhancement strategy includes a global position scale enhancement module, a local position scale enhancement module, and a saliency fusion module;

[0037] Step 3.2: Use the saliency balance fusion strategy module to normalize the gradient value of the input channel of the fused image in step 3.1, and then ， Downsample the fused image after normalization, perform linear interpolation on the sampled modules and output the enhanced fused image;

[0038] Step 3.3: For each iteration i, each image x in the enhanced fused image obtained in step 3.2 i and the superpixels generated in step 2 To form the support set

[0039] The query set for each iteration is in and It is to perform random conversion on the images in the support set.

[0040] The specific process of step 3.1 is as follows:

[0041] Step 3.1.1: Use the global position and scale enhancement module to perform a constrained Bessel transform on the annotated medical image dataset and pseudo-labels to obtain a medical image dataset with enhanced image diversity. Calculate the standard deviation of the Gaussian distribution and select a position scale factor based on the difference. Calculate the saliency map by taking the norm of the gradient values on the input channels of the medical image dataset with enhanced image diversity, performing downsampling and linear interpolation operations to obtain the global maximum gradient region.

[0042] Step 3.1.2: Use the local position scaling module to perform a constrained Bessel transform on the medical image dataset with the annotated images of the maximum gradient region in the global image obtained in step 3.1.1 and the pseudo-labels to obtain a medical image dataset with enhanced image diversity. Then, calculate the standard deviation of the Gaussian distribution, select the position scaling factor based on the difference, and calculate the saliency map by taking the norm of the gradient values on the input channel of the medical image dataset with enhanced image diversity, performing downsampling operations, and linear interpolation operations to obtain the maximum gradient region in the foreground area.

[0043] Step 3.1.3: Fuse the global maximum gradient region obtained in step 3.1.1 with the maximum gradient region in the foreground region obtained in step 3.1.2 to obtain a fused image.

[0044] The step 4 is based on the small sample segmentation model of the prototype network, and the structure is:

[0045] 1) Feature extraction module: used to use the feature extraction function f θ (·) Extract features from the support set and query set to generate

[0046] 2) Adaptive local prototype pooling module: used to extract features from the support set Perform summation and mean operation to generate local prototype p l,mn (c) and class-level prototypes The foreground and background are divided by generating local prototypes and class-level prototypes;

[0047] 3) Similarity-based classifier module: used to calculate the similarity between the features extracted from the foreground in the support set and the query set, and obtain the classification result of the foreground in the query set;

[0048] The local prototype in the adaptive local prototype pooling module is represented as:

[0049]

[0050] in, Represents the image with sequence number l in the query set.

[0051] For each local prototype p l,mn Class c in (c) performs the following operations to perform category division:

[0052]

[0053] in, represents the superpixel generated in step 3 Use the average pooled value at position (m,n), where T is the lower threshold of the foreground, and T is between 0 and 1;

[0054] The adaptive local prototype pooling module is the class-level prototype in Expressed as:

[0055]

[0056] in, The query set is numbered 1 and the category is The binary mask of .

[0057] The specific process of step 5 is as follows:

[0058] Step 5.1: Input the support set and query set obtained in step 3 into the feature extraction module in step 4 for feature extraction to generate the support set and query set.

[0059] Step 5.2: Extract the features from the support set obtained in step 5.1 Input to the adaptive local prototype pooling module to generate local prototypes and class-level prototypes, and divide the generated local prototypes and class-level prototypes into foreground and background;

[0060] Step 5.3: Input the feature extraction of the foreground and query set obtained in step 5.2 into the similarity-based classifier module, calculate the similarity, and obtain the classification result of the foreground in the query set;

[0061] Step 5.4: The classification results of the foreground in the query set obtained in step 5.3 are normalized using a normalized exponential function to obtain the final prediction results after segmentation of the medical image dataset.

[0062] A small sample medical image segmentation system based on self-supervised learning, comprising:

[0063] Self-supervised module: This module uses the self-supervised learning maskSLIC algorithm to generate pseudo labels;

[0064] Data enhancement module: This module uses horizontal flipping, random cropping, random rotation and medical image data enhancement strategies to enhance medical image data;

[0065] Small sample image segmentation module: This module uses a fine-tuned small sample segmentation model to segment medical image tasks and obtains a small sample medical image segmentation result based on self-supervised learning.

[0066] A small sample medical image segmentation device based on self-supervised learning, comprising:

[0067] Memory: used for storing a computer program for implementing the small sample medical image segmentation method based on self-supervised learning;

[0068] A processor is used to implement the small sample medical image segmentation method based on self-supervised learning when executing the computer program.

[0069] A computer-readable storage medium comprising:

[0070] The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a small sample medical image segmentation method based on self-supervised learning.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] 1. The present invention uses the self-supervised learning maskSLIC algorithm to enhance the robustness of unlabeled medical image datasets. Compared with existing network structures such as U-Net and R2U-Net that do not use self-supervised learning algorithms, the present invention reduces the dependence on data labeling, thereby alleviating the pressure of medical image dataset labeling.

[0073] 2. The data enhancement strategies adopted in the present invention, such as horizontal flipping, random rotation, and random cropping, can more significantly improve the problem of small amount of medical image data compared to medical image segmentation methods that do not adopt data enhancement strategies, and therefore have the characteristics of good segmentation performance.

[0074] 3. The present invention adopts a medical image data enhancement strategy for medical images. Compared with the data enhancement strategy used in natural images, the medical image data enhancement strategy improves the problem of insufficient global enhancement diversity and performs class-specific transformation. Therefore, it has the characteristics of abundant global diversity and strong generalization ability.

[0075] 4. The present invention adopts a saliency balance fusion strategy to provide gradient information enhancement for the direction and intensity of specific classes in the foreground area, thereby improving the performance of medical image segmentation and thus having the characteristic of good segmentation performance.

[0076] 5. This invention utilizes the characteristics of small sample image segmentation. Compared with other current medical image segmentation networks, such as URPC, TransU-Net and other models, it adopts a small sample segmentation model based on the prototype structure, which effectively alleviates the problems of blurred foreground and background and poor noise resistance in medical images, and improves the generalization ability of the model. Therefore, it has the characteristics of strong generalization ability.

[0077] In summary, the present invention adopts data augmentation strategies and medical image data augmentation strategies, integrates the self-supervised learning maskSLIC algorithm, utilizes the characteristics of small sample image segmentation, and adopts a small sample segmentation model based on the prototype network. It can also achieve good results with a small data set. It is trained on labeled and unlabeled data sets, and has a wide range of applicable data sets, alleviating the pressure of medical data labeling, foreground-background imbalance, and poor noise resistance. Therefore, it has the characteristics of strong generalization ability, large amount of medical image data, abundant global diversity, and good segmentation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 Flow chart of the method of the present invention.

[0079] Figure 2 Flowchart for processing support and query sets for a few-shot prototype network model.

[0080] Figure 3 This is a partial result diagram of the maskSLIC self-supervised algorithm used in the present invention.

[0081] Figure 4 This is a comparison chart of the effects of the present invention and other mainstream methods. DETAILED DESCRIPTION

[0082] The present invention will be described in detail below with reference to the accompanying drawings.

[0083] See also Figure 1 ,Step 1: Obtain a medical image dataset of unlabeled images, and use data enhancement strategies to perform data enhancement on the medical image dataset of unlabeled images to obtain an enhanced medical image dataset; Due to the large variety of medical image datasets, we need to perform enhancement operations based on the specific characteristics of the medical image dataset. In common CT image processing, some common enhancement processing strategies can be performed, such as: horizontal flipping, random rotation, random cropping, and position scale enhancement operations in various directions. These operations can enhance data diversity and improve the robustness of the model, thereby improving the problem of small amount of medical image data;

[0084] The medical image dataset includes:

[0085] The ABIDE dataset is a multi-center, multinational MRI dataset that contains MRI scan images, gender and age information, cognitive and behavioral assessment data, etc. of 1,126 patients with autism and 1,181 typical people. These MRI scan images are generated by different imaging devices and use different imaging parameter settings, and the image quality and noise levels are also different. The ABIDE dataset provides a method for standardizing and correcting MRI data from different sources.

[0086] The CMR dataset collects a large amount of cardiac magnetic resonance imaging data, including data on papillary muscle movement, left ventricular morphology and function, coronary artery blood flow, and other aspects. This data provides important support for the diagnosis and treatment of cardiovascular diseases. In addition, the CMR dataset also contains a large amount of cardiac sample data, such as myocardial cells and vascular endothelial cells, which is also of great reference value for the study of the molecular mechanisms of heart disease.

[0087] The data enhancement strategy in step 1 includes:

[0088] 1) Horizontal flipping: This was performed using data augmentation operations in OpenCV (a cross-platform computer vision and machine learning software library released under the Apache 2.0 license) and the image class PIL method in the Python image processing library. First, a medical image dataset of unlabeled images was imported from PIL. Then, the images were cropped using Image.resize and rotated using Image.rotate.

[0089] 2) Random rotation: Use random angle rotation strategy for data augmentation; specifically: read the medical image dataset of unlabeled images, and then preprocess the images in the medical image dataset. First, use the image processing function transforms.resize to crop the image size of the medical image dataset, and then use the free angle rotation function transforms.Random.Rotation after the cropping to perform random rotation, where the first parameter is the rotation angle and the second parameter is the rotation center. Then, the randomly rotated medical image dataset is further cropped using the transforms.Resize function, and finally converted into a tensor variable.

[0090] 3) Random Cropping: Randomly select images from an unlabeled medical image dataset and perform random scale cropping to achieve data augmentation. Specifically, the unlabeled medical image dataset is read, the data is shuffled, and images are randomly selected from the dataset. The given images are randomly cropped to different sizes and aspect ratios using the RandomResizedCrop function in the image preprocessing package transforms.Compose. The cropped images are then scaled to the specified size. This means that images are randomly sampled and then scaled to the same size.

[0091] Step 2: Cluster the unlabeled medical image dataset and the enhanced medical image dataset obtained in step 1 using the self-supervised learning maskSLIC algorithm to generate pseudo labels;

[0092] The self-supervised learning maskSLIC algorithm in step 2 performs clustering processing, specifically:

[0093] Step 2.1: Obtain neighborhood cluster centers: Evenly distribute seed points for the medical image dataset after data augmentation in step 1 according to the set number of superpixels. Assume that the medical image dataset has a total of N pixels, which are pre-divided into K superpixels of the same size. Then the size of each superpixel is N / K, and the neighborhood cluster centers are obtained.

[0094] Step 2.2: Reselect a seed point within the neighborhood cluster center of the seed point: Calculate the gradient values of all pixels within the neighborhood cluster center obtained in step 2.1, and move the seed point to the location with the smallest gradient within the neighborhood cluster center. This is done to avoid the seed point falling on the contour boundary with a large gradient, which would affect the subsequent clustering effect.

[0095] Step 2.3: Assign a class label to each pixel in the neighborhood around each seed point obtained in step 2.2;

[0096] Step 2.4: Based on the assigned class labels obtained in step 2.3, use the Euclidean distance transform to iteratively find the seed point with the maximum distance from the neighborhood boundary and any other seed point;

[0097] The Euclidean distance transformation formula is as follows:

[0098]

[0099] Where D(x) represents the distance transform at position x, n represents the number of spatial dimensions, L is the set containing the background B and the marked point P, and B is the set of background labels without masks;

[0100] Maximum distance p * It can be obtained by the following formula:

[0101] p * =arg x maxD(x)

[0102] When the marked point P becomes P∪{p *}, corresponding to the next iteration, the iteration needs to be repeated N times until the seed point P with the maximum distance is obtained;

[0103] Step 2.5: Substitute the seed point with the maximum distance obtained in step 2.4 into the linear iterative clustering SLIC to obtain superpixels. The obtained superpixels are used to specify the pixels defined inside the label mask to generate pseudo labels.

[0104] Step 3: Perform data augmentation on the medical image dataset with annotated images and the pseudo labels obtained in step 2 to obtain the support set and query set;

[0105] Step 3.1: Use medical image data enhancement strategy to enhance the image data and obtain a fused image;

[0106] The medical image data enhancement module is divided into a global position scale enhancement module, a local position scale enhancement module and a saliency balance fusion module;

[0107] Step 3.1.1: Use the global position and scale enhancement module to perform a constrained Bessel transform on the annotated medical image dataset and pseudo-labels to obtain a medical image dataset with enhanced image diversity. Calculate the standard deviation of the Gaussian distribution and select a position scaling factor based on the difference to ensure that the globally position and scale enhanced medical image dataset has a similar appearance to the annotated medical image dataset and pseudo-labels. Obtain the maximum gradient region globally by taking the norm of the gradient values on the input channels of the enhanced image diversity dataset and performing downsampling and linear interpolation operations to calculate the saliency map.

[0108] Step 3.1.2: Use the local position scale enhancement module to perform a constrained Bessel transform on the annotated medical image dataset and pseudo-labels of the global maximum gradient region obtained in step 3.1.1 to obtain a medical image dataset with enhanced image diversity. Then, calculate the standard deviation of the Gaussian distribution and select a position scale factor based on the difference to ensure that the samples after local position scale enhancement have a similar appearance to the annotated medical image dataset and pseudo-labels. Obtain a saliency map by taking the norm of the gradient values on the input channels of the medical image dataset with enhanced image diversity, performing downsampling operations, and performing linear interpolation operations to obtain the maximum gradient region in the foreground area.

[0109] Step 3.1.3: Fuse the maximum gradient region in the global image obtained in step 3.1.1 with the maximum gradient region in the foreground image obtained in step 3.1.2 to obtain a fused image. The local position scale enhanced image is fused with the global position scale enhanced image through the saliency map. The large gradient regions in the global position scale enhanced image are retained, and the remaining regions are replaced with the local position scale enhanced regions. These regions can observe changes in appearance. This module only performs nonlinear transformation operations on the foreground region. The use of the position scale enhancement strategy can enhance the diversity and information content of the samples, making the segmentation results more accurate.

[0110] Step 3.2: Since the fused image contains two parts of the enhanced image, the saliency balance fusion strategy encourages the model to show linearity between the global position scale enhancement and the local position scale enhancement examples. It uses the gradient distribution as an indicator of the enhancement amplitude to solve the over-generalization problem of random enhancement. The saliency balance fusion strategy module is used to normalize the gradient value of the input channel of the fused image in step 3.1, and then ， The normalized fused image is downsampled, and the sampled module is linearly interpolated and smoothed to output the enhanced fused image. Compared with directly inputting local position scale enhanced samples, this linear behavior reduces the number of undesirable oscillations when predicting out-of-distribution samples, providing enhanced direction and strength for appropriate gradient information.

[0111] Step 3.3: For each iteration i, each image x in the enhanced fused image obtained in step 3.2 i and the superpixels generated in step 2 To form the support set

[0112] The query set for each iteration is in and It is to perform random conversion on the images in the support set.

[0113] The step 4 is based on the small sample segmentation model of the prototype network, and the structure is:

[0114] 1) Feature extraction module: used to use the feature extraction function f θ (·) Extract features from the support set and query set to generate

[0115] 2) Adaptive local prototype pooling module: used to extract features from the support set Perform summation and mean operation to generate local prototype p l,mn (c) and class-level prototypes The foreground and background are divided by generating local prototypes and class-level prototypes;

[0116] 3) Similarity-based classifier module: used to calculate the similarity between the features extracted from the foreground in the support set and the query set, and obtain the classification result of the foreground in the query set;

[0117] The adaptive local prototype pooling module is the local prototype in , which is expressed as:

[0118]

[0119] in, Represents the image with sequence number l in the query set.

[0120] In order for each local prototype p l,mn (c) Average the binary masks of the foreground class of class c to the same size and perform the following operations:

[0121]

[0122] in, represents the superpixel generated in step 3 Use the average pooled value at position (m,n). T is the lower threshold of the foreground, and T is set to 0.95;

[0123] The adaptive local prototype pooling module is the class-level prototype in Expressed as:

[0124]

[0125] in, The query set is numbered 1 and the category is The binary mask of .

[0126] Step 4: Construct a small sample segmentation model based on the prototype network; the structure of the small sample segmentation model based on the prototype network in step 4 is:

[0127] 1) Feature extraction module: used to use the feature extraction function f θ (·) Extract features from the support set and query set to generate

[0128] 2) Adaptive local prototype pooling module: used to extract features from the support set Perform summation and mean operation to generate local prototype p l,mn (c) and class-level prototypes The foreground and background are divided by generating local prototypes and class-level prototypes;

[0129] 3) Similarity-based classifier module: used to calculate the similarity between the features extracted from the foreground in the support set and the query set, and obtain the classification result of the foreground in the query set;

[0130] The local prototype in the adaptive local prototype pooling module is represented as:

[0131]

[0132] in, Represents the image with sequence number l in the query set.

[0133] In order for each local prototype p l,mn (c) Average the binary masks of the foreground class of class c to the same size and perform the following operations:

[0134]

[0135] in, represents the superpixel generated in step 3 Use the average pooled value at position (m,n), where T is the lower threshold of the foreground, and T is between 0 and 1;

[0136] Class-level prototypes in the adaptive local prototype pooling module Expressed as:

[0137]

[0138] in, The query set is numbered 1 and the category is The binary mask of ;

[0139] To improve the accuracy of the prototype network-based small-shot segmentation model, the support set and query set are pre-trained using a pre-trained model before being fed into the model. The pre-trained model uses the weights of the residual network ResNet-101 pre-trained on the real image dataset MS-COCO as the backbone network for feature extraction. The network consists of a 7×7 convolutional layer with a stride of 2, a maximum pooling layer, and four residual blocks. The output features of the first residual block in the network are 4 times smaller than the input resolution, and the network performs a triple downsampling operation on the last three dilated residual blocks.

[0140] See also Figure 2 , Step 5: Input the support set and query set obtained in step 3 into the small sample segmentation model obtained in step 4 to obtain the prediction results after the segmentation of the medical image dataset; the specific process is:

[0141] Step 5.1: Input the support set and query set obtained in step 3 into the feature extraction module in step 4 to extract features and generate the support set and query set.

[0142] Step 5.2: Extract the features from the support set obtained in step 5.1 Input to the adaptive local prototype pooling module to generate local prototypes and class-level prototypes, and divide the generated local prototypes and class-level prototypes into foreground and background;

[0143] Step 5.3: Input the feature extraction of the foreground and query set obtained in step 5.2 into the similarity-based classifier module, calculate the similarity, and obtain the classification result of the foreground in the query set;

[0144] Step 5.4: Normalize the classification results of the foreground in the query set obtained in step 5.3 using the Softmax function to obtain the final prediction results after segmentation of the medical image dataset.

[0145] The stochastic gradient descent algorithm with a batch size of 1 is used to minimize the batch size of 100,000 iterations of building the small sample segmentation model in step 4, where the learning rate is set to 0.001 and the decay rate is set to 0.98 every 1,000 iterations.

[0146] Step 6: Input the prediction result of the medical image dataset obtained in step 5 after segmentation into the small sample segmentation model obtained in step 4 to obtain a fine-tuned small sample segmentation model, and use the fine-tuned small sample segmentation model for medical image segmentation to obtain a small sample medical image segmentation result based on self-supervised learning.

[0147] See also Figure 3 ,in, Figure 3 (a) is the original medical image. Figure 3 (b) Generate pseudo labels using the maskSLIC algorithm. It can be seen that the use of the self-supervised learning algorithm maskSLIC can cluster similar features together, capture local similarities, reduce image redundancy, and speed up processing, making complex regional relationship analysis more feasible. It also obtains a more robust representation of noise and partial volume.

[0148] In addition, to make the experimental results more realistic and reliable, the present invention sets the ratio of the image dataset of the query set and the support set, and divides the query set into a test class and a training class for comparative experiments. The following two settings are adopted:

[0149] Setting 1: The standard setting, where the test class in the query set can appear in the context of the training classes in the query set, and objects of the test class may be implicitly involved in the training process. In this case, the test class is not truly “unseen” for the prototype-based few-shot segmentation model.

[0150] Setting 2: Strict setting. The test class in the query set cannot appear in the background of the training class in the query set. The present invention deletes the image slice training data containing the test class from the training class in the query set. This setting mode can ensure that the test class is truly "invisible" to the model.

[0151] In order to evaluate the performance of the small sample medical image segmentation model proposed in the present invention, the experimental results of the present invention are compared with some excellent methods currently designed for medical image segmentation.

[0152] Table 1: Experimental results obtained using standard setting 1 on the ABIDE dataset:

[0153] Method Liver R.kidney L.kidney Spleen MEAN ALPNet 78.55 83.11 78.16 70.58 77.60 Q-Net 81.74 87.98 78.36 75.99 81.02 SE-Net 20.02 47.96 45.78 47.30 42.51 PANet 76.81 83.35 75.05 75.77 77.74 Ours 82.12 87.88 80.11 76.02 81.53

[0154] Table 2: Experimental results using strict setting 2 on the ABIDE dataset:

[0155] Method Liver R.kidney L.kidney Spleen MEAN ALPNet 72.53 81.15 68.04 71.00 78.43 Q-Net 80.44 75.98 76.26 74.19 76.71 SE-Net 27.43 61.32 62.11 51.80 50.66 PANet 73.44 79.21 77.05 73.48 75.79 Ours 81.21 81.58 78.21 75.99 79.24

[0156] As can be seen from Tables 1 and 2, on the ABIDE dataset, the self-supervised small-sample medical image segmentation method adopted by the present invention (Ours) achieves significantly better results than the ALPNet, Q-Net, SE-Net, and PANet methods in the Liver, R.kidney, L.kidney, and Spleen classifications. In particular, the test results in the L.kidney and Liver classifications are significantly improved. This is mainly due to the fact that the small-sample prototype algorithm proposed in the present invention designs special local prototype and class-level prototype modules to address the foreground and background blur problem.

[0157] Table 3: Experimental results using standard setting 1 on the CMR dataset:

[0158] Method LV-BP LV-MYO RV MEAN ALPNet 85.42 63.38 74.07 74.29 Q-Net 88.25 56.92 78.19 74.45 SE-Net 58.04 25.18 12.86 32.02 PANet 87.63 60.23 71.98 73.28 Ours 89.21 63.11 73.02 75.11

[0159] Table 4: Experimental results using strict setting 2 on the CMR dataset:

[0160] Method LV-BP LV-MYO RV MEAN ALPNet 83.12 58.34 71.28 70.91 Q-Net 82.10 52.13 78.10 70.77 SE-Net 53.04 23.10 15.53 30.55 PANet 81.07 61.54 69.28 70.63 Ours 84.23 61.28 71.12 72.21

[0161] As can be seen from Tables 3 and 4, on the CMR dataset, the self-supervised small sample medical image segmentation method adopted by the present invention (Ours) achieves better test accuracy than the ALPNet, Q-Net, SE-Net, and PANet methods in LV-BP, LV-MYO, and RV classification, especially in LV-BP classification, where the test accuracy is significantly improved.

[0162] In summary, since there are some objects in the test class in the training class background in the standard setting 1 of the present invention, the experimental accuracy of the standard setting 1 is higher than that of the strict setting 2. However, it can be seen that due to the improvement strategies made by the present invention in data processing and prototype network, compared with other medical image segmentation models, the accuracy difference between the standard setting 1 and the strict setting 2 of the present invention is not very large, which also reflects the strong applicability of the present invention from another aspect.

[0163] See also Figure 4, it can be seen that the visual comparison results of the medical image segmentation method adopted by the present invention (Ours) and the Q-Net, ALPNet, SENet, PANet, and groundtruth medical image segmentation methods show that the model results proposed by the present invention are more accurate both in the segmentation boundaries of key organs and in the segmentation range, and the results are closer to the true values; especially compared with the SE-Net and ALPNet networks that incorrectly divide other parts, the superiority of the experimental results of the present invention can be seen even more clearly.

[0164] A small sample medical image segmentation system based on self-supervised learning, comprising:

[0165] Self-supervised module: This module uses the self-supervised learning maskSLIC algorithm to generate pseudo labels. Using these superpixels for training can intuitively improve the network's generalization ability for unseen classes of various shapes;

[0166] Data augmentation module: This module uses horizontal flipping, random cropping, random rotation, and position and scale augmentation strategies to perform data augmentation on medical images, improving the information content and diversity of the enhanced images;

[0167] Small sample image segmentation module: This module uses a fine-tuned small sample segmentation model to segment medical image tasks, obtaining a small sample medical image segmentation result based on self-supervised learning, which improves the accuracy of image segmentation.

[0168] A small sample medical image segmentation device based on self-supervised learning, comprising:

[0169] Memory: used for storing a computer program for implementing the small sample medical image segmentation method based on self-supervised learning;

[0170] A processor is used to implement the small sample medical image segmentation method based on self-supervised learning when executing the computer program.

[0171] A computer-readable storage medium comprising:

[0172] The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a small sample medical image segmentation method based on self-supervised learning.

Claims

1. A small sample medical image segmentation method based on self-supervised learning, characterized in that: The following steps are involved: Step 1: Obtain a medical image dataset of unlabeled images, and use a data enhancement strategy to perform data enhancement on the medical image dataset of unlabeled images to obtain an enhanced medical image dataset; Step 2: Cluster the unlabeled medical image dataset and the enhanced medical image dataset obtained in step 1 using the self-supervised learning maskSLIC algorithm to generate pseudo labels; Step 3: Use the medical image data augmentation strategy to perform data augmentation on the medical image dataset with annotated images and the pseudo labels obtained in step 2 to obtain the support set and query set; The specific process of step 3 is: Step 3.1: Using a medical image data enhancement strategy to enhance the image data to obtain a fused image; the medical image data enhancement strategy includes a global position scale enhancement module, a local position scale enhancement module, and a saliency fusion module; The specific process of step 3.1 is as follows: Step 3.1.1: Use the global position and scale enhancement module to perform a constrained Bessel transform on the annotated medical image dataset and pseudo-labels to obtain a medical image dataset with enhanced image diversity. Calculate the standard deviation of the Gaussian distribution and select a position scale factor based on the difference. Calculate the saliency map by taking the norm of the gradient values on the input channels of the medical image dataset with enhanced image diversity, performing downsampling and linear interpolation operations to obtain the global maximum gradient region. Step 3.1.2: Use the local position scaling module to perform a constrained Bessel transform on the medical image dataset with the annotated images of the maximum gradient region in the global image obtained in step 3.1.1 and the pseudo-labels to obtain a medical image dataset with enhanced image diversity. Then, calculate the standard deviation of the Gaussian distribution, select the position scaling factor based on the difference, and calculate the saliency map by taking the norm of the gradient values on the input channel of the medical image dataset with enhanced image diversity, performing downsampling operations, and linear interpolation operations to obtain the maximum gradient region in the foreground area. Step 3.1.3: Fuse the global maximum gradient region obtained in step 3.1.1 with the maximum gradient region in the foreground region obtained in step 3.1.2 to obtain a fused image; Step 3.2: Use the saliency balance fusion strategy module to normalize the fused image in step 3.1 through the gradient value of the input channel. Then, downsample the normalized fused image and perform linear interpolation on the sampled module to output the enhanced fused image. Step 3.3: For each iteration i, each image x in the enhanced fused image obtained in step 3.2 i and the superpixels generated in step 2 To form the support set The query set for each iteration is in and It is to perform random conversion on the images in the support set; Step 4: Build a small sample segmentation model based on the prototype network; Step 5: Input the support set and query set obtained in step 3 into the small sample segmentation model obtained in step 4 to obtain the prediction results after the segmentation of the medical image dataset; Step 6: Input the prediction result of the medical image dataset obtained in step 5 after segmentation into the small sample segmentation model obtained in step 4 to obtain a fine-tuned small sample segmentation model, and use the fine-tuned small sample segmentation model for medical image segmentation to obtain a small sample medical image segmentation result based on self-supervised learning.

2. The small sample medical image segmentation method based on self-supervised learning according to claim 1, characterized in that: The data enhancement strategy in step 1 includes: 1) Horizontal flipping: This was performed using data augmentation operations in OpenCV (a cross-platform computer vision and machine learning software library released under the Apache 2.0 license) and the image class PIL method in the Python image processing library. First, a medical image dataset of unlabeled images was imported from PIL. Then, the images were cropped using Image.resize and rotated using Image.rotate. 2) Random rotation: random angle rotation strategy is used for data enhancement; 3) Random Cropping: Randomly select images from the medical image dataset of unlabeled images and perform random scale cropping operations to achieve data enhancement.

3. The small sample medical image segmentation method based on self-supervised learning according to claim 1, characterized in that: Step 2 uses the self-supervised learning maskSLIC algorithm to perform clustering on the unlabeled medical image dataset and the enhanced medical image dataset to generate pseudo labels, specifically: Step 2.1: Obtain neighborhood cluster centers: Evenly distribute seed points for the medical image dataset after data augmentation in step 1 according to the set number of superpixels. Assume that the medical image dataset has a total of N pixels, which are pre-divided into K superpixels of the same size. Then the size of each superpixel is N / K, and the neighborhood cluster centers are obtained. Step 2.2: Reselect a seed point within the neighborhood cluster center of the seed point: Calculate the gradient values of all pixels within the neighborhood cluster center obtained in step 2.1, and move the seed point to the point with the minimum gradient within the neighborhood cluster center. Step 2.3: Assign a class label to each pixel in the neighborhood cluster center around each seed point obtained in step 2.2; Step 2.4: Based on the assigned class labels obtained in step 2.3, use the Euclidean distance transform to iteratively find the seed point with the maximum distance from the neighborhood boundary and any other seed point; The Euclidean distance transformation formula is as follows: Where D(x) represents the distance transform at position x, n represents the number of spatial dimensions, L is the set containing the background B and the marked point P, and B is the set of background labels without masks; Maximum distance p * It can be obtained by the following formula: p * =arg x maxD(x) When the marked point P becomes P∪{p * }, corresponding to the next iteration, the iteration needs to be repeated N times until the seed point P with the maximum distance is obtained; Step 2.5: Substitute the seed point with the maximum distance obtained in step 2.4 into the linear iterative clustering SLIC to obtain superpixels. The obtained superpixels are used to specify the pixels defined inside the label mask to generate pseudo labels.

4. The small sample medical image segmentation method based on self-supervised learning according to claim 1, characterized in that: The step 4 is based on the small sample segmentation model of the prototype network, and the structure is: 1) Feature extraction module: used to use the feature extraction function f θ (·) Extract features from the support set and query set to generate the support set and query set 2) Adaptive local prototype pooling module: used to extract features from the support set Perform summation and mean operation to generate local prototype p l,mn (c) and class-level prototypes The foreground and background are divided by generating local prototypes and class-level prototypes; 3) Similarity-based classifier module: used to calculate the similarity between the features extracted from the foreground in the support set and the query set, and obtain the classification result of the foreground in the query set; The local prototype in the adaptive local prototype pooling module is represented as: in, represents the image with sequence number l in the query set; For each local prototype p l,mn Class c in (c) performs the following operations to perform category division: in, represents the superpixel generated in step 3 Use the average pooled value at position (m,n), where T is the lower threshold of the foreground, and T is between 0 and 1; Class-level prototypes in the adaptive local prototype pooling module Expressed as: in, The query set is numbered 1 and the category is The binary mask of .

5. The small sample medical image segmentation method based on self-supervised learning according to claim 1, characterized in that: The specific process of step 5 is as follows: Step 5.1: Input the support set and query set obtained in step 3 into the feature extraction module in step 4 for feature extraction to generate the support set and query set Step 5.2: Extract the features from the support set obtained in step 5.1 Input to the adaptive local prototype pooling module to generate local prototypes and class-level prototypes, and then divide the generated local prototypes and class-level prototypes into foreground and background; Step 5.3: Input the feature extraction of the foreground and query set obtained in step 5.2 into the similarity-based classifier module, calculate the similarity, and obtain the classification result of the foreground in the query set; Step 5.4: The classification results of the foreground in the query set obtained in step 5.3 are normalized using a normalized exponential function to obtain the final prediction results after segmentation of the medical image dataset.

6. A small sample medical image segmentation system based on self-supervised learning based on the method of claim 1, characterized in that: include: Self-supervised module: This module uses the self-supervised learning maskSLIC algorithm to generate pseudo labels; Data enhancement module: This module uses horizontal flipping, random cropping, random rotation and medical image data enhancement strategies to enhance medical image data; Small sample image segmentation module: This module uses a fine-tuned small sample segmentation model to segment medical image tasks and obtains a small sample medical image segmentation result based on self-supervised learning.

7. A small sample medical image segmentation device based on self-supervised learning, characterized in that: include: Memory: used to store a computer program for implementing the small sample medical image segmentation method based on self-supervised learning according to any one of claims 1 to 5; A processor, configured to implement the small sample medical image segmentation method based on self-supervised learning as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement the small-sample medical image segmentation method based on self-supervised learning as described in any one of claims 1 to 5.

Images