Medical image segmentation method and system based on basic model assisted semi-supervised learning

Abstract

The invention discloses a medical image segmentation method and system based on basic model assisted semi-supervised learning. In a medical image segmentation task, by introducing an innovative method of combining a visual basic model (SAM) and a U-Net model, the labeling precision and the segmentation effect are remarkably improved. Compared with an existing semi-supervised learning method, the method has the advantages that the problem of low-quality false labels is solved by introducing a reliability scoring mechanism and a foreground conflict processing strategy, and unreliable images are effectively utilized through a region exchange strategy. Finally, the invention provides a new solution for solving the problems of low false label quality and high data annotation cost in medical image segmentation, greatly improves the performance of medical image segmentation, and shows an excellent segmentation effect on a plurality of reference data sets.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and medical image processing, and in particular, relates to a medical image segmentation method and system based on basic model-assisted semi-supervised learning. Background Art

[0002] In recent years, medical image segmentation has provided important information in disease diagnosis, quantitative analysis, and treatment planning, effectively improving doctors' work efficiency and accuracy. Although deep learning technology has achieved certain success in the field of medical image segmentation, traditional medical image segmentation methods usually rely on a large amount of high-quality annotated data for training, and the cost of obtaining these annotated data is extremely high. The annotation of medical images requires professional knowledge and rich experience, and is often limited by data availability and privacy policies. Especially in three-dimensional volumetric medical imaging modes (such as CT and MRI), each slice needs to be carefully depicted, which greatly increases the workload of manual annotation.

[0003] To address this challenge, semi-supervised learning (SSL) has been widely used in medical image segmentation. Existing SSL methods can effectively utilize a large amount of unlabeled data to enhance the generalization ability of the model, and usually include two training processes: consistency regularization and pseudo-label generation. However, these methods still have a significant performance gap with fully supervised learning methods when there is very little labeled data. This gap mainly stems from inappropriate perturbations and low-quality pseudo-labels, which lead to inaccurate decision boundaries and thus affect the learning quality of the model. In addition, over-reliance on traditional models (such as U-Net) may not be able to effectively cope with complex segmentation problems.

[0004] In recent years, visual base models represented by Segment Anything Model (SAM) have demonstrated excellent generalization capabilities in the field of image segmentation and have attracted a lot of research attention. Although SAM can complete medical image segmentation tasks after fine-tuning, its performance is still inferior to traditional semi-supervised methods. Considering that SAM can provide useful labels as prior information, it may be combined with SSL methods to further improve the performance of medical image segmentation. However, the semi-supervised learning method that applies the excellent generalization ability of SAM to medical image segmentation is still in the exploratory stage.

[0005] Existing methods have the following main problems when dealing with medical image segmentation:

[0006] Low-quality pseudo-labels: When existing semi-supervised learning methods generate pseudo-labels, the quality of the pseudo-labels is low, resulting in inaccurate decision boundaries of the model, which affects the quality of learning and segmentation effect.

[0007] Inadequate response to complex segmentation problems: Traditional methods show significant performance degradation when dealing with complex medical image segmentation problems, especially when there is very little labeled data, and cannot meet the requirements of practical applications.

[0008] Therefore, designing a medical image segmentation method that can effectively combine visual basic models (such as SAM) with semi-supervised learning methods to meet the above challenges and improve the performance of medical image segmentation is a technical problem that needs to be solved urgently. Summary of the invention

[0009] The purpose of the present invention is to solve the problem of how to effectively utilize unlabeled data and improve segmentation accuracy and robustness in the medical image segmentation task in the prior art, and to provide a medical image segmentation method and system based on basic model-assisted semi-supervised learning. The present invention combines the advantages of the U-Net model and the Segment Anything Model (SAM) to achieve effective utilization of unlabeled data and generate high-precision segmentation results.

[0010] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0011] In a first aspect, the present invention provides a medical image segmentation method based on basic model-assisted semi-supervised learning, and the specific process is as follows: a medical image to be segmented is input into a trained medical image segmentation model to obtain a segmentation result of the medical image; wherein the medical image segmentation model is obtained by fine-tuning a pre-trained U-Net model on an unlabeled data set and a labeled data set; the pre-trained U-Net model is obtained by training the U-Net model on a labeled data set in a supervised learning mode; the unlabeled data set only contains unlabeled medical images, and the labeled data set contains both labeled medical images and true labels;

[0012] During the fine-tuning process, the unlabeled medical image is input into the pre-trained U-Net model, a coarse segmentation probability map of the unlabeled medical image is generated and used as the first coarse segmentation probability map, the category with the largest probability value in the first coarse segmentation probability map is taken as the coarse segmentation pseudo-label of the unlabeled medical image and used as the first coarse segmentation pseudo-label, the prompt information required by the visual base model is generated based on the first coarse segmentation pseudo-label, the generated prompt information is passed through the prompt encoder of the visual base model to obtain the prompt feature, and the unlabeled medical image is processed by the encoder of the visual base model to obtain the image feature of the unlabeled medical image, and then the image feature and the prompt feature of the unlabeled medical image are input into the mask decoder of the visual base model, and the mask decoder generates a set of SAM foreground images, and the intersection-union ratio between the SAM foreground image and the first coarse segmentation pseudo-label of each category is calculated, and the calculation results of the intersection-union ratio are averaged to obtain the consistency score, which is sorted in descending order of the consistency score. The unlabeled medical images are classified and labeled, and the unlabeled medical images marked as reliable are processed through the joint judgment of SAM and U-Net and the foreground conflict processing strategy to generate the first fine segmentation pseudo-label. The unlabeled medical images marked as reliable and the first fine segmentation pseudo-label constitute a reliable data set, and the unlabeled medical images marked as unreliable constitute an unreliable data set. The unreliable data set is processed by the region exchange strategy, and a bidirectional displacement operation is performed between the labeled medical images and the unlabeled medical images marked as unreliable to generate synthetic images and synthetic labels as supervision in the fine-tuning process, and a perturbation operation is performed on each unlabeled medical image. For each unlabeled medical image, a weakly perturbed image and two independent strongly perturbed images are generated. The pre-trained U-Net model generates corresponding probability maps for the weakly perturbed image and each strongly perturbed image. The probability map of the weakly perturbed image is used as a supervisory signal to perform supervised learning on the probability maps of the two strongly perturbed images.

[0013] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0014] As a preferred embodiment of the first aspect, the network structure of the U-Net model is symmetrical and is divided into a downsampling path and an upsampling path;

[0015] In the downsampling path, the input medical image is first processed by the first convolution layer to generate a first feature map, the first feature map is downsampled by the maximum pooling layer to obtain a low-resolution first feature map, then the low-resolution first feature map is processed by the second convolution layer in turn to generate a second feature map, the second feature map is downsampled again by the maximum pooling layer to obtain a low-resolution second feature map, the low-resolution second feature map is processed by the third convolution layer to generate a third feature map, the third feature map is subjected to the maximum pooling operation to generate a low-resolution third feature map, the low-resolution third feature map is processed by the fourth convolution layer to generate a fourth feature map, the fourth feature map is subjected to the maximum pooling operation to obtain a low-resolution fourth feature map, finally, the low-resolution fourth feature map is processed by the fifth convolution layer to generate the fifth feature map and used as the input of the upsampling path;

[0016] In the upsampling path, the fifth feature map is upsampled and then spliced ​​with the fourth feature map in the downsampling path. After splicing, it is processed by the sixth convolution layer to generate the sixth feature map. After upsampling, the sixth feature map is spliced ​​with the third feature map in the downsampling path. After splicing, it is processed by the seventh convolution layer to generate the seventh feature map. After upsampling, the seventh feature map is spliced ​​with the second feature map in the downsampling path. After splicing, it is processed by the eighth convolution layer to generate the eighth feature map. After upsampling, the eighth feature map is spliced ​​with the first feature map in the downsampling path. After splicing, it is processed by the ninth convolution layer to generate the ninth feature map. Finally, the ninth feature map is processed by an output convolution layer, and the output channel number corresponds to the target classification number. The final segmentation result.

[0017] Furthermore, the convolution kernel size of the U-Net model is configured as follows:

[0018] In the downsampling path, all convolutional layers use a 3×3 convolution kernel size to gradually extract features, and a 2×2 maximum pooling layer is used to reduce the resolution of the feature map and generate a deeper feature map;

[0019] In the upsampling path, the convolution layers after splicing all use a convolution kernel size of 3×3, which is used to further process the spliced ​​feature map after each upsampling operation, thereby restoring the resolution of the medical image and extracting richer feature information; upsampling is achieved by transposed convolution with a convolution kernel size of 2×2 to increase the resolution of the feature map; the convolution kernel size of the output convolution layer is 1×1, which is used to convert the last generated feature map into the number of channels corresponding to the number of target classifications to generate the final segmentation result.

[0020] As a preferred embodiment of the first aspect, the specific generation process of the prompt information required by the visual base model is as follows: in the first coarse segmentation pseudo-label, only the largest continuous area in the non-empty foreground mask of each category except the background is always retained, and after obtaining the continuous area, the prompt information is preset, and the prompt information is a sparse prompt and / or a dense prompt, the sparse prompt is a point and / or a border, and the dense prompt is a category-level foreground mask captured from the first coarse segmentation pseudo-label;

[0021] If the sparse hint is a border, the minimum circumscribed rectangular area of ​​the continuous area is used as the border representing the sparse hint; if the sparse hint is a point, it is generated in an iterative manner. In each iteration, a point is sampled from the unlabeled medical image until the iteration ends, and all the sampled points are used as a set of points representing the sparse hint.

[0022] As a preferred embodiment of the first aspect above, the specific process of classifying and labeling the unlabeled medical images in descending order of consistency scores is as follows: the w unlabeled medical images with the lowest consistency scores are regarded as bottom medical images, the unlabeled medical images with consistency scores not lower than a preset threshold are marked as reliable, the unlabeled medical images with consistency scores lower than the preset threshold and belonging to bottom medical images are marked as unreliable, and the remaining unlabeled medical images are marked as neutral and no additional processing is performed.

[0023] As a preferred embodiment of the first aspect, the unlabeled medical image marked as reliable is processed by the foreground conflict processing strategy, and the specific process of generating the first fine segmentation pseudo-label is: taking the all-zero matrix as the initial state of the first fine segmentation pseudo-label, when a pixel in the unlabeled medical image marked as reliable is only marked as foreground by one category mask k of the SAM foreground map, the pixel is assigned to the category in the category mask, and the value of the point with the same position as the pixel in the all-zero matrix is ​​changed to k, when a pixel in the unlabeled medical image marked as reliable is simultaneously marked as foreground by multiple category masks of the SAM foreground map, the pixel is assigned to the corresponding category k in the first coarse segmentation pseudo-label, and the value of the point with the same position as the pixel in the all-zero matrix is ​​changed to k, until all the pixels in the unlabeled medical image marked as reliable are traversed to obtain a complete first fine segmentation pseudo-label.

[0024] As a preferred embodiment of the first aspect, the region exchange strategy is used to process the unreliable data set, and the specific process of generating the synthetic image and the synthetic label is as follows: for an unreliable unlabeled medical image marked as unreliable, a labeled medical image is randomly selected, and a binary mask with a value of 1 in the foreground area and 0 in the rest is pre-constructed, the size of the binary mask foreground area is smaller than the size of the unreliable unlabeled medical image marked as unreliable, all pixels in the foreground area of ​​the binary mask in the unreliable unlabeled medical image are exchanged with all pixels in the foreground area of ​​the binary mask in the randomly selected labeled medical image, and two synthetic images are generated after the exchange, each synthetic image having a value of 1 in the foreground area of ​​the binary mask. The synthesized images are each passed through the U-Net model, and a pseudo-label probability map of the synthesized image is obtained for each synthesized image. The unreliable unlabeled medical images marked as unreliable are passed through the U-Net model, and a pseudo-label probability map is obtained for the unreliable unlabeled medical images marked as unreliable. The maximum connected area of ​​each category in the unreliable pseudo-label probability map is extracted, and a new fine segmentation pseudo-label is generated and used as the second fine segmentation pseudo-label. All pixels in the second fine segmentation pseudo-label belonging to the binary mask foreground area are exchanged with all pixels in the binary mask foreground area in the true label of the labeled medical image, and a set of synthetic labels are generated after the exchange.

[0025] As a preferred embodiment of the first aspect, a perturbation operation is performed on each unlabeled medical image to generate a weakly perturbed image and a strongly perturbed image. The specific process is as follows: for each unlabeled medical image, a weak perturbation operation is first performed to obtain a weakly perturbated image, and then two different strong perturbation operations are performed on the weakly perturbated image to generate two strongly perturbated images; wherein the weak perturbation operation is to crop, rotate and / or flip the unlabeled medical image; and the strong perturbation operation is to perform Gaussian blur, color jitter and / or spatial deformation on the weakly perturbated image.

[0026] As a preferred embodiment of the first aspect, the loss function used in fine-tuning the pre-trained U-Net model is By label loss Reliable data loss Unlabeled loss and unreliable data loss The weighted sum is obtained, and the specific form is as follows:

[0027]

[0028] λ(t)=μ·exp(-ε·(1-t / t total ) 2 )

[0029]

[0030]

[0031] Among them, α, β, γ, τ, ε are all weight hyperparameters; λ(t) represents the loss weight at the tth iteration; t total Indicates the total number of iterations; represents the first cross entropy loss; represents the first Dice loss; P l Represents the probability map generated by the U-Net model for labeled medical images; Y l Indicates the true label corresponding to the labeled medical image; represents the second cross entropy loss; represents the second Dice loss; P r Represents the probability map generated by the U-Net model for unlabeled medical images marked as reliable; represents the fine segmentation pseudo-label corresponding to the unlabeled medical image marked as reliable; τ represents the confidence threshold; P w represents the probability map generated by the U-Net model for the weakly perturbed image; P s1 ,P s2 Represents the probability map generated by the U-Net model for a strongly perturbed image; Indicates the third Dice loss; Indicates the fourth Dice loss; and represents auxiliary variables; M represents the binary mask constructed in the region swap strategy; ⊙ represents the element-wise dot product; Indicates the fifth Dice loss; represents the sixth Dice loss; ψ represents the control parameter; P dm ,P lm Both represent pseudo-label probability maps generated by the U-Net model for synthetic images; Y dm ,Y lm All represent synthetic tags.

[0032] In a second aspect, the present invention provides a medical image segmentation system based on basic model-assisted semi-supervised learning, which comprises:

[0033] An image acquisition module, used for acquiring a medical image to be segmented;

[0034] A result acquisition module, used for inputting the medical image to be segmented into a trained medical image segmentation model to obtain the segmentation result of the medical image; wherein the medical image segmentation model is obtained by fine-tuning a pre-trained U-Net model on an unlabeled data set and a labeled data set; the pre-trained U-Net model is obtained by training the U-Net model on a labeled data set in a supervised learning mode; the unlabeled data set only contains unlabeled medical images, and the labeled data set contains both labeled medical images and true labels;

[0035] During the fine-tuning process, the unlabeled medical image is input into the pre-trained U-Net model, a coarse segmentation probability map of the unlabeled medical image is generated and used as the first coarse segmentation probability map, the category with the largest probability value in the first coarse segmentation probability map is taken as the coarse segmentation pseudo-label of the unlabeled medical image and used as the first coarse segmentation pseudo-label, the prompt information required by the visual base model is generated based on the first coarse segmentation pseudo-label, the generated prompt information is passed through the prompt encoder of the visual base model to obtain the prompt feature, and the unlabeled medical image is processed by the encoder of the visual base model to obtain the image feature of the unlabeled medical image, and then the image feature and the prompt feature of the unlabeled medical image are input into the mask decoder of the visual base model, and the mask decoder generates a set of SAM foreground images, and the intersection-union ratio between the SAM foreground image and the first coarse segmentation pseudo-label of each category is calculated, and the calculation results of the intersection-union ratio are averaged to obtain the consistency score, which is sorted in descending order of the consistency score. The unlabeled medical images are classified and labeled, and the unlabeled medical images marked as reliable are processed through the joint judgment of SAM and U-Net and the foreground conflict processing strategy to generate the first fine segmentation pseudo-label. The unlabeled medical images marked as reliable and the first fine segmentation pseudo-label constitute a reliable data set, and the unlabeled medical images marked as unreliable constitute an unreliable data set. The unreliable data set is processed by the region exchange strategy, and a bidirectional displacement operation is performed between the labeled medical images and the unlabeled medical images marked as unreliable to generate synthetic images and synthetic labels as supervision in the fine-tuning process, and a perturbation operation is performed on each unlabeled medical image. For each unlabeled medical image, a weakly perturbed image and two independent strongly perturbed images are generated. The pre-trained U-Net model generates corresponding probability maps for the weakly perturbed image and each strongly perturbed image. The probability map of the weakly perturbed image is used as a supervisory signal to perform supervised learning on the probability maps of the two strongly perturbed images.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention introduces a combination of the visual basic model SAM and the U-Net model in the medical image segmentation task to form a semi-supervised learning method, which aims to solve the problems of strong dependence on high-quality labeled data, low quality of pseudo-labels and poor adaptability to data perturbations in the current technology, and overcomes the limitations of SAM when used alone in processing complex medical images. Compared with the traditional semi-supervised learning method, the present invention introduces the SAM model, utilizes its excellent generalization ability in semantic segmentation tasks, and the professionalism of the U-Net model in medical image processing, to achieve effective utilization of unlabeled data. Through the comprehensive application of consistency scoring, foreground conflict processing strategy and region exchange strategy, the present invention can improve the quality of pseudo-labels and effectively process data with different reliability, providing higher accuracy and robustness for medical image segmentation tasks. This method significantly improves the segmentation performance of the medical image segmentation model under limited labeled data, reduces the dependence on manual annotation, and provides a novel and effective solution for the application of semi-supervised learning in the field of medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of the steps of the present invention;

[0039] Figure 2 It is a schematic diagram of the foreground conflict handling strategy of the present invention;

[0040] Figure 3 It is a schematic diagram of the regional exchange strategy of the present invention;

[0041] Figure 4 It is a schematic diagram of the specific process of fine-tuning the medical image segmentation model of the present invention;

[0042] Figure 5 A training and testing flow chart of the medical image segmentation model SESAME in an embodiment of the present invention;

[0043] Figure 6 It is a schematic diagram of the test results of an embodiment of the present invention;

[0044] Figure 7 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0046] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0047] In the field of medical image segmentation, although deep learning technology has made significant progress, traditional segmentation methods often rely on a large amount of labeled data, and the acquisition of these labeled data is not only expensive but also time-consuming. Especially in three-dimensional medical imaging (such as CT and MRI), the workload of labeling is extremely large. In addition, the performance of existing semi-supervised learning methods is still difficult to match that of fully supervised learning methods when faced with a small amount of labeled data. The main reasons are the low quality of pseudo-labels and the poor adaptability of the model to perturbations. In view of the above problems, the present invention proposes a medical image segmentation method based on semi-supervised learning assisted by a basic model. The method combines the powerful generalization ability of Segment Anything Model (SAM) and the domain-specific knowledge of the U-Net model, and optimizes the segmentation quality of unlabeled images by using the foreground mask generated by SAM as prior information. Specifically, the present invention designs a foreground conflict processing strategy to handle the category conflict in the foreground mask generated by SAM, ensuring the consistency of the segmentation results. In addition, a region exchange strategy is introduced to effectively alleviate the problem of inconsistent data distribution by exchanging pixel regions between unreliable images and labeled images. This method not only significantly improves the accuracy and robustness of medical image segmentation, but also can make full use of the information of unlabeled data when the labeled data is limited, laying a solid foundation for its wide application in medical diagnosis and treatment planning in the future.

[0048] Therefore, if Figure 1 As shown, the medical image segmentation method based on basic model assisted semi-supervised learning provided by the present invention specifically includes the following steps:

[0049] The specific process of the medical image segmentation method is as follows: inputting the medical image to be segmented into a trained medical image segmentation model to obtain a segmentation result of the medical image; wherein the medical image segmentation model is obtained by fine-tuning a pre-trained U-Net model on an unlabeled data set and a labeled data set; the pre-trained U-Net model is obtained by training the U-Net model on a labeled data set in a supervised learning mode; the unlabeled data set only contains unlabeled medical images, and the labeled data set contains both labeled medical images and true labels;

[0050] During the fine-tuning process, the unlabeled medical image is input into the pre-trained U-Net model, a coarse segmentation probability map of the unlabeled medical image is generated and used as the first coarse segmentation probability map, the category with the largest probability value in the first coarse segmentation probability map is taken as the coarse segmentation pseudo-label of the unlabeled medical image and used as the first coarse segmentation pseudo-label, the prompt information required by the visual base model is generated based on the first coarse segmentation pseudo-label, the generated prompt information is passed through the prompt encoder of the visual base model to obtain the prompt feature, and the unlabeled medical image is processed by the encoder of the visual base model to obtain the image feature of the unlabeled medical image, and then the image feature and the prompt feature of the unlabeled medical image are input into the mask decoder of the visual base model, and the mask decoder generates a set of SAM foreground images, and the intersection-union ratio between the SAM foreground image and the first coarse segmentation pseudo-label of each category is calculated, and the calculation results of the intersection-union ratio are averaged to obtain the consistency score, which is sorted in descending order of the consistency score. The unlabeled medical images are classified and labeled, and the unlabeled medical images marked as reliable are processed through the joint judgment of SAM and U-Net and the foreground conflict processing strategy to generate the first fine segmentation pseudo-label. The unlabeled medical images marked as reliable and the first fine segmentation pseudo-label constitute a reliable data set, and the unlabeled medical images marked as unreliable constitute an unreliable data set. The unreliable data set is processed by the region exchange strategy, and a bidirectional displacement operation is performed between the labeled medical images and the unlabeled medical images marked as unreliable to generate synthetic images and synthetic labels as supervision in the fine-tuning process, and a perturbation operation is performed on each unlabeled medical image. For each unlabeled medical image, a weakly perturbed image and two independent strongly perturbed images are generated. The pre-trained U-Net model generates corresponding probability maps for the weakly perturbed image and each strongly perturbed image. The probability map of the weakly perturbed image is used as a supervisory signal to perform supervised learning on the probability maps of the two strongly perturbed images.

[0051] It should be noted that before the above medical image segmentation model is actually applied to medical image segmentation, the U-Net model is first pre-trained using labeled medical images. During the pre-training process, a combination of cross entropy loss and Dice loss is used as the loss function to ensure that the medical image segmentation model can achieve high-precision segmentation when processing medical images. This combination of loss functions helps to improve the sensitivity of the medical image segmentation model to the boundaries of different categories and enhance the accuracy and robustness of the segmentation results.

[0052] In a preferred embodiment of the present invention, a medical image segmentation model (Semi-supervised learning method assisted by SAM for medical imagesegmentation, SESAME) is designed based on the above-mentioned visual basic model, thereby providing a medical image segmentation method based on basic model assisted semi-supervised learning. In the present invention, thanks to the powerful performance of the visual basic model SAM and its versatility in medical image segmentation, the medical image segmentation model adopts a basic model-assisted U-Net and SAM joint judgment structure, and uses the trained U-Net model as the medical image segmentation model. Its network structure is symmetrical, divided into a downsampling path (encoder) and an upsampling path (decoder), and the specific structure is as follows:

[0053] In the downsampling path, the input medical image is first processed by the first convolution layer to generate a first feature map, the first feature map is downsampled by the maximum pooling layer to obtain a low-resolution first feature map, then the low-resolution first feature map is processed by the second convolution layer in turn to generate a second feature map, the second feature map is downsampled again by the maximum pooling layer to obtain a low-resolution second feature map, the low-resolution second feature map is processed by the third convolution layer to generate a third feature map, the third feature map is subjected to the maximum pooling operation to generate a low-resolution third feature map, the low-resolution third feature map is processed by the fourth convolution layer to generate a fourth feature map, the fourth feature map is subjected to the maximum pooling operation to obtain a low-resolution fourth feature map, finally, the low-resolution fourth feature map is processed by the fifth convolution layer to generate the fifth feature map and used as the input of the upsampling path;

[0054] In the upsampling path, the fifth feature map is upsampled and then spliced ​​with the fourth feature map in the downsampling path. After splicing, it is processed by the sixth convolution layer to generate the sixth feature map. After upsampling, the sixth feature map is spliced ​​with the third feature map in the downsampling path. After splicing, it is processed by the seventh convolution layer to generate the seventh feature map. After upsampling, the seventh feature map is spliced ​​with the second feature map in the downsampling path. After splicing, it is processed by the eighth convolution layer to generate the eighth feature map. After upsampling, the eighth feature map is spliced ​​with the first feature map in the downsampling path. After splicing, it is processed by the ninth convolution layer to generate the ninth feature map. Finally, the ninth feature map is processed by an output convolution layer, and the output channel number corresponds to the target classification number. The final segmentation result.

[0055] Furthermore, in an embodiment of the present invention, the convolution kernel size of the U-Net model is configured as follows: in the downsampling path, all convolution layers use a convolution kernel size of 3×3 for gradually extracting features, and use a maximum pooling layer of 2×2 to reduce the resolution of the feature map and generate a deeper feature map; in the upsampling path, the convolution layers passed after splicing all use a convolution kernel size of 3×3 for further processing the spliced ​​feature map after each upsampling operation, thereby restoring the resolution of the image and extracting richer feature information; upsampling is achieved by using a transposed convolution with a convolution kernel size of 2×2 to increase the resolution of the feature map; the convolution kernel size of the output convolution layer is 1×1, which is used to convert the last generated feature map into the number of channels corresponding to the number of target categories to generate the final segmentation result.

[0056] In the U-Net model of the embodiment of the present invention, considering that the input of the medical image segmentation model SESAME can be both an unlabeled medical image and a labeled medical image, the two are not distinguished in describing the specific structure of the medical image segmentation model, and are uniformly represented by "medical image". The specific processing flow of the U-Net model is as follows: In the downsampling path of the U-Net model, the input medical image of size 256×256 is first processed by a first convolution layer of size 3×3 to generate a first feature map of size 256×256×64. Then, the first feature map is downsampled through a 2×2 maximum pooling layer to obtain a low-resolution first feature map of size 128×128×64. Subsequently, the low-resolution first feature map is processed by a second convolution layer of size 3×3 in turn to generate a second feature map, and is downsampled to 64×64×128 through the maximum pooling layer to obtain a low-resolution second feature map. Then, the low-resolution second feature map is processed by a third convolutional layer of size 3×3 to generate a third feature map, and is downsampled to 32×32×256 again through a maximum pooling operation to obtain a low-resolution third feature map. Next, the low-resolution third feature map is processed by a fourth convolutional layer of size 3×3 to generate a fourth feature map, and is downsampled to 16×16×512 through a maximum pooling operation to obtain a low-resolution fourth feature map. Finally, the low-resolution fourth feature map is processed by a fifth convolutional layer of size 3×3 to generate a fifth feature map of size 8×8×1024, which is used as the input of the upsampling path. In the upsampling path, the fifth feature map is first upsampled to restore its resolution to 16×16×512, and is concatenated with the fourth feature map in the downsampling path to form a feature map containing more contextual information. The feature map is then processed by a sixth convolutional layer of size 3×3 to generate the sixth feature map. Next, the sixth feature map is upsampled again to 32×32×256 and concatenated with the third feature map in the downsampling path, and then processed by the seventh convolution layer of size 3×3 to generate the seventh feature map. Next, the seventh feature map is upsampled to 64×64×128 and concatenated with the second feature map in the downsampling path, and processed by the eighth convolution layer of size 3×3 to generate the eighth feature map. Finally, the eighth feature map is upsampled to the original image size of 256×256×64, and concatenated with the first feature map in the downsampling path, and processed by the ninth convolution layer of size 3×3 to generate the ninth feature map. Finally, the ninth feature map is processed by an output convolution layer, and the final segmentation result with the number of output channels corresponding to the number of target categories is obtained, ensuring a high-precision segmentation effect.

[0057] The specific fine-tuning process of the medical image segmentation model of the present invention is described in detail below. Figure 4As shown in the figure, the whole process is mainly divided into three steps: coarse segmentation pseudo-label generation, joint judgment and improved semi-supervised perturbation.

[0058] In the first step, the U-Net model is first pre-trained on a labeled dataset, and the pre-trained U-Net model generates coarse segmentation pseudo-labels based on the unlabeled medical images, and uses these pseudo-labels to generate different categories of prompt information, including sparse prompts (such as boxes or points) and dense prompts (such as masks). These coarse segmentation pseudo-labels are then input into the visual basic model (Segment Anything Model, SAM) together with the unlabeled medical images, which processes the unlabeled dataset through an encoder, a prompt encoder, and a mask decoder to generate a set of SAM foreground maps. Specifically, the encoder of SAM generates deep features (image features) of the unlabeled medical images, and the generated prompt information is processed by the SAM prompt encoder to generate prompt features that match the image features. The prompt features are input into the mask decoder of SAM together with the image features to generate SAM foreground maps.

[0059] It should be noted that the specific form of the visual base model SAM used here is not limited, as long as it has general medical knowledge and includes three main components: an encoder, a prompt encoder, and a mask decoder, it can be combined with the SESAME model of the present invention in the segmentation task.

[0060] In the second step, in order to evaluate the reliability of unlabeled medical images, the present invention designs a scoring method based on the image-level consistency between the coarse segmentation pseudo-labels generated by U-Net and the foreground image with hint pseudo-labels generated by SAM, that is, the coarse segmentation pseudo-labels and the generated SAM foreground images are jointly judged, and the consistency score of the unlabeled medical images is calculated by calculating the intersection over union (IoU). Specifically, the intersection over union (IOU) between the SAM foreground image and the first coarse segmentation pseudo-label of each category is calculated, and the calculation results of the intersection over union are averaged to obtain the consistency score. Then, the unlabeled medical images are classified according to reliability into three categories: "reliable", "neutral" and "unreliable", and fine segmentation pseudo-labels are generated for the unlabeled medical images marked as reliable (i.e., reliable images) according to the foreground conflict processing strategy. Finally, the unreliable data set is processed by the region exchange strategy, and a bidirectional displacement operation is performed between the labeled image and the unreliable image to generate a new synthetic image and a synthetic label, which significantly improves the accuracy and robustness of medical image segmentation.

[0061] In the third step, data augmentation is performed on all unlabeled medical images to generate three different versions of images (i.e., a weakly perturbed image and two independent strongly perturbed images) to improve the segmentation effect. Specifically, for each unlabeled medical image, a weakly perturbed image is first generated through a weak perturbation operation, and then the weakly perturbed image is subjected to two independent strong perturbation operations to generate two different strongly perturbed images, each of which is supervised by the prediction results of the weakly perturbed image. Among them, the weak perturbation operation combines operations such as cropping, rotation, and flipping; the strong perturbation operation includes Gaussian blur, color jitter, and spatial deformation (CutMix).

[0062] Through the above three steps, the present invention can significantly improve the accuracy and robustness of medical image segmentation, and effectively utilize unlabeled data to improve the performance of the medical image segmentation model SESAME, effectively enhancing the segmentation performance of the medical image segmentation model for unlabeled medical images, so that the medical image segmentation model can be more robust and accurate when processing enhanced medical images.

[0063] The principles of generating prompt information, classifying and marking unlabeled medical images, foreground conflict processing strategy and region exchange strategy in the present invention and the specific processing flow of each strategy are described in detail below.

[0064] It should be noted that, in the present invention, the specific generation process of the prompt information required by the visual base model is as follows: in the first coarse segmentation pseudo-label, only the largest continuous area in the non-empty foreground mask of each category except the background is always retained, and after obtaining the continuous area, the prompt information is preset, and the prompt information is a sparse prompt and / or a dense prompt, the sparse prompt is a point and / or a border, and the dense prompt is a category-level foreground mask captured from the first coarse segmentation pseudo-label;

[0065] If the sparse hint is a border, the minimum circumscribed rectangular area of ​​the continuous area is used as the border representing the sparse hint; if the sparse hint is a point, it is generated in an iterative manner. In each iteration, a point is sampled from the unlabeled medical image until the iteration ends, and all the sampled points are used as a set of points representing the sparse hint.

[0066] In this embodiment, in order to generate the prompt information required by the visual basic model (Segment Anything Model, SAM), this embodiment performs the following operations: First, to avoid possible noise, only the largest continuous area in the non-empty foreground mask of each category except the background is always retained in the coarse segmentation pseudo label After the continuous region is obtained, prompt information is preset, and the prompt information is a sparse prompt and / or a dense prompt. The sparse prompt includes a point and a border, and the dense prompt is a mask.

[0067] If the sparse hint is a border, the contiguous region The minimum bounding rectangle area is used as the border representing the sparse hint;

[0068] If the sparse hint is a point, it is generated in an iterative manner. In each iteration, a point is sampled from the unlabeled medical image until the iteration ends, and all the sampled points are used as a set of points representing the sparse hint; further, if the sampled point in each iteration satisfies the requirement of being inside the coarse segmentation pseudo-label, it is regarded as a positive point, otherwise it is regarded as a negative point. In addition to the sparse hint, the present invention also captures the category-level foreground mask from the coarse segmentation pseudo-label as a mask representing the dense hint.

[0069] It should be noted that, in the present invention, the specific process of classifying and labeling unlabeled medical images in descending order of consistency scores is as follows: the w unlabeled medical images with the lowest consistency scores are taken as bottom medical images, the unlabeled medical images whose consistency scores are not lower than a preset threshold are marked as reliable, the unlabeled medical images whose consistency scores are lower than the preset threshold and belong to bottom medical images are marked as unreliable, and the remaining unlabeled medical images are marked as neutral and no additional processing is performed.

[0070] In this embodiment, the value of w is half of the total number of unlabeled medical images. That is, among all unlabeled medical images, 50% of the unlabeled medical images at the bottom (i.e., with the lowest consistency score) are regarded as bottom medical images. For example, if there are 200 unlabeled medical images after sorting, the 100 unlabeled medical images with lower consistency scores at the back are regarded as bottom medical images. Then, the unlabeled medical images with a consistency score not less than 0.85 are marked as reliable, the unlabeled medical images with a consistency score less than 0.85 and belonging to the bottom medical images are marked as unreliable, and the remaining unlabeled medical images are marked as neutral and no additional processing is performed. The number and quality of high-confidence pseudo-labels with prompts are determined by setting this threshold and selecting the 50% sorting ratio. Of course, the threshold value and the value of w are both a method of implementing image segmentation with better effect given in this embodiment, and those skilled in the art can also select appropriate values ​​according to the actual image segmentation results.

[0071] It should be noted that, in the present invention, the foreground conflict processing strategy is used to generate a fine segmentation pseudo-label that improves segmentation accuracy. The specific process of processing the unlabeled medical image marked as reliable by the foreground conflict processing strategy to generate the first fine segmentation pseudo-label is as follows: the all-zero matrix is ​​used as the initial state of the first fine segmentation pseudo-label. When a pixel in the unlabeled medical image marked as reliable is only marked as foreground by a category mask k of the SAM foreground map, the pixel is assigned to the category in the category mask, and the value of the point with the same position as the pixel in the all-zero matrix is ​​changed to k. When a pixel in the unlabeled medical image marked as reliable is simultaneously marked as foreground by multiple category masks of the SAM foreground map, the pixel is assigned to the corresponding category k in the first coarse segmentation pseudo-label, and the value of the point with the same position as the pixel in the all-zero matrix is ​​changed to k, until all pixels in the unlabeled medical image marked as reliable are traversed, and a complete first fine segmentation pseudo-label is obtained.

[0072] Furthermore, in the process of generating the fine segmentation pseudo-labels of the unlabeled medical images marked as reliable, the foreground conflict processing strategy is adopted in this embodiment to deal with the category prediction conflict that may be caused by the SAM foreground map output by the visual base model. Specifically, when a pixel in the reliable image is marked as foreground by only one category mask of SAM, the pixel is assigned to the category. If the same pixel is marked as foreground by multiple category masks at the same time, the pixel is assigned to the corresponding category in the coarse segmentation pseudo-label generated by U-Net. Through this strategy, corresponding fine segmentation pseudo-labels are generated for each reliable image, and the reliable data set composed of these fine segmentation pseudo-labels and the reliable image is treated equally with the labeled data set in the subsequent steps, thereby improving the accuracy and robustness of the segmentation results. Figure 2 The processing flow of the strategy is demonstrated, starting from the SAM foreground map generated by SAM, through the generation of coarse segmentation pseudo-labels and the processing of foreground conflict handling strategy, and finally forming fine segmentation pseudo-labels, providing more accurate supervision information for the medical image segmentation model.

[0073] It should be noted that, in the present invention, the region exchange strategy is used to process the unreliable data set, and the specific process of generating the synthetic image and the synthetic label is as follows: for the unreliable unlabeled medical image marked as unreliable, a labeled medical image is randomly selected, and a binary mask with a value of 1 in the foreground area and 0 in the rest is pre-constructed, the size of the binary mask foreground area is smaller than the size of the unreliable unlabeled medical image marked as unreliable, all the pixels in the foreground area of ​​the binary mask in the unreliable unlabeled medical image marked as unreliable are exchanged with all the pixels in the foreground area of ​​the binary mask in the randomly selected labeled medical image, and two synthetic images are generated after the exchange, each synthetic image having a value of 1 in the foreground area of ​​the binary mask. The synthesized images are each passed through the U-Net model, and a pseudo-label probability map of the synthesized image is obtained for each synthesized image. The unreliable unlabeled medical images marked as unreliable are passed through the U-Net model, and a pseudo-label probability map is obtained for the unreliable unlabeled medical images marked as unreliable. The maximum connected area of ​​each category in the unreliable pseudo-label probability map is extracted, and a new fine segmentation pseudo-label is generated and used as the second fine segmentation pseudo-label. All pixels in the second fine segmentation pseudo-label belonging to the binary mask foreground area are exchanged with all pixels in the binary mask foreground area in the true label of the labeled medical image, and a set of synthetic labels are generated after the exchange.

[0074] In this embodiment, a region exchange strategy is used to alleviate the problem of inconsistent data distribution between unlabeled medical images marked as unreliable and labeled medical images. Figure 3 As shown, for an unlabeled medical image X that is marked as unreliable d , randomly select a labeled medical image X l , pre-construct a binary mask with a value of 1 in the foreground area and 0 in the rest, the size of the binary mask foreground area is smaller than the size of the unlabeled medical image marked as unreliable, and exchange all pixels in the foreground area of ​​the binary mask in the unreliable unlabeled medical image with all pixels in the foreground area of ​​the binary mask in the randomly selected labeled medical image, and generate two synthetic images after the exchange, where the labeled medical image X l Corresponding synthetic image X dm , unlabeled medical image X is marked as unreliable d Corresponding synthetic image X lm Each synthetic image is passed through the U-Net model, and each synthetic image corresponds to a pseudo-label probability map of the synthetic image, where the synthetic image X dm Corresponding pseudo label probability map P dm , synthetic image X lm Corresponding pseudo label probability map P lmThe unlabeled medical images marked as unreliable are passed through the U-Net model, and a pseudo-label probability map is obtained for the unreliable unlabeled medical images and used as the unreliable pseudo-label probability map. The maximum connected area of ​​each category in the unreliable pseudo-label probability map is extracted, and a new fine segmentation pseudo-label is generated and used as the second fine segmentation pseudo-label Compare all pixels in the foreground area of ​​the binary mask in the second fine segmentation pseudo-label with the true label Y of the labeled medical image l All pixels in the binary mask foreground area are swapped, and a set of synthetic labels are generated after the swap to guide the U-Net model to predict the synthetic image, where the synthetic label Y dm Corresponding to the true label Y l , synthetic label Y lm Corresponding to the second fine segmentation pseudo label This strategy effectively improves the segmentation accuracy of the medical image segmentation model when processing unreliable images, and enables unreliable images to be fully utilized in training.

[0075] It should be noted that, in this embodiment, the height of the binary mask foreground area is between 2% and 40% of the height of the unlabeled medical image marked as unreliable (i.e., the unreliable image), and its width is between 2% and 40% of the width of the unreliable image. This selection method is only a preferred method with better segmentation accuracy. The binary mask foreground area of ​​appropriate size can also be selected according to the actual size of the unreliable image, so it is not limited in the present invention.

[0076] It should be noted that in the present invention, the loss function used in fine-tuning the pre-trained U-Net model is By label loss Reliable data loss Unlabeled loss and unreliable data loss The weighted sum is obtained, and the specific form is as follows:

[0077]

[0078] λ(t)=μ·exp(-ε·(1-t / t total ) 2 )

[0079] Among them, α, β, γ, τ, ε are all weight hyperparameters; λ(t) represents the loss weight at the tth iteration; t total Indicates the total number of iterations. In this embodiment, α=β=γ=1, μ=0.1, ε=5.

[0080] Specifically, the labeled loss is for the prediction of U-Net on the labeled dataset, which combines the first cross entropy loss and the first Dice loss

[0081]

[0082] Among them, P l Represents the probability map generated by the U-Net model for labeled medical images; Y l Represents the true label corresponding to the labeled medical image.

[0083] The reliable data loss is for U-Net's predictions on reliable datasets, and uses the same loss combination as the labeled loss:

[0084]

[0085] in, represents the second cross entropy loss; represents the second Dice loss; P r Represents the probability map generated by the U-Net model for unlabeled medical images marked as reliable; Represents the fine segmentation pseudo labels corresponding to the unlabeled medical images that are labeled as reliable.

[0086] The unlabeled loss is based on the high-confidence prediction of the weakly perturbed image, calculates the consistency between the strongly perturbed image and the prediction, and uses the Dice loss to measure it. At the same time, the pixels with less noise are filtered by setting the confidence threshold v:

[0087]

[0088] Among them, P w represents the probability map generated by the U-Net model for the weakly perturbed image; P s1 ,P s2 Represents the probability map generated by the U-Net model for a strongly perturbed image; Indicates the third Dice loss; Indicates the fourth Dice loss.

[0089] In this embodiment, the confidence threshold τ is set to 0.5.

[0090] Unreliable data loss It is the region exchange strategy loss, which calculates the Dice loss between the prediction of the synthetic image generated after region exchange and its corresponding supervisory signal, and adjusts the influence of unreliable images in training through a control parameter ψ:

[0091]

[0092] in, and Represents auxiliary variables; represents the fifth Dice loss; M represents the binary mask constructed in the region swap strategy; ⊙ represents the element-wise dot product; represents the sixth Dice loss; P dm ,P lm Both represent pseudo-label probability maps generated by the U-Net model for synthetic images; Y dm ,Y lm All represent synthetic tags.

[0093] In this embodiment, the control parameter ψ is set to 0.5.

[0094] It should be noted that in the present invention, the medical image may be magnetic resonance imaging (MRI) or CT imaging. In this embodiment, the medical image is derived from the Prostate MR lmage Segmentation dataset, the Automatic CardiacDiagnosis Challenge dataset, and the LiTS2017 dataset. Among them, the Prostate MR lmageSegmentation dataset is used to obtain magnetic resonance imaging of the prostate area;

[0095] The Automatic Cardiac Diagnosis Challenge dataset is used to obtain dynamic magnetic resonance imaging of the heart; the LiTS2017 dataset is used to obtain CT imaging of the liver.

[0096] The present invention will now use a specific example to demonstrate the application effect of the medical image segmentation method based on basic model-assisted semi-supervised learning described in the above embodiment on a specific data set, so as to facilitate understanding of the essence of the present invention.

[0097] Example

[0098] The specific implementation process of the medical image segmentation method and system based on basic model assisted semi-supervised learning adopted in this embodiment is as described above and will not be repeated here.

[0099] The overall process in this embodiment can be divided into three stages: data preprocessing, model training, and image prediction. Figure 5 shown.

[0100] 1. Data preprocessing stage

[0101] Step 1: Preprocess the original medical image by cutting and flipping it, then perform data enhancement and process it into images of the same size (256×256).

[0102] Step 2: Label the preprocessed medical images and use tools such as Labelme to perform vector wireframe annotation to generate GroundTruth for subsequent medical image segmentation model training.

[0103] 2. Model training

[0104] Step 1: Build a training data set and divide the training data set into batches according to a fixed batch size, with a total of N.

[0105] Step 2, sequentially select a batch of training samples with index i from the training data set, where i∈{0,1,…,N}. Use each batch of training samples to train the above-mentioned medical image segmentation model SESAME. The specific structure of SESAME is as described above and will not be repeated here. It adopts an encoder-decoder structure. During the training process, the label loss, reliable data loss, unlabel loss and unreliable data loss of each training sample are calculated, and the network parameters in the entire medical image segmentation model are adjusted according to the total loss of all training samples in the batch until all batches of the training data set participate in the model training. After reaching the specified number of iterations, the model converges and the training is completed.

[0106] 3. Medical Image Semantic Segmentation

[0107] The images in the test set are directly used as input to the trained medical image segmentation model SESAME, and the probability of each pixel category is finally predicted. The category with the highest probability is selected as the final output through activation functions such as Sigmoid, thereby achieving medical image segmentation.

[0108] In this embodiment, the test results are as follows: Figure 6 shown. Figure 6 In the figure, the first sub-image from left to right is the segmentation result of magnetic resonance imaging of the human heart, the second sub-image is the segmentation result of magnetic resonance imaging of the prostate, and the third sub-image is the segmentation result of CT imaging of the liver. It can be seen that the medical image segmentation model SESAME, which introduces the basic model SAM and combines it with U-Net, can effectively improve the accuracy of semi-supervised medical image segmentation. By utilizing the generalization ability of SAM and the expertise of U-Net, the present invention achieves more accurate semantic segmentation of medical images, especially in maintaining structural details and improving classification accuracy. Compared with traditional methods, the present invention shows better performance in processing complex medical images, and provides a promising solution for improving the accuracy and robustness of medical image segmentation. It provides strong technical support for possible future applications.

[0109] It should also be noted that the medical image segmentation method based on basic model assisted semi-supervised learning in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a medical image segmentation system based on basic model assisted semi-supervised learning corresponding to the medical image segmentation method based on basic model assisted semi-supervised learning provided in the above embodiment, such as Figure 7 As shown, it includes:

[0110] An image acquisition module, used for acquiring a medical image to be segmented;

[0111] A result acquisition module, used for inputting the medical image to be segmented into a trained medical image segmentation model to obtain the segmentation result of the medical image; wherein the medical image segmentation model is obtained by fine-tuning a pre-trained U-Net model on an unlabeled data set and a labeled data set; the pre-trained U-Net model is obtained by training the U-Net model on a labeled data set in a supervised learning mode; the unlabeled data set only contains unlabeled medical images, and the labeled data set contains both labeled medical images and true labels;

[0112] During the fine-tuning process, the unlabeled medical image is input into the pre-trained U-Net model, a coarse segmentation probability map of the unlabeled medical image is generated and used as the first coarse segmentation probability map, the category with the largest probability value in the first coarse segmentation probability map is taken as the coarse segmentation pseudo-label of the unlabeled medical image and used as the first coarse segmentation pseudo-label, the prompt information required by the visual base model is generated based on the first coarse segmentation pseudo-label, the generated prompt information is passed through the prompt encoder of the visual base model to obtain the prompt feature, and the unlabeled medical image is processed by the encoder of the visual base model to obtain the image feature of the unlabeled medical image, and then the image feature and the prompt feature of the unlabeled medical image are input into the mask decoder of the visual base model, and the mask decoder generates a set of SAM foreground images, and the intersection-union ratio between the SAM foreground image and the first coarse segmentation pseudo-label of each category is calculated, and the calculation results of the intersection-union ratio are averaged to obtain the consistency score, which is sorted in descending order of the consistency score. The unlabeled medical images are classified and labeled, and the unlabeled medical images marked as reliable are processed through the joint judgment of SAM and U-Net and the foreground conflict processing strategy to generate the first fine segmentation pseudo-label. The unlabeled medical images marked as reliable and the first fine segmentation pseudo-label constitute a reliable data set, and the unlabeled medical images marked as unreliable constitute an unreliable data set. The unreliable data set is processed by the region exchange strategy, and a bidirectional displacement operation is performed between the labeled medical images and the unlabeled medical images marked as unreliable to generate synthetic images and synthetic labels as supervision in the fine-tuning process, and a perturbation operation is performed on each unlabeled medical image. For each unlabeled medical image, a weakly perturbed image and two independent strongly perturbed images are generated. The pre-trained U-Net model generates corresponding probability maps for the weakly perturbed image and each strongly perturbed image. The probability map of the weakly perturbed image is used as a supervisory signal to perform supervised learning on the probability maps of the two strongly perturbed images.

[0113] It should be noted that the image acquisition module acquires the medical image to be segmented, and the result acquisition module acquires a trained medical image segmentation model, and both can be acquired online or offline. For online acquisition, the image acquisition module can acquire the medical image of the target object online through an imaging device, and the result acquisition module can train the medical image segmentation model through online training; for offline acquisition, the image acquisition module can acquire the medical image that has been acquired and stored by the imaging device, and the result acquisition module can read the medical image segmentation model stored on the storage medium after training.

[0114] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A medical image segmentation method based on basic model-assisted semi-supervised learning, characterized in that: The specific process of the medical image segmentation method is as follows: inputting the medical image to be segmented into a trained medical image segmentation model to obtain a segmentation result of the medical image; wherein the medical image segmentation model is obtained by fine-tuning a pre-trained U-Net model on an unlabeled data set and a labeled data set; the pre-trained U-Net model is obtained by training the U-Net model on a labeled data set in a supervised learning mode; the unlabeled data set only contains unlabeled medical images, and the labeled data set contains both labeled medical images and true labels; During the fine-tuning process, the unlabeled medical image is input into the pre-trained U-Net model, a coarse segmentation probability map of the unlabeled medical image is generated and used as the first coarse segmentation probability map, the category with the largest probability value in the first coarse segmentation probability map is taken as the coarse segmentation pseudo-label of the unlabeled medical image and used as the first coarse segmentation pseudo-label, the prompt information required by the visual base model is generated based on the first coarse segmentation pseudo-label, the generated prompt information is passed through the prompt encoder of the visual base model to obtain the prompt feature, and the unlabeled medical image is processed by the encoder of the visual base model to obtain the image feature of the unlabeled medical image, and then the image feature and the prompt feature of the unlabeled medical image are input into the mask decoder of the visual base model, and the mask decoder generates a set of SAM foreground images, and the intersection-union ratio between the SAM foreground image and the first coarse segmentation pseudo-label of each category is calculated, and the calculation results of the intersection-union ratio are averaged to obtain the consistency score, which is sorted in descending order of the consistency score. The unlabeled medical images are classified and labeled, and the unlabeled medical images marked as reliable are processed through the joint judgment of SAM and U-Net and the foreground conflict processing strategy to generate the first fine segmentation pseudo-label. The unlabeled medical images marked as reliable and the first fine segmentation pseudo-label constitute a reliable data set, and the unlabeled medical images marked as unreliable constitute an unreliable data set. The unreliable data set is processed by the region exchange strategy, and a bidirectional displacement operation is performed between the labeled medical images and the unlabeled medical images marked as unreliable to generate synthetic images and synthetic labels as supervision in the fine-tuning process, and a perturbation operation is performed on each unlabeled medical image. For each unlabeled medical image, a weakly perturbed image and two independent strongly perturbed images are generated. The pre-trained U-Net model generates corresponding probability maps for the weakly perturbed image and each strongly perturbed image. The probability map of the weakly perturbed image is used as a supervisory signal to perform supervised learning on the probability maps of the two strongly perturbed images.

2. The medical image segmentation method based on basic model-assisted semi-supervised learning as claimed in claim 1, characterized in that: The network structure of the U-Net model is symmetrical and is divided into a downsampling path and an upsampling path; In the downsampling path, the input medical image is first processed by the first convolution layer to generate a first feature map, the first feature map is downsampled by the maximum pooling layer to obtain a low-resolution first feature map, then the low-resolution first feature map is processed by the second convolution layer in turn to generate a second feature map, the second feature map is downsampled again by the maximum pooling layer to obtain a low-resolution second feature map, the low-resolution second feature map is processed by the third convolution layer to generate a third feature map, the third feature map is subjected to the maximum pooling operation to generate a low-resolution third feature map, the low-resolution third feature map is processed by the fourth convolution layer to generate a fourth feature map, the fourth feature map is subjected to the maximum pooling operation to obtain a low-resolution fourth feature map, finally, the low-resolution fourth feature map is processed by the fifth convolution layer to generate the fifth feature map and used as the input of the upsampling path; In the upsampling path, the fifth feature map is upsampled and then spliced ​​with the fourth feature map in the downsampling path. After splicing, it is processed by the sixth convolution layer to generate the sixth feature map. After upsampling, the sixth feature map is spliced ​​with the third feature map in the downsampling path. After splicing, it is processed by the seventh convolution layer to generate the seventh feature map. After upsampling, the seventh feature map is spliced ​​with the second feature map in the downsampling path. After splicing, it is processed by the eighth convolution layer to generate the eighth feature map. After upsampling, the eighth feature map is spliced ​​with the first feature map in the downsampling path. After splicing, it is processed by the ninth convolution layer to generate the ninth feature map. Finally, the ninth feature map is processed by an output convolution layer, and the output channel number corresponds to the target classification number. The final segmentation result.

3. A medical image segmentation method based on semi-supervised learning assisted by a basic model as claimed in claim 2, characterized in that: The convolution kernel size of the U-Net model is configured as follows: In the downsampling path, all convolutional layers use a 3×3 convolution kernel size to gradually extract features, and a 2×2 maximum pooling layer is used to reduce the resolution of the feature map and generate a deeper feature map; In the upsampling path, the convolutional layers after splicing all use a convolution kernel size of 3×3, which is used to further process the spliced ​​feature maps after each upsampling operation, thereby restoring the resolution of the medical image and extracting richer feature information; Upsampling is achieved by using transposed convolution with a convolution kernel size of 2×2 to increase the resolution of the feature map. The convolution kernel size of the output convolution layer is 1×1, which is used to convert the last generated feature map into the number of channels corresponding to the number of target categories to generate the final segmentation result.

4. The medical image segmentation method based on basic model-assisted semi-supervised learning according to claim 1, characterized in that: The specific generation process of the prompt information required by the visual base model is as follows: in the first coarse segmentation pseudo-label, only the largest continuous area in the non-empty foreground mask of each category except the background is always retained, and after obtaining the continuous area, the prompt information is preset, and the prompt information is a sparse prompt and / or a dense prompt, the sparse prompt is a point and / or a border, and the dense prompt is a category-level foreground mask captured from the first coarse segmentation pseudo-label; If the sparse cue is a border, the minimum bounding rectangle of the continuous region is used as the border representing the sparse cue; If the sparse hint is a point, it is generated in an iterative manner. In each iteration, a point is sampled from the unlabeled medical image until the iteration ends, and all the sampled points are used as a set of points representing the sparse hint.

5. The medical image segmentation method based on basic model-assisted semi-supervised learning as claimed in claim 1, characterized in that: The specific process of classifying and labeling the unlabeled medical images in descending order of consistency scores is as follows: the w unlabeled medical images with the lowest consistency scores are taken as bottom medical images, the unlabeled medical images whose consistency scores are not lower than the preset threshold are marked as reliable, the unlabeled medical images whose consistency scores are lower than the preset threshold and belong to bottom medical images are marked as unreliable, and the remaining unlabeled medical images are marked as neutral and no additional processing is performed.

6. The medical image segmentation method based on basic model assisted semi-supervised learning as claimed in claim 1, characterized in that: The unlabeled medical image marked as reliable is processed by the foreground conflict processing strategy to generate the specific process of the first fine segmentation pseudo-label: the all-zero matrix is ​​used as the initial state of the first fine segmentation pseudo-label. When a pixel in the unlabeled medical image marked as reliable is marked as foreground by only one category mask k of the SAM foreground map, the pixel is assigned to the category in the category mask, and the value of the point with the same position as the pixel in the all-zero matrix is ​​changed to k. When a pixel in the unlabeled medical image marked as reliable is simultaneously marked as foreground by multiple category masks of the SAM foreground map, the pixel is assigned to the corresponding category k in the first coarse segmentation pseudo-label, and the value of the point with the same position as the pixel in the all-zero matrix is ​​changed to k, until all the pixels in the unlabeled medical image marked as reliable are traversed to obtain a complete first fine segmentation pseudo-label.

7. The medical image segmentation method based on basic model assisted semi-supervised learning as claimed in claim 1, characterized in that: The region exchange strategy is used to process the unreliable dataset to generate synthetic images and synthetic labels. The specific process is as follows: for an unreliable unlabeled medical image marked as unreliable, a labeled medical image is randomly selected, and a binary mask with a value of 1 in the foreground area and 0 in the rest is pre-constructed. The size of the binary mask foreground area is smaller than that of the unreliable unlabeled medical image marked as unreliable. All pixels in the foreground area of ​​the binary mask in the unreliable unlabeled medical image are exchanged with all pixels in the foreground area of ​​the binary mask in the randomly selected labeled medical image. After the exchange, two synthetic images are generated. Each synthetic image is processed by Through the U-Net model, each synthetic image corresponds to a pseudo-label probability map of the synthetic image, and the unreliable unlabeled medical images marked as unreliable are passed through the U-Net model, and the unreliable unlabeled medical images marked as unreliable are corresponding to a pseudo-label probability map and used as the unreliable pseudo-label probability map. The maximum connected area of ​​each category in the unreliable pseudo-label probability map is extracted, and a new fine segmentation pseudo-label is generated and used as the second fine segmentation pseudo-label. All pixels in the second fine segmentation pseudo-label belonging to the binary mask foreground area are exchanged with all pixels in the binary mask foreground area in the true label of the labeled medical image, and a set of synthetic labels are generated after the exchange.

8. The medical image segmentation method based on basic model-assisted semi-supervised learning as claimed in claim 1, characterized in that: The specific process of performing a perturbation operation on each unlabeled medical image to generate a weakly perturbed image and a strongly perturbed image is as follows: for each unlabeled medical image, a weak perturbation operation is first performed to obtain a weakly perturbed image, and then two different strong perturbation operations are performed on the weakly perturbated image to generate two strongly perturbated images; wherein the weak perturbation operation is to crop, rotate and / or flip the unlabeled medical image; and the strong perturbation operation is to perform Gaussian blur, color jitter and / or spatial deformation on the weakly perturbated image.

9. The medical image segmentation method based on basic model-assisted semi-supervised learning according to claim 7, characterized in that: The loss function used when fine-tuning the pre-trained U-Net model is By label loss Reliable data loss Unlabeled loss and unreliable data loss The weighted sum is obtained, and the specific form is as follows: λ(t)=μ·exp(-ε·(1-t / t total ) 2 ) Among them, α, β, γ, μ, ε are all weight hyperparameters; λ(t) represents the loss weight at the tth iteration; t total Indicates the total number of iterations; represents the first cross entropy loss; represents the first Dice loss; P l Represents the probability map generated by the U-Net model for labeled medical images; Y l Indicates the true label corresponding to the labeled medical image; represents the second cross entropy loss; represents the second Dice loss; P r Represents the probability map generated by the U-Net model for unlabeled medical images marked as reliable; represents the fine segmentation pseudo-label corresponding to the unlabeled medical image marked as reliable; τ represents the confidence threshold; P w represents the probability map generated by the U-Net model for the weakly perturbed image; P s1 ,P s2 Represents the probability map generated by the U-Net model for a strongly perturbed image; Indicates the third Dice loss; Indicates the fourth Dice loss; and represents auxiliary variables; M represents the binary mask constructed in the region swap strategy; ⊙ represents the element-wise dot product; Indicates the fifth Dice loss; represents the sixth Dice loss; ψ represents the control parameter; P dm ,P lm Both represent pseudo-label probability maps generated by the U-Net model for synthetic images; Y dm ,Y lm All represent synthetic tags.

10. A medical image segmentation system based on basic model-assisted semi-supervised learning, characterized in that: include: An image acquisition module, used for acquiring a medical image to be segmented; A result acquisition module, used for inputting the medical image to be segmented into a trained medical image segmentation model to obtain the segmentation result of the medical image; wherein the medical image segmentation model is obtained by fine-tuning a pre-trained U-Net model on an unlabeled data set and a labeled data set; the pre-trained U-Net model is obtained by training the U-Net model on a labeled data set in a supervised learning mode; the unlabeled data set only contains unlabeled medical images, and the labeled data set contains both labeled medical images and true labels; During the fine-tuning process, the unlabeled medical image is input into the pre-trained U-Net model, a coarse segmentation probability map of the unlabeled medical image is generated and used as the first coarse segmentation probability map, the category with the largest probability value in the first coarse segmentation probability map is taken as the coarse segmentation pseudo-label of the unlabeled medical image and used as the first coarse segmentation pseudo-label, the prompt information required by the visual base model is generated based on the first coarse segmentation pseudo-label, the generated prompt information is passed through the prompt encoder of the visual base model to obtain the prompt feature, and the unlabeled medical image is processed by the encoder of the visual base model to obtain the image feature of the unlabeled medical image, and then the image feature and the prompt feature of the unlabeled medical image are input into the mask decoder of the visual base model, and the mask decoder generates a set of SAM foreground images, and the intersection-union ratio between the SAM foreground image and the first coarse segmentation pseudo-label of each category is calculated, and the calculation results of the intersection-union ratio are averaged to obtain the consistency score, which is sorted in descending order of the consistency score. The unlabeled medical images are classified and labeled, and the unlabeled medical images marked as reliable are processed through the joint judgment of SAM and U-Net and the foreground conflict processing strategy to generate the first fine segmentation pseudo-label. The unlabeled medical images marked as reliable and the first fine segmentation pseudo-label constitute a reliable data set, and the unlabeled medical images marked as unreliable constitute an unreliable data set. The unreliable data set is processed by the region exchange strategy, and a bidirectional displacement operation is performed between the labeled medical images and the unlabeled medical images marked as unreliable to generate synthetic images and synthetic labels as supervision in the fine-tuning process, and a perturbation operation is performed on each unlabeled medical image. For each unlabeled medical image, a weakly perturbed image and two independent strongly perturbed images are generated. The pre-trained U-Net model generates corresponding probability maps for the weakly perturbed image and each strongly perturbed image. The probability map of the weakly perturbed image is used as a supervisory signal to perform supervised learning on the probability maps of the two strongly perturbed images.

Images