Medical image segmentation method and system based on basic model-assisted semi-supervised learning

Abstract

The present invention discloses a medical image segmentation method and system based on basic model-assisted semi-supervised learning. In the medical image segmentation task, the present invention significantly improves the labeling accuracy and segmentation effect by introducing an innovative method that combines the visual basic model Segment Anything Model (SAM) with the U-Net model. Compared with the existing semi-supervised learning methods, the present invention solves the problem of low-quality pseudo-labels by introducing a reliability scoring mechanism and a foreground conflict processing strategy, and effectively utilizes unreliable images through a region exchange strategy. Ultimately, the present invention provides a new solution to the problems of low pseudo-label quality and high data annotation cost in medical image segmentation, greatly improves the performance of medical image segmentation, and demonstrates superior segmentation effects on multiple benchmark datasets.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and medical image processing, and in particular relates to a medical image segmentation method and system based on basic model-assisted semi-supervised learning. Background Art

[0002] In recent years, medical image segmentation has provided crucial information for disease diagnosis, quantitative analysis, and treatment planning, effectively improving doctors' work efficiency and accuracy. Although deep learning technology has achieved some success in the field of medical image segmentation, traditional medical image segmentation methods typically rely on large amounts of high-quality annotated data for training, which is extremely expensive to obtain. Medical image annotation requires specialized knowledge and extensive experience, and is often limited by data availability and privacy policies. In particular, in three-dimensional volumetric medical imaging modalities (such as CT and MRI), each slice requires detailed delineation, which significantly increases the workload of manual annotation.

[0003] To address this challenge, semi-supervised learning (SSL) has been widely used in medical image segmentation. Existing SSL methods can effectively utilize a large amount of unlabeled data to enhance the generalization ability of the model, and usually include two training processes: consistency regularization and pseudo-label generation. However, these methods still have a significant performance gap compared with fully supervised learning methods when there is very little labeled data. This gap is mainly due to inappropriate perturbations and low-quality pseudo-labels, which lead to inaccurate decision boundaries and thus affect the learning quality of the model. In addition, excessive reliance on traditional models (such as U-Net) may not be able to effectively cope with complex segmentation problems.

[0004] In recent years, visually based models, such as the Segment Anything Model (SAM), have demonstrated excellent generalization capabilities in image segmentation, attracting significant research attention. While SAM can perform medical image segmentation tasks after fine-tuning, its performance remains inferior to traditional semi-supervised methods. Given that SAM can provide useful labels as prior information, it could potentially be combined with SSL methods to further improve medical image segmentation performance. However, applying SAM's excellent generalization capabilities to semi-supervised learning for medical image segmentation remains exploratory.

[0005] Existing methods have the following main problems when dealing with medical image segmentation:

[0006] Low-quality pseudo-labels: When existing semi-supervised learning methods generate pseudo-labels, the quality of the pseudo-labels is low, resulting in inaccurate decision boundaries of the model, which affects the quality of learning and segmentation effect.

[0007] Inadequate response to complex segmentation problems: Traditional methods show significant performance degradation when dealing with complex medical image segmentation problems, especially when there is very little labeled data, and cannot meet the requirements of practical applications.

[0008] Therefore, designing a medical image segmentation method that can effectively combine visual basic models (such as SAM) with semi-supervised learning methods to meet the above challenges and improve the performance of medical image segmentation is a technical problem that needs to be solved urgently. Summary of the Invention

[0009] The present invention aims to address the existing problem of effectively utilizing unlabeled data and improving segmentation accuracy and robustness in medical image segmentation tasks, and to provide a medical image segmentation method and system based on semi-supervised learning assisted by a basic model. By combining the advantages of the U-Net model and the Segment Anything Model (SAM), the present invention effectively utilizes unlabeled data and generates high-precision segmentation results.

[0010] In order to achieve the above-mentioned object of the invention, the present invention specifically adopts the following technical solutions:

[0011] In a first aspect, the present invention provides a medical image segmentation method based on basic model-assisted semi-supervised learning, the specific process of which is as follows: a medical image to be segmented is input into a trained medical image segmentation model to obtain a segmentation result of the medical image; wherein the medical image segmentation model is obtained by fine-tuning a pre-trained U-Net model on an unlabeled dataset and a labeled dataset; the pre-trained U-Net model is obtained by training the U-Net model on a labeled dataset in a supervised learning mode; the unlabeled dataset only contains unlabeled medical images, and the labeled dataset contains both labeled medical images and true labels;

[0012] During the fine-tuning process, the unlabeled medical image is input into the pre-trained U-Net model to generate a coarse segmentation probability map of the unlabeled medical image and use it as the first coarse segmentation probability map. The category with the largest probability value in the first coarse segmentation probability map is taken as the coarse segmentation pseudo-label of the unlabeled medical image and used as the first coarse segmentation pseudo-label. The prompt information required by the visual base model is generated based on the first coarse segmentation pseudo-label. The generated prompt information is passed through the prompt encoder of the visual base model to obtain the prompt feature, and the unlabeled medical image is processed through the encoder of the visual base model to obtain the image feature of the unlabeled medical image. Subsequently, the image feature and prompt feature of the unlabeled medical image are input into the mask decoder of the visual base model. The mask decoder generates a set of SAM foreground images, calculates the intersection-over-union ratio between the SAM foreground image and the first coarse segmentation pseudo-label of each category, and takes the average of the calculated results of the intersection-over-union ratio to obtain the consistency score, which is sorted in descending order of the consistency score. The unlabeled medical images are classified and labeled, and the unlabeled medical images marked as reliable are processed through the joint judgment of SAM and U-Net and the foreground conflict processing strategy to generate the first fine segmentation pseudo label. The reliable unlabeled medical images and the first fine segmentation pseudo label constitute a reliable data set, and the unreliable unlabeled medical images marked as unreliable constitute an unreliable data set. The unreliable data set is processed using the region exchange strategy, and a bidirectional displacement operation is performed between the labeled medical images and the unreliable unlabeled medical images to generate synthetic images and synthetic labels as supervision in the fine-tuning process. A perturbation operation is performed on each unlabeled medical image, and a weakly perturbed image and two independent strongly perturbed images are generated for each unlabeled medical image. The pre-trained U-Net model generates corresponding probability maps for the weakly perturbed image and each strongly perturbed image. The probability map of the weakly perturbed image is used as the supervision signal to perform supervised learning on the probability maps of the two strongly perturbed images.

[0013] Based on the above solution, each step can be implemented in the following preferred specific manner.

[0014] As a preferred embodiment of the first aspect, the network structure of the U-Net model is symmetrical and is divided into a downsampling path and an upsampling path;

[0015] In the downsampling path, the input medical image is first processed by the first convolution layer to generate a first feature map, the first feature map is downsampled by the maximum pooling layer to obtain a low-resolution first feature map, then the low-resolution first feature map is processed by the second convolution layer in sequence to generate a second feature map, the second feature map is downsampled again by the maximum pooling layer to obtain a low-resolution second feature map, the low-resolution second feature map is processed by the third convolution layer to generate a third feature map, the third feature map is processed by the maximum pooling operation to generate a low-resolution third feature map, the low-resolution third feature map is processed by the fourth convolution layer to generate a fourth feature map, the fourth feature map is processed by the maximum pooling operation to obtain a low-resolution fourth feature map, finally, the low-resolution fourth feature map is processed by the fifth convolution layer to generate the fifth feature map and used as the input of the upsampling path;

[0016] In the upsampling path, the fifth feature map is upsampled and spliced ​​with the fourth feature map in the downsampling path. After splicing, it is processed by the sixth convolution layer to generate the sixth feature map. After upsampling, the sixth feature map is spliced ​​with the third feature map in the downsampling path. After splicing, it is processed by the seventh convolution layer to generate the seventh feature map. After upsampling, the seventh feature map is spliced ​​with the second feature map in the downsampling path. After splicing, it is processed by the eighth convolution layer to generate the eighth feature map. After upsampling, the eighth feature map is spliced ​​with the first feature map in the downsampling path. After splicing, it is processed by the ninth convolution layer to generate the ninth feature map. Finally, the ninth feature map is processed by an output convolution layer, and the number of output channels corresponds to the number of target categories.

[0017] Furthermore, the convolution kernel size of the U-Net model is configured as follows:

[0018] In the downsampling path, all convolutional layers use a 3×3 convolution kernel size to gradually extract features, and a 2×2 maximum pooling layer is used to reduce the resolution of the feature map and generate a deeper feature map;

[0019] In the upsampling path, the convolution layers after splicing all use a 3×3 convolution kernel size, which is used to further process the spliced ​​feature map after each upsampling operation, thereby restoring the resolution of the medical image and extracting richer feature information; upsampling is achieved by transposed convolution with a 2×2 convolution kernel size to increase the resolution of the feature map; the convolution kernel size of the output convolution layer is 1×1, which is used to convert the last generated feature map into the number of channels corresponding to the number of target categories to generate the final segmentation result.

[0020] As a preferred embodiment of the first aspect, the specific generation process of the prompt information required by the visual base model is as follows: in the first coarse segmentation pseudo-label, only the largest continuous area in the non-empty foreground mask of each category excluding the background is always retained; after obtaining the continuous area, prompt information is preset, wherein the prompt information is a sparse prompt and / or a dense prompt, wherein the sparse prompt is a point and / or a border, and the dense prompt is a category-level foreground mask captured from the first coarse segmentation pseudo-label;

[0021] If the sparse hint is a border, the minimum circumscribed rectangular area of ​​the continuous area is used as the border representing the sparse hint; if the sparse hint is a point, it is generated in an iterative manner. In each iteration, a point is sampled from the unlabeled medical image until the iteration ends, and all the sampled points are used as a set of points representing the sparse hint.

[0022] As a preferred embodiment of the first aspect above, the specific process of classifying and labeling unlabeled medical images in descending order of consistency scores is as follows: the w unlabeled medical images with the lowest consistency scores are regarded as bottom medical images, the unlabeled medical images with a consistency score not lower than a preset threshold are marked as reliable, the unlabeled medical images with a consistency score lower than the preset threshold and belonging to the bottom medical images are marked as unreliable, and the remaining unlabeled medical images are marked as neutral and no additional processing is performed.

[0023] As a preferred embodiment of the first aspect mentioned above, the foreground conflict processing strategy is used to process the unlabeled medical image marked as reliable, and the specific process of generating the first fine segmentation pseudo-label is as follows: the all-zero matrix is ​​used as the initial state of the first fine segmentation pseudo-label; when a pixel in the unlabeled medical image marked as reliable is marked as foreground by only one category mask k of the SAM foreground image, the pixel is assigned to the category in the category mask, and the value of the point with the same position as the pixel in the all-zero matrix is ​​changed to k; when a pixel in the unlabeled medical image marked as reliable is simultaneously marked as foreground by multiple category masks of the SAM foreground image, the pixel is assigned to the corresponding category k in the first coarse segmentation pseudo-label, and the value of the point with the same position as the pixel in the all-zero matrix is ​​changed to k, until all pixels in the unlabeled medical image marked as reliable are traversed to obtain a complete first fine segmentation pseudo-label.

[0024] As a preferred embodiment of the first aspect, the region exchange strategy is used to process the unreliable data set, and the specific process of generating a synthetic image and a synthetic label is as follows: for an unreliable unlabeled medical image marked as unreliable, a labeled medical image is randomly selected, and a binary mask with a value of 1 in the foreground area and 0 in the rest is pre-constructed, the size of the binary mask foreground area is smaller than the size of the unreliable unlabeled medical image marked as unreliable, all pixels in the binary mask foreground area of ​​the unreliable unlabeled medical image are exchanged with all pixels in the binary mask foreground area of ​​the randomly selected labeled medical image, and two synthetic images are generated after the exchange, each synthetic image having a value of 1 in the foreground area. The composite images are each passed through the U-Net model, and a pseudo-label probability map of the composite image is obtained for each composite image. The unreliable unlabeled medical images marked as unreliable are passed through the U-Net model, and a pseudo-label probability map is obtained for the unreliable unlabeled medical images marked as unreliable. The maximum connected area of ​​each category in the unreliable pseudo-label probability map is extracted, and a new fine segmentation pseudo-label is generated and used as the second fine segmentation pseudo-label. All pixels in the second fine segmentation pseudo-label belonging to the binary mask foreground area are exchanged with all pixels in the binary mask foreground area in the true label of the labeled medical image, and a set of composite labels is generated after the exchange.

[0025] As a preferred embodiment of the first aspect, a perturbation operation is performed on each unlabeled medical image to generate a weakly perturbed image and a strongly perturbed image. The specific process is as follows: for each unlabeled medical image, a weak perturbation operation is first performed to obtain a weakly perturbated image, and then two different strong perturbation operations are performed on the weakly perturbated image to generate two strongly perturbated images; wherein the weak perturbation operation is cropping, rotating and / or flipping the unlabeled medical image; and the strong perturbation operation is Gaussian blurring, color dithering and / or spatial deformation of the weakly perturbated image.

[0026] As a preferred embodiment of the first aspect above, the loss function used in fine-tuning the pre-trained U-Net model is By label loss Reliable data loss Unlabeled loss and unreliable data loss The weighted sum is obtained, and the specific form is as follows:

[0027]

[0028] λ(t)=μ·exp(-ε·(1-t / t total ) 2 )

[0029]

[0030]

[0031] Among them, α, β, γ, τ, ε are all weight hyperparameters; λ(t) represents the loss weight at the tth iteration; t total Indicates the total number of iterations; represents the first cross entropy loss; represents the first Dice loss; P l Y represents the probability map generated by the U-Net model for labeled medical images; l Indicates the true label corresponding to the labeled medical image; represents the second cross entropy loss; represents the second Dice loss; P r Represents the probability map generated by the U-Net model for unlabeled medical images marked as reliable; represents the fine segmentation pseudo-label corresponding to the unlabeled medical image marked as reliable; τ represents the confidence threshold; P w Represents the probability map generated by the weak perturbation image through the U-Net model; P s1 ,P s2 Represents the probability map generated by the U-Net model for a strongly perturbed image; represents the third Dice loss; Indicates the fourth Dice loss; and represents auxiliary variables; M represents the binary mask constructed in the region swap strategy; ⊙ represents the element-wise dot product; Indicates the fifth Dice loss; represents the sixth Dice loss; ψ represents the control parameter; P dm ,P lm Both represent pseudo label probability maps generated by the U-Net model for synthetic images; Y dm ,Y lm All represent synthetic tags.

[0032] In a second aspect, the present invention provides a medical image segmentation system based on basic model-assisted semi-supervised learning, comprising:

[0033] An image acquisition module, used for acquiring medical images to be segmented;

[0034] a result acquisition module, configured to input the medical image to be segmented into a trained medical image segmentation model to obtain a segmentation result of the medical image; wherein the medical image segmentation model is obtained by fine-tuning a pre-trained U-Net model on an unlabeled dataset and a labeled dataset; the pre-trained U-Net model is obtained by training the U-Net model on a labeled dataset in a supervised learning mode; the unlabeled dataset contains only unlabeled medical images, and the labeled dataset contains both labeled medical images and true labels;

[0035] During the fine-tuning process, the unlabeled medical image is input into the pre-trained U-Net model to generate a coarse segmentation probability map of the unlabeled medical image and use it as the first coarse segmentation probability map. The category with the largest probability value in the first coarse segmentation probability map is taken as the coarse segmentation pseudo-label of the unlabeled medical image and used as the first coarse segmentation pseudo-label. The prompt information required by the visual base model is generated based on the first coarse segmentation pseudo-label. The generated prompt information is passed through the prompt encoder of the visual base model to obtain the prompt feature, and the unlabeled medical image is processed through the encoder of the visual base model to obtain the image feature of the unlabeled medical image. Subsequently, the image feature and prompt feature of the unlabeled medical image are input into the mask decoder of the visual base model. The mask decoder generates a set of SAM foreground images, calculates the intersection-over-union ratio between the SAM foreground image and the first coarse segmentation pseudo-label of each category, and takes the average of the calculated results of the intersection-over-union ratio to obtain the consistency score, which is sorted in descending order of the consistency score. The unlabeled medical images are classified and labeled, and the unlabeled medical images marked as reliable are processed through the joint judgment of SAM and U-Net and the foreground conflict processing strategy to generate the first fine segmentation pseudo label. The reliable unlabeled medical images and the first fine segmentation pseudo label constitute a reliable data set, and the unreliable unlabeled medical images marked as unreliable constitute an unreliable data set. The unreliable data set is processed using the region exchange strategy, and a bidirectional displacement operation is performed between the labeled medical images and the unreliable unlabeled medical images to generate synthetic images and synthetic labels as supervision in the fine-tuning process. A perturbation operation is performed on each unlabeled medical image, and a weakly perturbed image and two independent strongly perturbed images are generated for each unlabeled medical image. The pre-trained U-Net model generates corresponding probability maps for the weakly perturbed image and each strongly perturbed image. The probability map of the weakly perturbed image is used as the supervision signal to perform supervised learning on the probability maps of the two strongly perturbed images.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention introduces a combination of the visual base model SAM and the U-Net model in the medical image segmentation task, forming a semi-supervised learning method, which aims to solve the problems of strong dependence on high-quality labeled data, low quality of pseudo-labels and poor adaptability to data perturbations in the current technology, while overcoming the limitations of SAM when used alone in processing complex medical images. Compared with traditional semi-supervised learning methods, the present invention introduces the SAM model, utilizes its excellent generalization ability in semantic segmentation tasks, and the professionalism of the U-Net model in medical image processing, to achieve effective utilization of unlabeled data. Through the comprehensive application of consistency scoring, foreground conflict processing strategy and region exchange strategy, the present invention can improve the quality of pseudo-labels and effectively process data with different reliability, providing higher accuracy and robustness for medical image segmentation tasks. This method significantly improves the segmentation performance of medical image segmentation models under limited labeled data, reduces dependence on manual annotation, and provides a novel and effective solution for the application of semi-supervised learning in the field of medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of the steps of the present invention;

[0039] Figure 2 Schematic diagram of the foreground conflict handling strategy of the present invention;

[0040] Figure 3 A schematic diagram of the regional exchange strategy of the present invention;

[0041] Figure 4 Schematic diagram of the specific process of fine-tuning the medical image segmentation model of the present invention;

[0042] Figure 5 A training and testing flowchart of the medical image segmentation model SESAME in an embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of test results of an embodiment of the present invention;

[0044] Figure 7 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0046] In the description of the present invention, it should be understood that the terms "first" and "second" are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly specifying the number of technical features being described. Therefore, features defined as "first" or "second" may explicitly or implicitly include at least one of such features.

[0047] In the field of medical image segmentation, although deep learning technology has made significant progress, traditional segmentation methods often rely on large amounts of labeled data, which is not only expensive but also time-consuming to obtain. In particular, in three-dimensional medical imaging (such as CT and MRI), the labeling workload is extremely large. Furthermore, existing semi-supervised learning methods still struggle to match the performance of fully supervised learning methods when faced with small amounts of labeled data, primarily due to the low quality of pseudo-labels and the model's poor adaptability to perturbations. To address these issues, the present invention proposes a medical image segmentation method based on semi-supervised learning assisted by a basic model. This method combines the powerful generalization capabilities of the Segment Anything Model (SAM) with the domain-specific knowledge of the U-Net model. By utilizing the foreground mask generated by the SAM as prior information, it optimizes the segmentation quality of unlabeled images. Specifically, the present invention designs a foreground conflict handling strategy to address category conflicts in the foreground mask generated by the SAM, ensuring the consistency of the segmentation results. Furthermore, a region exchange strategy is introduced to effectively alleviate the problem of inconsistent data distribution by exchanging pixel regions between unreliable and labeled images. This method not only significantly improves the accuracy and robustness of medical image segmentation, but also can fully utilize the information of unlabeled data when labeled data is limited, laying a solid foundation for its widespread application in medical diagnosis and treatment planning in the future.

[0048] Therefore, if Figure 1 As shown, the medical image segmentation method based on basic model assisted semi-supervised learning provided by the present invention specifically includes the following steps:

[0049] The specific process of the medical image segmentation method is as follows: a medical image to be segmented is input into a trained medical image segmentation model to obtain a segmentation result of the medical image; wherein the medical image segmentation model is obtained by fine-tuning a pre-trained U-Net model on an unlabeled dataset and a labeled dataset; the pre-trained U-Net model is obtained by training the U-Net model on a labeled dataset under a supervised learning mode; the unlabeled dataset only contains unlabeled medical images, and the labeled dataset contains both labeled medical images and true labels;

[0050] During the fine-tuning process, the unlabeled medical image is input into the pre-trained U-Net model to generate a coarse segmentation probability map of the unlabeled medical image and use it as the first coarse segmentation probability map. The category with the largest probability value in the first coarse segmentation probability map is taken as the coarse segmentation pseudo-label of the unlabeled medical image and used as the first coarse segmentation pseudo-label. The prompt information required by the visual base model is generated based on the first coarse segmentation pseudo-label. The generated prompt information is passed through the prompt encoder of the visual base model to obtain the prompt feature, and the unlabeled medical image is processed through the encoder of the visual base model to obtain the image feature of the unlabeled medical image. Subsequently, the image feature and prompt feature of the unlabeled medical image are input into the mask decoder of the visual base model. The mask decoder generates a set of SAM foreground images, calculates the intersection-over-union ratio between the SAM foreground image and the first coarse segmentation pseudo-label of each category, and takes the average of the calculated results of the intersection-over-union ratio to obtain the consistency score, which is sorted in descending order of the consistency score. The unlabeled medical images are classified and labeled, and the unlabeled medical images marked as reliable are processed through the joint judgment of SAM and U-Net and the foreground conflict processing strategy to generate the first fine segmentation pseudo label. The reliable unlabeled medical images and the first fine segmentation pseudo label constitute a reliable data set, and the unreliable unlabeled medical images marked as unreliable constitute an unreliable data set. The unreliable data set is processed using the region exchange strategy, and a bidirectional displacement operation is performed between the labeled medical images and the unreliable unlabeled medical images to generate synthetic images and synthetic labels as supervision in the fine-tuning process. A perturbation operation is performed on each unlabeled medical image, and a weakly perturbed image and two independent strongly perturbed images are generated for each unlabeled medical image. The pre-trained U-Net model generates corresponding probability maps for the weakly perturbed image and each strongly perturbed image. The probability map of the weakly perturbed image is used as the supervision signal to perform supervised learning on the probability maps of the two strongly perturbed images.

[0051] It is important to note that before the above-mentioned medical image segmentation model was actually applied to medical image segmentation, it was first pre-trained using labeled medical images. During the pre-training process, a combination of cross-entropy loss and Dice loss was used as the loss function to ensure that the medical image segmentation model could achieve high-precision segmentation results when processing medical images. This combination of loss functions helps to improve the medical image segmentation model's sensitivity to different class boundaries and enhance the accuracy and robustness of the segmentation results.

[0052] In a preferred embodiment of the present invention, a medical image segmentation model (Semi-supervised learning method assisted by SAM for medical imagesegmentation, SESAME) is designed based on the above-mentioned visual basic model, thereby providing a medical image segmentation method based on basic model-assisted semi-supervised learning. In the present invention, thanks to the powerful performance of the visual basic model SAM and its versatility in medical image segmentation, the medical image segmentation model adopts a basic model-assisted U-Net and SAM joint judgment structure, and uses the trained U-Net model as the medical image segmentation model. Its network structure is symmetrical and is divided into a downsampling path (encoder) and an upsampling path (decoder). The specific structure is as follows:

[0053] In the downsampling path, the input medical image is first processed by the first convolution layer to generate a first feature map, the first feature map is downsampled by the maximum pooling layer to obtain a low-resolution first feature map, then the low-resolution first feature map is processed by the second convolution layer in sequence to generate a second feature map, the second feature map is downsampled again by the maximum pooling layer to obtain a low-resolution second feature map, the low-resolution second feature map is processed by the third convolution layer to generate a third feature map, the third feature map is processed by the maximum pooling operation to generate a low-resolution third feature map, the low-resolution third feature map is processed by the fourth convolution layer to generate a fourth feature map, the fourth feature map is processed by the maximum pooling operation to obtain a low-resolution fourth feature map, finally, the low-resolution fourth feature map is processed by the fifth convolution layer to generate the fifth feature map and used as the input of the upsampling path;

[0054] In the upsampling path, the fifth feature map is upsampled and spliced ​​with the fourth feature map in the downsampling path. After splicing, it is processed by the sixth convolution layer to generate the sixth feature map. After upsampling, the sixth feature map is spliced ​​with the third feature map in the downsampling path. After splicing, it is processed by the seventh convolution layer to generate the seventh feature map. After upsampling, the seventh feature map is spliced ​​with the second feature map in the downsampling path. After splicing, it is processed by the eighth convolution layer to generate the eighth feature map. After upsampling, the eighth feature map is spliced ​​with the first feature map in the downsampling path. After splicing, it is processed by the ninth convolution layer to generate the ninth feature map. Finally, the ninth feature map is processed by an output convolution layer, and the number of output channels corresponds to the number of target categories.

[0055] Furthermore, in an embodiment of the present invention, the convolution kernel size of the U-Net model is configured as follows: in the downsampling path, all convolution layers adopt a convolution kernel size of 3×3 for gradually extracting features, and a maximum pooling layer of 2×2 is adopted to reduce the resolution of the feature map and generate a deeper feature map; in the upsampling path, the convolution layers passed after splicing all adopt a convolution kernel size of 3×3 for further processing the spliced ​​feature map after each upsampling operation, thereby restoring the resolution of the image and extracting richer feature information; upsampling is achieved by transposed convolution with a convolution kernel size of 2×2 to increase the resolution of the feature map; the convolution kernel size of the output convolution layer is 1×1, which is used to convert the finally generated feature map into the number of channels corresponding to the number of target categories to generate the final segmentation result.

[0056] In the U-Net model of the embodiment of the present invention, considering that the input of the medical image segmentation model SESAME can be both unlabeled medical images and labeled medical images, the two are not distinguished in the description of the specific structure of the medical image segmentation model, and are uniformly represented by "medical image". The specific processing flow of the U-Net model is as follows: In the downsampling path of the U-Net model, the input medical image of size 256×256 is first processed by a first convolution layer of size 3×3 to generate a first feature map of size 256×256×64. Then, the first feature map is downsampled through a 2×2 maximum pooling layer to obtain a low-resolution first feature map of size 128×128×64. Subsequently, the low-resolution first feature map is sequentially processed by a second convolution layer of size 3×3 to generate a second feature map, and is downsampled to 64×64×128 through a maximum pooling layer to obtain a low-resolution second feature map. The low-resolution second feature map then undergoes a third convolutional layer of size 3×3 to generate the third feature map. This is then downsampled again to 32×32×256 using max pooling, resulting in the third low-resolution feature map. The third low-resolution feature map then undergoes a fourth convolutional layer of size 3×3 to generate the fourth feature map. This is then downsampled to 16×16×512 using max pooling, resulting in the fourth low-resolution feature map. Finally, the fourth low-resolution feature map undergoes a fifth convolutional layer of size 3×3 to generate the fifth feature map of size 8×8×1024, which serves as the input to the upsampling path. In the upsampling path, the fifth feature map is first upsampled to restore its resolution to 16×16×512 and concatenated with the fourth feature map from the downsampling path to form a feature map containing more contextual information. This feature map then undergoes a sixth convolutional layer of size 3×3 to generate the sixth feature map. Next, the sixth feature map is upsampled again to 32×32×256 and concatenated with the third feature map in the downsampling path. It is then processed through the seventh convolutional layer of size 3×3 to generate the seventh feature map. Next, the seventh feature map is upsampled to 64×64×128 and concatenated with the second feature map in the downsampling path. It is processed through the eighth convolutional layer of size 3×3 to generate the eighth feature map. Finally, the eighth feature map is upsampled to the original image size of 256×256×64 and concatenated with the first feature map in the downsampling path. It is processed through the ninth convolutional layer of size 3×3 to generate the ninth feature map. Finally, the ninth feature map is processed through an output convolutional layer, and the final segmentation result with the number of output channels corresponding to the number of target categories is obtained, ensuring a high-precision segmentation effect.

[0057] The specific fine-tuning process of the medical image segmentation model of the present invention is described in detail below. Figure 4As shown in the figure, the whole process is mainly divided into three steps: coarse segmentation pseudo-label generation, joint judgment and improved semi-supervised perturbation.

[0058] In the first step, a U-Net model is pre-trained on a labeled dataset. The pre-trained U-Net model generates coarse segmentation pseudo-labels based on the unlabeled medical images. These pseudo-labels are then used to generate different categories of cues, including sparse cues (such as boxes or points) and dense cues (such as masks). These coarse segmentation pseudo-labels, along with the unlabeled medical images, are then fed into a visual foundational model (Segment Anything Model, SAM). This model processes the unlabeled dataset through an encoder, a cue encoder, and a mask decoder to generate a set of SAM foreground maps. Specifically, the SAM encoder extracts deep features (image features) from the unlabeled medical images, while the generated cue information is processed by the SAM cue encoder to generate cue features that match the image features. These cue features, along with the image features, are fed into the SAM mask decoder to generate the SAM foreground maps.

[0059] It should be noted that the specific form of the visual base model SAM used here is not limited, as long as it has general medical knowledge and includes three main components: encoder, prompt encoder and mask decoder, and can be combined with the SESAME model of the present invention in the segmentation task.

[0060] In the second step, to assess the reliability of unlabeled medical images, the present invention designs an image-level consistency scoring method based on the coarse segmentation pseudo-labels generated by U-Net and the foreground image with hint pseudo-labels generated by SAM. Specifically, the coarse segmentation pseudo-labels are jointly judged with the generated SAM foreground image, and the consistency score of the unlabeled medical images is calculated by calculating the intersection over union (IoU). Specifically, the IoU between the SAM foreground image and the first coarse segmentation pseudo-label of each category is calculated, and the IoU results are averaged to obtain the consistency score. The unlabeled medical images are then classified according to their reliability into three categories: "reliable," "neutral," and "unreliable." Fine segmentation pseudo-labels are generated for the unlabeled medical images marked as reliable (i.e., reliable images) using a foreground conflict resolution strategy. Finally, the unreliable dataset is processed using a region swap strategy, performing a bidirectional shift operation between the labeled and unreliable images to generate new synthetic images and synthetic labels, significantly improving the accuracy and robustness of medical image segmentation.

[0061] In the third step, data augmentation is performed on all unlabeled medical images to generate three different versions of images (i.e., a weakly perturbed image and two independent strongly perturbed images) to improve the segmentation effect. Specifically, for each unlabeled medical image, a weakly perturbed image is first generated through a weak perturbation operation. Then, two independent strongly perturbation operations are performed on the weakly perturbated image to generate two different strongly perturbed images. Each strongly perturbated image is supervised by the prediction results of the weakly perturbated image. Among them, the weak perturbation operation combines operations such as cropping, rotation, and flipping; the strong perturbation operation includes Gaussian blur, color jittering, and spatial deformation (CutMix).

[0062] Through the above three steps, the present invention can significantly improve the accuracy and robustness of medical image segmentation, and effectively utilize unlabeled data to improve the performance of the medical image segmentation model SESAME, effectively enhancing the segmentation performance of the medical image segmentation model for unlabeled medical images, making the medical image segmentation model more robust and accurate when processing enhanced medical images.

[0063] The principles of generating prompt information, classifying and labeling unlabeled medical images, foreground conflict processing strategy and region exchange strategy in the present invention, as well as the specific processing flow of each strategy are described in detail below.

[0064] It should be noted that in the present invention, the specific process of generating the prompt information required by the visual base model is as follows: in the first coarse segmentation pseudo-label, only the largest continuous area in the non-empty foreground mask of each category excluding the background is always retained. After obtaining the continuous area, prompt information is preset. The prompt information is sparse prompt and / or dense prompt. The sparse prompt is a point and / or a border, and the dense prompt is a category-level foreground mask captured from the first coarse segmentation pseudo-label;

[0065] If the sparse hint is a border, the minimum circumscribed rectangular area of ​​the continuous area is used as the border representing the sparse hint; if the sparse hint is a point, it is generated in an iterative manner. In each iteration, a point is sampled from the unlabeled medical image until the iteration ends, and all the sampled points are used as a set of points representing the sparse hint.

[0066] In this embodiment, in order to generate the prompt information required by the visual basic model (Segment Anything Model, SAM), this embodiment performs the following operations: First, to avoid possible noise, only the largest continuous area in the non-empty foreground mask of each category except the background is always retained in the coarse segmentation pseudo label. After the continuous region is obtained, prompt information is preset, where the prompt information is a sparse prompt and / or a dense prompt. The sparse prompt includes a point and a border, and the dense prompt is a mask.

[0067] If the sparse hint is a bounding box, the continuous region The minimum bounding rectangle area is used as the border representing the sparse hint;

[0068] If the sparse cues are points, they are generated iteratively. At each iteration, a point is sampled from the unlabeled medical image. All sampled points are used as a set of points representing the sparse cues until the iteration ends. Furthermore, if the sampled points at each iteration are within the coarse segmentation pseudo-label, they are considered positive points; otherwise, they are considered negative points. In addition to the sparse cues, the present invention also captures a class-level foreground mask from the coarse segmentation pseudo-label as a mask representing the dense cues.

[0069] It should be noted that, in the present invention, the specific process of classifying and labeling unlabeled medical images in descending order of consistency scores is as follows: the w unlabeled medical images with the lowest consistency scores are regarded as bottom medical images, the unlabeled medical images with a consistency score not lower than a preset threshold are marked as reliable, the unlabeled medical images with a consistency score lower than the preset threshold and belonging to the bottom medical images are marked as unreliable, and the remaining unlabeled medical images are marked as neutral and no additional processing is performed.

[0070] In this embodiment, the value of w is half of the total number of unlabeled medical images. That is, among all unlabeled medical images, the unlabeled medical images at the bottom (i.e., with the lowest consistency score) in the 50% ratio are regarded as bottom medical images. For example, if there are 200 unlabeled medical images after sorting, the 100 unlabeled medical images with the lowest consistency scores at the end are regarded as bottom medical images. Unlabeled medical images with a consistency score of not less than 0.85 are then marked as reliable, unlabeled medical images with a consistency score less than 0.85 and belonging to the bottom medical images are marked as unreliable, and the remaining unlabeled medical images are marked as neutral and do not undergo additional processing. The number and quality of high-confidence pseudo-labels with prompts are determined by setting this threshold and selecting the 50% sorting ratio. Of course, the threshold value and the value of w are both a method for implementing image segmentation with better effect given in this embodiment. Those skilled in the art can also select appropriate values ​​based on the actual image segmentation results.

[0071] It should be noted that, in the present invention, the foreground conflict processing strategy is used to generate a fine segmentation pseudo-label that improves segmentation accuracy. The foreground conflict processing strategy is used to process the unlabeled medical image marked as reliable, and the specific process of generating the first fine segmentation pseudo-label is as follows: the all-zero matrix is ​​used as the initial state of the first fine segmentation pseudo-label. When a pixel in the unlabeled medical image marked as reliable is marked as foreground by only one category mask k of the SAM foreground map, the pixel is assigned to the category in the category mask, and the value of the point with the same position as the pixel in the all-zero matrix is ​​changed to k. When a pixel in the unlabeled medical image marked as reliable is simultaneously marked as foreground by multiple category masks of the SAM foreground map, the pixel is assigned to the corresponding category k in the first coarse segmentation pseudo-label, and the value of the point with the same position as the pixel in the all-zero matrix is ​​changed to k, until all pixels in the unlabeled medical image marked as reliable are traversed, and a complete first fine segmentation pseudo-label is obtained.

[0072] Furthermore, in the process of generating fine segmentation pseudo labels for unlabeled medical images marked as reliable, a foreground conflict processing strategy is adopted in this embodiment to deal with the category prediction conflicts that may be caused by the SAM foreground map output by the visual base model. Specifically, when a pixel in a reliable image is marked as foreground by only one category mask of SAM, the pixel is assigned to that category. If the same pixel is marked as foreground by multiple category masks at the same time, the pixel is assigned to the corresponding category in the coarse segmentation pseudo label generated by U-Net. Through this strategy, corresponding fine segmentation pseudo labels are generated for each reliable image, and the reliable dataset composed of these fine segmentation pseudo labels and the reliable image is treated equally with the labeled dataset in the subsequent steps, thereby improving the accuracy and robustness of the segmentation results. Figure 2 The processing flow of the strategy is demonstrated, starting from the SAM foreground map generated by SAM, through the generation of coarse segmentation pseudo-labels and the processing of foreground conflict handling strategy, and finally forming fine segmentation pseudo-labels, providing more accurate supervision information for the medical image segmentation model.

[0073] It should be noted that, in the present invention, the region exchange strategy is adopted to process the unreliable data set, and the specific process of generating the synthetic image and synthetic label is as follows: for the unreliable unlabeled medical image marked as unreliable, a labeled medical image is randomly selected, and a binary mask with a value of 1 in the foreground area and 0 in the rest is pre-constructed, the size of the binary mask foreground area is smaller than the size of the unreliable unlabeled medical image marked as unreliable, all the pixels in the binary mask foreground area of ​​the unreliable unlabeled medical image are exchanged with all the pixels in the binary mask foreground area of ​​the randomly selected labeled medical image, and two synthetic images are generated after the exchange, each synthetic image The composite images are each passed through the U-Net model, and a pseudo-label probability map of the composite image is obtained for each composite image. The unreliable unlabeled medical images marked as unreliable are passed through the U-Net model, and a pseudo-label probability map is obtained for the unreliable unlabeled medical images marked as unreliable. The maximum connected area of ​​each category in the unreliable pseudo-label probability map is extracted, and a new fine segmentation pseudo-label is generated and used as the second fine segmentation pseudo-label. All pixels in the second fine segmentation pseudo-label belonging to the binary mask foreground area are exchanged with all pixels in the binary mask foreground area in the true label of the labeled medical image, and a set of composite labels is generated after the exchange.

[0074] In this embodiment, a region exchange strategy is used to alleviate the problem of inconsistent data distribution between unlabeled medical images marked as unreliable and labeled medical images. Figure 3 As shown, for an unlabeled medical image X that is labeled as unreliable d , randomly select a labeled medical image X l , pre-construct a binary mask with a value of 1 in the foreground area and 0 in the rest, the size of the binary mask foreground area is smaller than the size of the unlabeled medical image marked as unreliable, and swap all pixels in the binary mask foreground area in the unreliable unlabeled medical image with all pixels in the binary mask foreground area in the randomly selected labeled medical image, and generate two synthetic images after the swap, where the labeled medical image X l Corresponding synthetic image X dm , unlabeled medical image X marked as unreliable d Corresponding synthetic image X lm Each synthetic image is passed through the U-Net model, and each synthetic image corresponds to a pseudo label probability map of the synthetic image, where the synthetic image X dm Corresponding pseudo label probability map P dm , synthetic image X lm Corresponding pseudo label probability map P lmThe unlabeled medical images marked as unreliable are passed through the U-Net model, and a pseudo-label probability map is obtained for the unreliable unlabeled medical images marked as unreliable and used as the unreliable pseudo-label probability map. The maximum connected area of ​​each category in the unreliable pseudo-label probability map is extracted to generate a new fine segmentation pseudo-label and use it as the second fine segmentation pseudo-label. Compare all pixels in the foreground area of ​​the binary mask in the second fine segmentation pseudo-label with the true label Y of the labeled medical image l All pixels belonging to the foreground area of ​​the binary mask are swapped, and a set of synthetic labels are generated after the swap to guide the U-Net model to predict the synthetic image, where the synthetic label Y dm Corresponding to the true label Y l , synthetic label Y lm Corresponding to the second fine segmentation pseudo label This strategy effectively improves the segmentation accuracy of the medical image segmentation model when processing unreliable images, and enables unreliable images to be fully utilized in training.

[0075] It should be noted that, in this embodiment, the height of the binary mask foreground area is between 2% and 40% of the height of the unlabeled medical image marked as unreliable (i.e., the unreliable image), and its width is between 2% and 40% of the width of the unreliable image. This selection method is only a preferred method with better segmentation accuracy. The binary mask foreground area of ​​appropriate size can also be selected according to the actual size of the unreliable image, so it is not limited in the present invention.

[0076] It should be noted that in the present invention, the loss function used in fine-tuning the pre-trained U-Net model is By label loss Reliable data loss Unlabeled loss and unreliable data loss The weighted sum is obtained, and the specific form is as follows:

[0077]

[0078] λ(t)=μ·exp(-ε·(1-t / t total ) 2 )

[0079] Among them, α, β, γ, τ, ε are all weight hyperparameters; λ(t) represents the loss weight at the tth iteration; t total In this embodiment, α=β=γ=1, μ=0.1, and ε=5.

[0080] Specifically, the labeled loss is for the prediction of U-Net on the labeled dataset, which is combined with the first cross entropy loss and the first Dice loss

[0081]

[0082] Among them, P l Y represents the probability map generated by the U-Net model for labeled medical images; l Represents the true label corresponding to the labeled medical image.

[0083] The reliable data loss is for U-Net's predictions on reliable datasets, and uses the same loss combination as the labeled loss:

[0084]

[0085] in, represents the second cross entropy loss; represents the second Dice loss; P r Represents the probability map generated by the U-Net model for unlabeled medical images marked as reliable; Represents the fine segmentation pseudo labels corresponding to the unlabeled medical images that are labeled as reliable.

[0086] The unlabeled loss is based on the high-confidence prediction of the weakly perturbed image, calculates the consistency between the strongly perturbed image and the prediction, and uses the Dice loss to measure it. At the same time, the confidence threshold v is set to filter out pixels with less noise:

[0087]

[0088] Among them, P w Represents the probability map generated by the weak perturbation image through the U-Net model; P s1 ,P s2 Represents the probability map generated by the U-Net model for a strongly perturbed image; represents the third Dice loss; Indicates the fourth Dice loss.

[0089] In this embodiment, the confidence threshold τ is set to 0.5.

[0090] Unreliable data loss It is the region exchange strategy loss, which calculates the Dice loss between the prediction of the synthetic image generated after region exchange and its corresponding supervisory signal, and adjusts the influence of unreliable images in training through a control parameter ψ:

[0091]

[0092] in, and Represents auxiliary variables; represents the fifth Dice loss; M represents the binary mask constructed in the region swap strategy; ⊙ represents the element-wise dot product; represents the sixth Dice loss; P dm ,P lm Both represent pseudo label probability maps generated by the U-Net model for synthetic images; Y dm ,Y lm All represent synthetic tags.

[0093] In this embodiment, the control parameter ψ is set to 0.5.

[0094] It should be noted that in the present invention, the medical image can be a magnetic resonance imaging (MRI) or CT image. In this embodiment, the medical images are derived from the Prostate MR Imaging Segmentation dataset, the Automatic Cardiac Diagnosis Challenge dataset, and the LiTS2017 dataset. The Prostate MR Imaging Segmentation dataset is used to obtain magnetic resonance imaging of the prostate area;

[0095] The Automatic Cardiac Diagnosis Challenge dataset is used to obtain dynamic magnetic resonance imaging of the heart; the LiTS2017 dataset is used to obtain CT imaging of the liver.

[0096] The present invention will use a specific example below to demonstrate the application effect of the medical image segmentation method based on basic model assisted semi-supervised learning described in the above embodiment on a specific data set, so as to facilitate understanding of the essence of the present invention.

[0097] Example

[0098] The specific implementation process of the medical image segmentation method and system based on basic model assisted semi-supervised learning adopted in this embodiment is as described above and will not be repeated here.

[0099] The overall process of this embodiment can be divided into three stages: data preprocessing, model training, and image prediction. Figure 5 shown.

[0100] 1. Data preprocessing stage

[0101] Step 1: Preprocess the original medical images obtained by performing operations such as image cropping and flipping, and then perform data enhancement and process them into images of the same size (256×256).

[0102] Step 2: Label the preprocessed medical images and use tools such as Labelme to perform vector wireframe annotation to generate GroundTruth for subsequent medical image segmentation model training.

[0103] 2. Model training

[0104] Step 1: Build a training dataset and divide the training dataset into batches according to a fixed batch size, with a total of N.

[0105] In step 2, a batch of training samples indexed by i is sequentially selected from the training dataset, where i∈{0,1,…,N}. The medical image segmentation model SESAME is trained using the training samples from each batch. The specific structure of SESAME is described above and will not be repeated here. It uses an encoder-decoder architecture. During training, the label loss, reliable data loss, unlabel loss, and unreliable data loss are calculated for each training sample. Based on the total loss of all training samples in the batch, the network parameters of the entire medical image segmentation model are adjusted until all batches of the training dataset participate in model training. After the specified number of iterations is reached, the model converges and training is completed.

[0106] 3. Medical Image Semantic Segmentation

[0107] The images in the test set are directly used as input to the trained medical image segmentation model SESAME, and the probability of each pixel category is finally predicted. The category with the highest probability is selected as the final output through activation functions such as Sigmoid, thereby achieving medical image segmentation.

[0108] In this embodiment, the test results are as follows: Figure 6 shown. Figure 6 In the figure, the first sub-image from left to right is the segmentation result of magnetic resonance imaging of the human heart, the second sub-image is the segmentation result of magnetic resonance imaging of the prostate, and the third sub-image is the segmentation result of CT imaging of the liver. It can be seen that the medical image segmentation model SESAME, which introduces the basic model SAM and combines it with U-Net, can effectively improve the accuracy of semi-supervised medical image segmentation. By leveraging the generalization ability of SAM and the expertise of U-Net, the present invention achieves more accurate semantic segmentation of medical images, especially in maintaining structural details and improving classification accuracy. Compared with traditional methods, the present invention shows better performance in processing complex medical images, and provides a promising solution for improving the accuracy and robustness of medical image segmentation. It provides strong technical support for possible future applications.

[0109] It should also be noted that the medical image segmentation method based on basic model assisted semi-supervised learning in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a medical image segmentation system based on basic model assisted semi-supervised learning corresponding to the medical image segmentation method based on basic model assisted semi-supervised learning provided in the above embodiment, such as Figure 7 As shown, it includes:

[0110] An image acquisition module, used for acquiring medical images to be segmented;

[0111] a result acquisition module, configured to input the medical image to be segmented into a trained medical image segmentation model to obtain a segmentation result of the medical image; wherein the medical image segmentation model is obtained by fine-tuning a pre-trained U-Net model on an unlabeled dataset and a labeled dataset; the pre-trained U-Net model is obtained by training the U-Net model on a labeled dataset in a supervised learning mode; the unlabeled dataset contains only unlabeled medical images, and the labeled dataset contains both labeled medical images and true labels;

[0112] During the fine-tuning process, the unlabeled medical image is input into the pre-trained U-Net model to generate a coarse segmentation probability map of the unlabeled medical image and use it as the first coarse segmentation probability map. The category with the largest probability value in the first coarse segmentation probability map is taken as the coarse segmentation pseudo-label of the unlabeled medical image and used as the first coarse segmentation pseudo-label. The prompt information required by the visual base model is generated based on the first coarse segmentation pseudo-label. The generated prompt information is passed through the prompt encoder of the visual base model to obtain the prompt feature, and the unlabeled medical image is processed through the encoder of the visual base model to obtain the image feature of the unlabeled medical image. Subsequently, the image feature and prompt feature of the unlabeled medical image are input into the mask decoder of the visual base model. The mask decoder generates a set of SAM foreground images, calculates the intersection-over-union ratio between the SAM foreground image and the first coarse segmentation pseudo-label of each category, and takes the average of the calculated results of the intersection-over-union ratio to obtain the consistency score, which is sorted in descending order of the consistency score. The unlabeled medical images are classified and labeled, and the unlabeled medical images marked as reliable are processed through the joint judgment of SAM and U-Net and the foreground conflict processing strategy to generate the first fine segmentation pseudo label. The reliable unlabeled medical images and the first fine segmentation pseudo label constitute a reliable data set, and the unreliable unlabeled medical images marked as unreliable constitute an unreliable data set. The unreliable data set is processed using the region exchange strategy, and a bidirectional displacement operation is performed between the labeled medical images and the unreliable unlabeled medical images to generate synthetic images and synthetic labels as supervision in the fine-tuning process. A perturbation operation is performed on each unlabeled medical image, and a weakly perturbed image and two independent strongly perturbed images are generated for each unlabeled medical image. The pre-trained U-Net model generates corresponding probability maps for the weakly perturbed image and each strongly perturbed image. The probability map of the weakly perturbed image is used as the supervision signal to perform supervised learning on the probability maps of the two strongly perturbed images.

[0113] It should be noted that the image acquisition module acquires the medical image to be segmented, and the result acquisition module acquires a trained medical image segmentation model. Both acquisition methods can be online or offline. For online acquisition, the image acquisition module can acquire medical images of the target object online using an imaging device, and the result acquisition module can train the medical image segmentation model through online training. For offline acquisition, the image acquisition module can acquire medical images that have already been acquired and stored by the imaging device, and the result acquisition module can read the trained medical image segmentation model stored on a storage medium.

[0114] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A medical image segmentation method based on basic model-assisted semi-supervised learning, characterized in that: The specific process of the medical image segmentation method is as follows: a medical image to be segmented is input into a trained medical image segmentation model to obtain a segmentation result of the medical image; wherein the medical image segmentation model is obtained by fine-tuning a pre-trained U-Net model on an unlabeled dataset and a labeled dataset; the pre-trained U-Net model is obtained by training the U-Net model on a labeled dataset under a supervised learning mode; the unlabeled dataset only contains unlabeled medical images, and the labeled dataset contains both labeled medical images and true labels; During the fine-tuning process, the unlabeled medical image is input into the pre-trained U-Net model to generate a coarse segmentation probability map of the unlabeled medical image and use it as the first coarse segmentation probability map. The category with the largest probability value in the first coarse segmentation probability map is taken as the coarse segmentation pseudo-label of the unlabeled medical image and used as the first coarse segmentation pseudo-label. The prompt information required by the visual base model is generated based on the first coarse segmentation pseudo-label. The generated prompt information is passed through the prompt encoder of the visual base model to obtain the prompt feature, and the unlabeled medical image is processed through the encoder of the visual base model to obtain the image feature of the unlabeled medical image. Subsequently, the image feature and prompt feature of the unlabeled medical image are input into the mask decoder of the visual base model. The mask decoder generates a set of SAM foreground images, calculates the intersection-over-union ratio between the SAM foreground image and the first coarse segmentation pseudo-label of each category, and takes the average of the calculated results of the intersection-over-union ratio to obtain the consistency score, which is sorted in descending order of the consistency score. The unlabeled medical images are classified and labeled, and the unlabeled medical images marked as reliable are processed through the joint judgment of SAM and U-Net and the foreground conflict processing strategy to generate the first fine segmentation pseudo label. The reliable unlabeled medical images and the first fine segmentation pseudo label constitute a reliable data set, and the unreliable unlabeled medical images marked as unreliable constitute an unreliable data set. The unreliable data set is processed using the region exchange strategy, and a bidirectional displacement operation is performed between the labeled medical images and the unreliable unlabeled medical images to generate synthetic images and synthetic labels as supervision in the fine-tuning process. A perturbation operation is performed on each unlabeled medical image, and a weakly perturbed image and two independent strongly perturbed images are generated for each unlabeled medical image. The pre-trained U-Net model generates corresponding probability maps for the weakly perturbed image and each strongly perturbed image. The probability map of the weakly perturbed image is used as the supervision signal to perform supervised learning on the probability maps of the two strongly perturbed images.

2. The medical image segmentation method based on basic model-assisted semi-supervised learning according to claim 1, characterized in that: The network structure of the U-Net model is symmetrical and is divided into a downsampling path and an upsampling path; In the downsampling path, the input medical image is first processed by the first convolution layer to generate a first feature map, the first feature map is downsampled by the maximum pooling layer to obtain a low-resolution first feature map, then the low-resolution first feature map is processed by the second convolution layer in sequence to generate a second feature map, the second feature map is downsampled again by the maximum pooling layer to obtain a low-resolution second feature map, the low-resolution second feature map is processed by the third convolution layer to generate a third feature map, the third feature map is processed by the maximum pooling operation to generate a low-resolution third feature map, the low-resolution third feature map is processed by the fourth convolution layer to generate a fourth feature map, the fourth feature map is processed by the maximum pooling operation to obtain a low-resolution fourth feature map, finally, the low-resolution fourth feature map is processed by the fifth convolution layer to generate the fifth feature map and used as the input of the upsampling path; In the upsampling path, the fifth feature map is upsampled and spliced ​​with the fourth feature map in the downsampling path. After splicing, it is processed by the sixth convolution layer to generate the sixth feature map. After upsampling, the sixth feature map is spliced ​​with the third feature map in the downsampling path. After splicing, it is processed by the seventh convolution layer to generate the seventh feature map. After upsampling, the seventh feature map is spliced ​​with the second feature map in the downsampling path. After splicing, it is processed by the eighth convolution layer to generate the eighth feature map. After upsampling, the eighth feature map is spliced ​​with the first feature map in the downsampling path. After splicing, it is processed by the ninth convolution layer to generate the ninth feature map. Finally, the ninth feature map is processed by an output convolution layer, and the number of output channels corresponds to the number of target categories.

3. The medical image segmentation method based on semi-supervised learning assisted by a basic model according to claim 2, characterized in that: The convolution kernel size of the U-Net model is configured as follows: In the downsampling path, all convolutional layers use a 3×3 convolution kernel size to gradually extract features, and a 2×2 maximum pooling layer is used to reduce the resolution of the feature map and generate a deeper feature map; In the upsampling path, the convolutional layers after splicing all use a 3×3 convolution kernel size to further process the spliced ​​feature maps after each upsampling operation, thereby restoring the resolution of the medical image and extracting richer feature information; Upsampling is achieved by using a transposed convolution with a convolution kernel size of 2×2 to increase the resolution of the feature map. The convolution kernel size of the output convolution layer is 1×1, which is used to convert the final feature map into the number of channels corresponding to the number of target categories to generate the final segmentation result.

4. The medical image segmentation method based on basic model-assisted semi-supervised learning according to claim 1, characterized in that: The specific process for generating the cue information required by the visual base model is as follows: in the first coarse segmentation pseudo-label, only the largest continuous region in the non-empty foreground mask of each category excluding the background is always retained. After obtaining the continuous region, cue information is preset. The cue information is sparse cue and / or dense cue. The sparse cue is a point and / or a bounding box, and the dense cue is a category-level foreground mask captured from the first coarse segmentation pseudo-label. If the sparse cue is a border, the minimum bounding rectangle of the continuous region is used as the border representing the sparse cue; If the sparse hint is a point, it is generated in an iterative manner. At each iteration, a point is sampled from the unlabeled medical image until the iteration ends, and all the sampled points are used as a set of points representing the sparse hint.

5. The medical image segmentation method based on basic model-assisted semi-supervised learning according to claim 1, characterized in that: The specific process of classifying and labeling unlabeled medical images in descending order of consistency scores is as follows: the w unlabeled medical images with the lowest consistency scores are regarded as bottom medical images, the unlabeled medical images with consistency scores not lower than a preset threshold are marked as reliable, the unlabeled medical images with consistency scores lower than the preset threshold and belonging to the bottom medical images are marked as unreliable, and the remaining unlabeled medical images are marked as neutral and no additional processing is performed.

6. The medical image segmentation method based on basic model-assisted semi-supervised learning according to claim 1, characterized in that: The foreground conflict processing strategy is used to process the unlabeled medical image marked as reliable to generate the first fine segmentation pseudo-label. The specific process is as follows: the all-zero matrix is ​​used as the initial state of the first fine segmentation pseudo-label. When a pixel in the unlabeled medical image marked as reliable is marked as foreground by only one category mask k of the SAM foreground map, the pixel is assigned to the category in the category mask, and the value of the point with the same position as the pixel in the all-zero matrix is ​​changed to k. When a pixel in the unlabeled medical image marked as reliable is simultaneously marked as foreground by multiple category masks of the SAM foreground map, the pixel is assigned to the corresponding category k in the first coarse segmentation pseudo-label, and the value of the point with the same position as the pixel in the all-zero matrix is ​​changed to k, until all pixels in the unlabeled medical image marked as reliable are traversed to obtain the complete first fine segmentation pseudo-label.

7. The medical image segmentation method based on basic model-assisted semi-supervised learning according to claim 1, characterized in that: The region exchange strategy is used to process the unreliable dataset to generate synthetic images and synthetic labels. The specific process is as follows: for an unreliable unlabeled medical image, a labeled medical image is randomly selected, and a binary mask with a value of 1 in the foreground area and 0 in the rest is pre-constructed. The size of the foreground area of ​​the binary mask is smaller than that of the unreliable unlabeled medical image. All pixels in the foreground area of ​​the binary mask in the unreliable unlabeled medical image are exchanged with all pixels in the foreground area of ​​the binary mask in the randomly selected labeled medical image. After the exchange, two synthetic images are generated. Each synthetic image is processed separately. Through the U-Net model, each synthetic image corresponds to a pseudo-label probability map of the synthetic image, and the unreliable unlabeled medical image marked as unreliable is passed through the U-Net model, and the unreliable unlabeled medical image marked as unreliable is corresponding to a pseudo-label probability map and used as the unreliable pseudo-label probability map. The maximum connected area of ​​each category in the unreliable pseudo-label probability map is extracted, and a new fine segmentation pseudo-label is generated and used as the second fine segmentation pseudo-label. All pixels in the second fine segmentation pseudo-label belonging to the binary mask foreground area are exchanged with all pixels in the binary mask foreground area in the true label of the labeled medical image, and a set of synthetic labels are generated after the exchange.

8. The medical image segmentation method based on basic model-assisted semi-supervised learning according to claim 1, characterized in that: The specific process of performing a perturbation operation on each unlabeled medical image to generate a weakly perturbed image and a strongly perturbed image is as follows: for each unlabeled medical image, a weak perturbation operation is first performed to obtain a weakly perturbated image, and then two different strong perturbation operations are performed on the weakly perturbated image to generate two strongly perturbated images; wherein, the weak perturbation operation is to crop, rotate and / or flip the unlabeled medical image; and the strong perturbation operation is to Gaussian blur, color dithering and / or spatial deformation of the weakly perturbated image.

9. The medical image segmentation method based on basic model-assisted semi-supervised learning according to claim 7, characterized in that: The loss function used in fine-tuning the pre-trained U-Net model is By label loss Reliable data loss Unlabeled loss and unreliable data loss The weighted sum is obtained, and the specific form is as follows: λ(t)=μ·exp(-ε·(1-t / t total ) 2 ) Among them, α, β, γ, μ, and ε are all weight hyperparameters; λ(t) represents the loss weight at the tth iteration; t total Indicates the total number of iterations; represents the first cross entropy loss; represents the first Dice loss; P l Y represents the probability map generated by the U-Net model for labeled medical images; l Indicates the true label corresponding to the labeled medical image; represents the second cross entropy loss; represents the second Dice loss; P r Represents the probability map generated by the U-Net model for unlabeled medical images marked as reliable; represents the fine segmentation pseudo-label corresponding to the unlabeled medical image marked as reliable; τ represents the confidence threshold; P w Represents the probability map generated by the weak perturbation image through the U-Net model; P s1 ,P s2 Represents the probability map generated by the U-Net model for a strongly perturbed image; represents the third Dice loss; Indicates the fourth Dice loss; and represents auxiliary variables; M represents the binary mask constructed in the region swap strategy; ⊙ represents the element-wise dot product; Indicates the fifth Dice loss; represents the sixth Dice loss; ψ represents the control parameter; P dm ,P lm Both represent pseudo label probability maps generated by the U-Net model for synthetic images; Y dm ,Y lm All represent synthetic tags.

10. A medical image segmentation system based on basic model-assisted semi-supervised learning, characterized in that: include: An image acquisition module, used for acquiring medical images to be segmented; a result acquisition module, configured to input the medical image to be segmented into a trained medical image segmentation model to obtain a segmentation result of the medical image; wherein the medical image segmentation model is obtained by fine-tuning a pre-trained U-Net model on an unlabeled dataset and a labeled dataset; the pre-trained U-Net model is obtained by training the U-Net model on a labeled dataset in a supervised learning mode; the unlabeled dataset contains only unlabeled medical images, and the labeled dataset contains both labeled medical images and true labels; During the fine-tuning process, the unlabeled medical image is input into the pre-trained U-Net model to generate a coarse segmentation probability map of the unlabeled medical image and use it as the first coarse segmentation probability map. The category with the largest probability value in the first coarse segmentation probability map is taken as the coarse segmentation pseudo-label of the unlabeled medical image and used as the first coarse segmentation pseudo-label. The prompt information required by the visual base model is generated based on the first coarse segmentation pseudo-label. The generated prompt information is passed through the prompt encoder of the visual base model to obtain the prompt feature, and the unlabeled medical image is processed through the encoder of the visual base model to obtain the image feature of the unlabeled medical image. Subsequently, the image feature and prompt feature of the unlabeled medical image are input into the mask decoder of the visual base model. The mask decoder generates a set of SAM foreground images, calculates the intersection-over-union ratio between the SAM foreground image and the first coarse segmentation pseudo-label of each category, and takes the average of the calculated results of the intersection-over-union ratio to obtain the consistency score, which is sorted in descending order of the consistency score. The unlabeled medical images are classified and labeled, and the unlabeled medical images marked as reliable are processed through the joint judgment of SAM and U-Net and the foreground conflict processing strategy to generate the first fine segmentation pseudo label. The reliable unlabeled medical images and the first fine segmentation pseudo label constitute a reliable data set, and the unreliable unlabeled medical images marked as unreliable constitute an unreliable data set. The unreliable data set is processed using the region exchange strategy, and a bidirectional displacement operation is performed between the labeled medical images and the unreliable unlabeled medical images to generate synthetic images and synthetic labels as supervision in the fine-tuning process. A perturbation operation is performed on each unlabeled medical image, and a weakly perturbed image and two independent strongly perturbed images are generated for each unlabeled medical image. The pre-trained U-Net model generates corresponding probability maps for the weakly perturbed image and each strongly perturbed image. The probability map of the weakly perturbed image is used as the supervision signal to perform supervised learning on the probability maps of the two strongly perturbed images.

Images