An anti-knowledge forgetting mutual guidance type semi-supervised medical image segmentation method

Abstract

The application discloses an anti-knowledge forgetting mutual guidance type semi-supervised medical image segmentation method, first, a double network model composed of two initialized different networks is constructed, and the internal difference is used as an implicit disturbance source; secondly, a double-path data fusion strategy is designed, structured fusion of region blocks of labeled and unlabeled images is carried out, a forced model is generated, and mixed images of two data domains are simultaneously processed in a single iteration, so that forgetting of learned labeled knowledge in the semi-supervised learning process is effectively inhibited; then, a mutual guidance learning mechanism is adopted, the prediction result of one network is used to generate high-quality pseudo labels, and a unified mixed supervision signal is constructed for the mixed images; finally, the model is trained by jointly optimizing a loss function customized for the mixed samples. The application is suitable for two-dimensional and three-dimensional image segmentation tasks, can significantly improve segmentation performance and robustness in a small amount of labeled data scene, and can enhance the universality and ease of use of the method.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and relates to a computer vision method based on deep learning, more particularly to an anti-knowledge forgetting mutual guidance type semi-supervised medical image segmentation method, which is particularly suitable for high-precision automatic segmentation of medical images in a scenario with scarce labeled data, and can support processing of two-dimensional (2D) and three-dimensional (3D) image data. BACKGROUND

[0002] In recent years, with the rapid development of computing power and the increasing maturity of deep learning theory, deep learning models represented by convolutional neural networks have made revolutionary breakthroughs in the field of computer vision. In the key branch of medical image analysis, image segmentation technology, which identifies and outlines the anatomical structures or lesion regions of interest at the pixel (or voxel) level, has crucial clinical significance for the quantitative diagnosis, treatment planning, surgical navigation, and prognosis evaluation of diseases. For example, in the diagnosis of heart diseases, accurate segmentation of the ventricle and myocardium is the basis for calculating key functional indicators such as ejection fraction and ventricular volume; in oncology, accurate segmentation of tumor regions is a prerequisite for evaluating their size, growth rate, and developing radiotherapy plans.

[0003] Fully supervised deep learning segmentation models represented by U-Net and its variants have been able to achieve or even surpass the level of human experts in some tasks by training on large-scale, high-quality pixel-level labeled datasets. The success of these models essentially learns the complex mapping relationship from the input image to its corresponding segmentation map (i.e., label). However, this successful paradigm has an extremely stringent prerequisite: the need for a large amount of "gold standard" labeled data meticulously hand-drawn by domain experts (such as experienced radiologists) pixel by pixel or voxel by voxel

[0004] In the real clinical workflow, obtaining such large-scale, high-quality labeled datasets faces a series of severe challenges. First, the labeling process itself is a labor-intensive task that is extremely time-consuming and labor-intensive. A doctor may need to spend tens of minutes or even hours to complete the complete labeling of a three-dimensional medical image (such as CT or MR). Second, the labeling cost is high, including not only the cost of doctors' manpower, but also the cost of data management and quality control. Third, artificial labeling inevitably introduces subjectivity. Different experts, or even the same expert at different times, may have different judgments on ambiguous tissue boundaries or complex lesion morphology. This inter-observer variability introduces noise and uncertainty into the model training, affecting the stability and reliability of its final performance.

[0005] Due to the existence of the above-mentioned "data bottleneck", the application range and promotion speed of the full-supervised learning method are greatly limited. In many clinical research or application scenarios, there are often a large amount of original medical image data that is easy to obtain, but only a small part of the data is finely annotated by experts. How to effectively utilize these massive unlabeled data to assist and enhance the model training on a small amount of labeled data has become a core problem that needs to be solved in the field.

[0006] Semi-supervised learning emerged in this context, and its core goal is to learn a model that performs better than the model trained only on labeled data, given both a small amount of labeled data and a large amount of unlabeled data. In the field of image segmentation, the consistency regularization-based method is the mainstream paradigm of current semi-supervised learning. This paradigm is based on a smoothness assumption: for a small perturbation of a sample in the input space, the prediction results of a good-performing model should remain stable and consistent. In other words, if different, but semantically preserved data augmentations are performed on the same image, the model's predictions on these augmented images should be similar to each other.

[0007] However, although the semi-supervised segmentation method based on consistency regularization is theoretically reasonable and has made some progress in practice, the existing technology generally faces two insurmountable technical bottlenecks, which limit the further improvement of its performance:

[0008] First, the "label knowledge forgetting" problem. Most existing semi-supervised methods, such as Mean Teacher and UAMT, usually adopt a supervised and unsupervised task separation learning paradigm. When dealing with labeled data, the model is dominated by the standard segmentation loss; while dealing with a large number of unlabeled data, it is dominated by the consistency loss. This separation leads the model to pay excessive attention to maintaining the internal consistency of the prediction on the unlabeled data, and gradually "forget" the discriminative knowledge learned from the small amount of labeled data about the accurate object boundary and subtle structure. This phenomenon eventually leads to model performance drift, failing to fully utilize the valuable supervision signal provided by the gold standard data.

[0009] Second, the perturbation strategy is complex and relies on manual design. Existing consistency learning methods highly depend on external perturbations at the data level, i.e., creating different "views" for samples by designing complex data augmentation pipelines. However, there is no universal optimal data augmentation strategy, which is strongly related to specific tasks and data characteristics. This forces researchers to invest a lot of effort in tedious parameter tuning experiments, greatly damaging the generality, reproducibility and transferability of the method, increasing the technical threshold and cost of practical application.

[0010] In summary, in order to promote the substantial development of semi-supervised medical image segmentation technology, it is urgently needed to develop a new learning framework. The framework needs to solve the problem of“label knowledge forgetting”mechanically, and at the same time, get rid of the dependence on complex manual data augmentation strategy, and seek a more endogenous and more adaptive disturbance mode. The present application is just proposed to cope with these technical challenges. SUMMARY

[0011] The main purpose of the present application is to overcome the two technical bottlenecks of“label knowledge forgetting”and“disturbance strategy complexity”in the existing semi-supervised image segmentation technology, and provide a new method of anti-knowledge forgetting mutual guidance type semi-supervised medical image segmentation. The present application aims to effectively prevent the model from forgetting the labeled knowledge when learning the unlabeled data through a unified learning framework, and use the internal difference of the network as an adaptive disturbance source, so as to significantly improve the accuracy, robustness and universality of image segmentation under the condition of only a small amount of labeled data.

[0012] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0013] (1) Dual-network collaborative model and implicit disturbance mechanism: a collaborative model composed of two segmentation networks with the same structure but different weight initialization is constructed. The model state difference caused by the internal randomness of the two networks in the training process is ingeniously used as an implicit and adaptive network-level disturbance source, which fundamentally replaces the dependence on external manual data augmentation strategy.

[0014] (2) Dual-path data fusion strategy to solve“knowledge forgetting”: the present application proposes an innovative dual-path data fusion strategy, the fundamental purpose of which is to break the processing barrier of labeled and unlabeled data in training iteration. By“transplanting”a region block of unlabeled image onto a labeled image, a hybrid image containing both data distribution characteristics in space is created. This design is the key to the present application to resist the problem of“label knowledge forgetting”, which forces the model to review the learned accurate knowledge (from the real label region) and explore new unknown information (from the pseudo label region) at the same time in a single parameter update, thereby ensuring the continuous consolidation of labeled knowledge in mechanism.

[0015] (3) Mutual guidance learning and unified hybrid supervision: a mutual guidance learning mechanism based on dual-network model is established. The mechanism includes a pseudo label purification step to generate high-quality pseudo labels, thereby ensuring the stability of mutual guidance learning. Finally, a unified hybrid supervision signal composed of real labels and high-quality pseudo labels is constructed for the above hybrid image, and the model is trained by jointly optimizing a loss function customized for the hybrid sample.

[0016] The specific technical scheme of the present application can be described as:

[0017] A mutual guidance type semi-supervised medical image segmentation method against knowledge forgetting, comprising the following steps:

[0018] (a) providing a labeled image dataset and an unlabeled image dataset;

[0019] (b) establishing a double network model comprising a first segmentation network and a second segmentation network, wherein the network weight parameters of the first segmentation network and the second segmentation network are set to different initial values;

[0020] (c) performing a data fusion step, obtaining a first image from the labeled image dataset and a second image from the unlabeled image dataset; according to a preset content mask, combining the region derived from the second image with the region derived from the first image to form a mixed image aiming to force the double network model to simultaneously process the labeled and unlabeled data distribution characteristics in a single iteration;

[0021] (d) performing a model training step to train the double network model; the training step comprises:

[0022] (d1) using the first segmentation network to predict the second image to generate a first pseudo label;

[0023] (d2) constructing a hybrid supervision signal composed of the real label corresponding to the first image and the first pseudo label, and using the hybrid supervision signal to supervise the prediction result of the double network model on the mixed image;

[0024] (e) repeating the model training step according to a preset loss function until the double network model converges, and using the obtained model to perform medical image segmentation.

[0025] In one embodiment of the present application, after generating the first pseudo label in step (d1), a connected component analysis step is further performed on the first pseudo label, and the largest connected region is selected to form a structurally purified pseudo label, which is used to provide a stable and reliable supervision signal in the mutual guidance learning process of the double network model.

[0026] In one embodiment of the present application, before step (a), a pre-processing step is further included for the labeled image dataset and the unlabeled image dataset, which comprises at least one of the following steps: resampling the image data to a unified voxel spacing; normalizing the voxel intensity values of the image; and performing data enhancement on the image, wherein the data enhancement includes random rotation, random flipping or random scaling.

[0027] One embodiment of the present application, before step (c) obtaining the first image and the second image, further comprises a double-flow batch sampling step for forming a mixed batch containing labeled data and unlabeled data, the double-flow batch sampling step comprises:

[0028] providing sample index lists for the labeled image data set and the unlabeled image data set respectively, and presetting a total batch size and the number of labeled samples in the batch;

[0029] When generating each training batch, according to the preset number, randomly sampling from the labeled sample index list and the unlabeled sample index list respectively to obtain two groups of sample indexes;

[0030] merge the two groups of sample indexes to form the mixed batch, so as to ensure that the dual network model is provided with labeled and unlabeled data at the same time in a single iteration.

[0031] One embodiment of the present application, in step (c), the content mask is a binary mask configured to extract a central content region from the second image and embed the central content region into the first image to generate the mixed image.

[0032] One embodiment of the present application, the data fusion step (c) specifically comprises:

[0033] (c1) dividing the labeled images in a batch into a first labeled sub-batch and a second labeled sub-batch, and dividing the unlabeled images in the batch into a first unlabeled sub-batch and a second unlabeled sub-batch;

[0034] (c2) generating a first mixed image by randomly generating a first binary content mask batch and combining background regions from the first unlabeled sub-batch with content regions from the first labeled sub-batch based on the mask;

[0035] (c3) generating a second mixed image by randomly generating a second binary content mask batch and combining background regions from the second labeled sub-batch with content regions from the second unlabeled sub-batch based on the mask.

[0036] One embodiment of the present application, the generation method of the first binary content mask and the second binary content mask batch comprises:

[0037] independently and randomly generating a rectangular mask for each group of images to be combined;

[0038] The size of the rectangular mask is set as a preset proportion of the image size, and the position of the rectangular mask in the image is determined by random sampling.

[0039] In one embodiment of the present application, in step (d2), the mixed supervision signal is constructed by:

[0040] The content mask is used to take the real label of the first image as the supervision signal of the background region, and take the pseudo label generated by the second network after predicting the second image as the supervision signal of the central content region, and then the mixed supervision signal is spliced.

[0041] In one embodiment of the present application, the dual-network model is trained by the following mutual guidance method:

[0042] Step 11, the second segmentation network is used to predict the first mixed image and the second mixed image respectively;

[0043] Step 12, the step of supervising the first segmentation network comprises:

[0044] Step 121, the second segmentation network is used to predict the images from the first unlabelled sub-batch to generate a first pseudo label, and predict the images from the second unlabelled sub-batch to generate a second pseudo label;

[0045] Step 122, the first mixed supervision signal is constructed by combining the content region of the real label from the first labelled sub-batch and the background region of the first pseudo label, and the prediction result of the first mixed image by the first segmentation network is supervised by the first mixed supervision signal;

[0046] Step 123, the second mixed supervision signal is constructed by combining the background region of the real label from the second labelled sub-batch and the content region of the second pseudo label, and the prediction result of the second mixed image by the first segmentation network is supervised by the second mixed supervision signal.

[0047] In one embodiment of the present application, in step (d), the model training step further comprises: predicting the second image by the second segmentation network to generate a second pseudo label, and using the second pseudo label to supervise the first segmentation network. The method can be described as training the dual-network model by the following mutual guidance method:

[0048] Step 21, the first segmentation network is used to predict the first mixed image and the second mixed image respectively;

[0049] Step 22, the step of supervising the second segmentation network comprises:

[0050] Step 221, using the first segmentation network to predict the images derived from the first unlabeled sub-batch to generate the first pseudo label, and to predict the images derived from the second unlabeled sub-batch to generate the second pseudo label;

[0051] Step 222, by combining the content area derived from the first labeled sub-batch real label with the background area of the first pseudo label, a first mixed supervision signal is constructed, and the prediction result of the second segmentation network for the first mixed image is supervised by the first mixed supervision signal;

[0052] Step 223, by combining the background area derived from the second labeled sub-batch real label with the content area of the second pseudo label, a second mixed supervision signal is constructed, and the prediction result of the second segmentation network for the second mixed image is supervised by the second mixed supervision signal.

[0053] In one embodiment of the present application, in step (e), the preset loss function is a total objective function defined as the weighted sum of a supervised loss and an unsupervised consistency loss; wherein the supervised loss includes cross-entropy loss calculated based on the mask respectively, and the unsupervised consistency loss includes Dice loss calculated on the whole mixed image.

[0054] In one embodiment of the present application, in step (e), the model training adopts a stochastic gradient descent optimizer, and the learning rate update strategy adopts a polynomial decay strategy or a combination strategy containing a platform period and a cosine annealing period.

[0055] Compared with the prior art, the technical scheme proposed in the present application solves the problem of knowledge forgetting in semi-supervised training through the organic combination of the above innovative points, and ensures that good image segmentation results can be achieved by using a small amount of samples for training. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiment description will be briefly introduced as follows.

[0057] Figure 1 is the overall flowchart of the semi-supervised image segmentation method proposed in one preferred embodiment of the present application, which shows the main stages from data preparation to model inference.

[0058] Figure 2 is the detailed structure diagram of the anti-knowledge forgetting mutual guidance type semi-supervised medical image segmentation method framework proposed in one preferred embodiment of the present application, which illustrates the data flow, supervision signal flow and interaction relationship between networks.

[0059] Figure 3is the embodiment effect verification part of the application, and is a visualization comparison chart of segmentation results of the method of the application and various existing advanced methods on a 5% labeled ACDC data set in the form of an image.

[0060] Figure 4 is the embodiment effect verification part of the application, and is a visualization comparison chart of segmentation results of the method of the application and various existing advanced methods on a 10% labeled ACDC data set in the form of an image.

[0061] Figure 5 is the embodiment effect verification part of the application, and is a visualization comparison chart of segmentation results of the method of the application and various existing advanced methods on a 5% labeled LA data set in the form of an image.

[0062] Figure 6 is the embodiment effect verification part of the application, and is a visualization comparison chart of segmentation results of the method of the application and various existing advanced methods on a 10% labeled LA data set in the form of an image. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical scheme and advantages of the application more clear, complete, the technical scheme of the application will be further described in detail below in combination with the drawings and specific embodiments.

[0064] In order to make the purpose, technical scheme and advantages of the application more clear, complete, the technical scheme of the application will be further described in detail below in combination with the drawings and specific embodiments.

[0065] The application can be applied to two-dimensional and three-dimensional image segmentation tasks, significantly improves the segmentation performance and robustness in the scene of a small amount of labeled data, and enhances the universality and ease of use of the method. The following are several specific embodiments of the application.

[0066] Embodiment one: an implementation mode applied to two-dimensional (2D) cardiac MR image segmentation

[0067] This embodiment will take a specific clinical application scenario as the background, that is, under the condition of only a small amount of annotation, the high-precision automatic segmentation of two-dimensional cardiac magnetic resonance image slices, to elaborate each technical detail and implementation step of the method of the application.

[0068] Reference Figure 1 The overall implementation process of the embodiment mainly includes data preparation and loading (step 1-1), double network model construction (step 1-2), core training loop (steps 1-3, 1-4, 1-5), and final model inference application (step 1-6) and the like main stages.

[0069] Step 1-1: Data preparation and loading

[0070] The core task of this step is to establish an efficient and standardized data processing pipeline to provide high-quality and format-unified data input for subsequent model training.

[0071] (1) Data acquisition and division: This embodiment uses the internationally published ACDC (Automated Cardiac Diagnosis Challenge) dataset as the experimental data source. The dataset contains cardiac MR movie sequences (Cine-MR) of 100 subjects, and the segmentation labels of left ventricle (LV), right ventricle (RV) and myocardium (MYO) three anatomical structures manually outlined by clinical experts as the gold standard. In order to simulate the semi-supervised scene of a small amount of annotation commonly seen in clinical practice, this embodiment strictly divides the dataset according to the subject ID. For example, under a 10% annotation rate experimental setting, 7 subjects' data (in the ACDC dataset, this corresponds to 136 2D image slices) are randomly selected from 70 training samples as the labeled training set where N L is the number of labeled samples. The data of the remaining 63 subjects is used as the unlabeled training set where N U is the number of unlabeled samples. Another 20 subjects' data is used as the validation set, and 10 subjects' data is used as the final test set for model selection and objective evaluation of the final performance.

[0072] (2) Data preprocessing and enhancement: In order to eliminate the data heterogeneity brought by different scanning devices, scanning parameters and patient posture and the like factors, and improve the robustness and generalization ability of the model, this embodiment performs a series of standardized preprocessing and data enhancement operations on all obtained image data. In a preferred embodiment, this process is encapsulated in a data conversion class named random data augmentation generator, and is applied to each sample of data loading. The class can include one or more of the following operation modules:

[0073] Random Geometric Transformation Module: This module aims to improve the model's robustness to changes in the orientation and position of the target object. It performs a random combination of geometric transformations on the input image and its corresponding label image with a preset probability of 0.5. This combination may include: first, randomly selecting an angle from a set of preset rotation angles {0, 90, 180, 270} for image rotation; then, randomly selecting one from a set of preset flipping methods {horizontal flip, vertical flip} for operation. Alternatively, it may perform a small random angle rotation on the image and label within a small continuous range (-20 degrees to +20 degrees) with another preset probability of 0.25.

[0074] Size normalization module: This module is responsible for scaling the images and labels, which may have varied sizes after the aforementioned geometric transformations, to a fixed size suitable for network input using an interpolation algorithm. For example, all 2D image slices can be uniformly scaled to 256×256 pixels. In a preferred implementation, bilinear interpolation is used for image scaling to ensure the smoothness and continuity of image content. However, to maintain the discreteness and accuracy of the category indexes in the label image, nearest neighbor interpolation must be used for scaling the label image.

[0075] (3) Construction of the Dual-Stream Batch Sampler: To achieve the core idea of ​​this invention, namely, to stably and proportionally mix labeled and unlabeled data during training, this embodiment employs a specialized batch sampling strategy, specifically implemented as a sampler class named Dual-Stream Batch Sampler. The working principle of this sampler is as follows:

[0076] Initialization: During the initialization phase, the sampler receives two separate lists of indices: one for labeled samples within the entire dataset and another for unlabeled samples. It also receives two key parameters: the total batch size N. B And the number of labeled samples that should be included in each batch. Number of unlabeled samples Then, based on the total batch size N B Subtract the number of labeled samples The calculation yielded the result.

[0077] Iterative Batch Generation: At the start of each training round, the sampler internally creates two iterators: one for labeled indices (which are initially randomly shuffled), and the other for unlabeled indices (designed to loop infinitely and be randomly shuffled each time). During each training batch generation, it strictly adheres to a preset number of indices, extracting a specified number from both the labeled and unlabeled iterators. Then, it merges these two sets of indices to form a complete mixed batch index list containing both data sources, which is then provided to the data loader.

[0078] The double-flow sampling mechanism of the present application ensures that the gradient calculated by the model at each parameter update is derived from both supervised and unsupervised signals, which is the basis for implementing the knowledge preservation mechanism described in the present application, and ensures the unity of learning from the source of data input.

[0079] Step 1-2: Double-network segmentation model construction

[0080] With reference to Figure 2 , the present application constructs a double-network collaborative model composed of two parallel segmentation networks to realize mutual guidance learning.

[0081] (1) Network infrastructure: In a preferred embodiment, both segmentation networks F A and F B in the double-network model adopt the U-Net architecture. U-Net is a classic encoder-decoder structure tailored for medical image segmentation tasks. The encoder part, i.e., the feature extraction path, is usually composed of four or five consecutive down-sampling modules. Each down-sampling module contains two or more convolution layers (ConvBlock) and a maximum pooling layer with a step size of 2. ConvBlock is preferably composed of a 3x3 two-dimensional convolution layer, a batch normalization layer, and an activation function in series. As the network depth increases, the output feature map size of each down-sampling module is halved, while the number of channels is doubled. The number of feature channels of the encoder is set to [16, 32, 64, 128, 256] in turn. The decoder part of the encoder, i.e., the up-sampling path, forms a symmetrical structure with the encoder. Each up-sampling module increases the resolution of the feature map by a factor of two through bilinear interpolation with a step size of 2, and then concatenates the feature map of the corresponding layer of the encoder in the channel dimension. This design of skip connection is one of the cores of U-Net, which allows the decoder to recover spatial details while utilizing the low-level features extracted by the encoder in the early stage, which are rich in high-level semantic information, thereby achieving accurate positioning of the target boundary. The concatenated features are then fused and refined by a ConvBlock. Finally, the last layer of the decoder maps the number of channels of the feature map to the final number of classes through a 1x1 or 3x3 convolution layer, outputting a pixel-level segmentation prediction result.

[0082] (2) Parameter initialization: In order to make the two networks constitute an effective "function view" that disturbs each other, the weight parameters θ A of network F A and the weight parameters θ B of network F BDifferent random initializations are performed. In practice, this can be achieved by setting different random number seeds for the initialization process of the two models. This simple yet crucial operation ensures that even at the very first step of training, two structurally identical networks will produce tiny but non-zero prediction differences for the same input image. This difference persists throughout the training, under the influence of random factors such as stochastic gradient descent, Dropout, etc., thus providing a constant, endogenous, self-adaptive network-level perturbation for mutual guidance learning, avoiding the complexity of introducing external data augmentation.

[0083] Step 1-3: Execution of the dual-path data fusion strategy

[0084] This step is the core technical means to achieve the "anti-knowledge forgetting" goal of the present application. The root cause of "label knowledge forgetting" in the traditional semi-supervised learning paradigm lies in the separation of training iterations on labeled and unlabeled data. The dual-path data fusion strategy designed by the present application is to fundamentally break this separation. In each mixed batch provided by the data loader in step 1-1, the mixed image samples capable of effectively resisting knowledge forgetting are constructed through the following data internal fusion operations.

[0085] (1) Batch subdivision: a mixed batch containing labeled samples and unlabeled samples is further logically subdivided into four sub-batches, and the labeled batch is divided into the first half (X l1 , Y l1 ) and the second half (X l2 , Y l2 ), and the unlabeled batch is also divided into the first half X u1 and the second half X u2 .

[0086] (2) Content mask generation: this embodiment dynamically generates a content mask through a mask generator. This function independently and randomly generates a rectangular mask for each image in the input image sub-batch. In a preferred setting, the width and height of the rectangle are set to a fixed proportion of 2 / 3 of the size of the input image. The top-left corner coordinates (w, h) of the rectangle are uniformly randomly sampled from all possible positions to ensure sufficient randomness in each fusion position. The generated mask M is a binary tensor with the same size as the image, where the pixel values in the rectangular region are 1, representing the "to-be-pasted region" or "content region"; the pixel values outside the rectangular region are 0, representing the "background region".

[0087] (3) Mixed image generation: The present application constructs multiple data fusion flows by combining different sub-batches and masks. In the preferred embodiment, the following two symmetric mixed flows are constructed:

[0088] 1. First mixed flow: This flow aims to embed the content of the labeled image into the background of the unlabeled image. Its mathematical expression is:

[0089] X u←l = M1 o X l1 + (1 - M1) o X u1

[0090] where X l1 and X u1 represent image batch tensors, M1 is the first set of random masks generated for this fusion operation (1 outside, 0 inside), and o represents element-wise multiplication.

[0091] 2. Second mixed flow: This flow aims to embed the content of the unlabeled image into the background of the labeled image. Its mathematical expression is:

[0092] X l←u = (1 - M2) o X l2 + M2 o X u2

[0093] where M2 is the second set of random masks generated for this fusion operation.

[0094] In this way, the model is forced to process visual features from both labeled and unlabeled distributions in one forward propagation, thereby structurally enforcing knowledge retention and interaction, effectively combating knowledge forgetting.

[0095] Steps 1-4: High-quality pseudo-label generation and mutual guidance learning

[0096] This step is another core of the present application, aiming to provide high-quality and reliable supervision signals for unlabeled data and mixed data.

[0097] (1) Generation of high-quality pseudo-labels: One key innovation of the present application is that pseudo-labels are generated based on clean, original unlabeled images, rather than on mixed, information-disturbed images, and are purified after generation. For example, the required pseudo-labels for the guidance network F A are generated by the "companion" network F B .

[0098] a) Initial pseudo-label prediction: First, input the original unlabeled image sub-batches X u1 and X u2 to the network F B to obtain the predicted probability map and

[0099] b). Category decision: By performing argmax operation on the category dimension, i.e. selecting the category with the highest prediction probability for each pixel, the initial pseudo-labels are obtained and The mathematical expression of its construction method is:

[0100]

[0101] c). Pseudo-label post-processing: In order to significantly improve the quality of pseudo-labels and remove noise points or isolated small regions caused by model uncertainty, the present application introduces a maximum connected component post-processing step. This processing plays a role of non-maximum suppression. For each foreground category, it first extracts all pixels predicted as this category to form a binary image; then, it uses connected component analysis algorithm to find all independent connected regions; finally, it calculates the area of each connected region and only keeps the largest one, and all other small isolated regions are removed. After this step, the final high-quality pseudo-labels for supervision are obtained.

[0102] It is worth emphasizing that the maximum connected component (LCC) post-processing step, in the mutual guidance learning framework of the present application, is not just a regular denoising operation, but plays the role of a "quality control valve" to ensure the stability of the entire learning system. In mutual guidance learning, if the pseudo-labels generated by one network contain a large number of anatomically unreasonable isolated noise, these noise will be passed as incorrect supervision information to the other network, leading to the accumulation and amplification of errors between the two networks, and even causing training to collapse. By embedding the LCC purification step in the internal loop of mutual guidance learning, the present application effectively cuts off the chain of error propagation by using the strong domain prior that "anatomical structures are usually continuous and complete". It ensures that the pseudo-labels used for guidance in each iteration have high structural reliability. This purified high-quality pseudo-label, in cooperation with another core innovation of the present application "dual-path data fusion strategy", guarantees the accuracy of the final "mixed supervision signal" and is the key to the stable convergence and outstanding performance of the entire framework.

[0103] (2) Construction of mixed supervision signal: Using the above high-quality pseudo-labels, construct the spatially aligned mixed supervision signal for the mixed image generated in step 1-3. For example, for the network F A The predicted mixed label of the second mixed stream X l←u The mathematical expression of its construction method is:

[0104] ​

[0105] where Y l2 is the ground truth label of the sub-batch X l2 corresponding to the real label, is the high-quality pseudo label generated by the network F B and post-processed by LCC.

[0106] Step 1-5: Model optimization and joint loss function

[0107] (1) Joint loss function: A composite loss function is used to evaluate and optimize the model's predictions. This function finely combines the Dice loss and cross-entropy loss, and can handle mixed supervision signals.

[0108] Cross-entropy loss: For a mixed input, its corresponding supervision signal is also mixed. The calculation of cross-entropy loss is based on region division by mask. For the region in the model prediction corresponding to the real label, calculate the cross-entropy loss between it and the real label, and multiply it by a higher supervised weight. For the region corresponding to the pseudo label, calculate the cross-entropy loss between it and the high-quality pseudo label, and multiply it by an unsupervised weight u weight . The value of u weight will gradually increase from 0 to a preset value through a heating function as the training progresses, which realizes a curriculum learning from easy to difficult.

[0109] Dice loss: Unlike cross-entropy loss, Dice loss pays more attention to the degree of overlap of regions. In a preferred embodiment, the Dice loss is calculated on the entire mixed image, i.e. the overall Dice similarity between the model's prediction of the entire mixed image and the complete mixed supervision signal composed of real labels and pseudo labels.

[0110] Overall loss: For a single network (e.g. F A ), its total loss is the average of the losses after processing all mixed streams. The final loss of the entire framework is the sum of the total losses of the two networks:

[0111] L total = L A + L B

[0112] (2) Optimizer and training process:

[0113] Optimizer: Momentum stochastic gradient descent is used as the optimizer, with the momentum parameter preferably set to 0.9 and the weight decay set to 0.0001.

[0114] Learning rate strategy: The initial learning rate is set to 0.01. During the training process, the learning rate is dynamically adjusted using a polynomial decay strategy, and its update rule can be expressed as:

[0115]

[0116] where lr current is the learning rate of the current iteration, iter current is the current iteration number, iter max is the total number of preset training iterations, and P is the power of decay.

[0117] Training execution: In each training step, the aforementioned data loading, fusion, pseudo-label generation, forward propagation, and loss calculation are performed, followed by backward propagation and optimizer update. This process is repeated until the maximum number of iterations is reached.

[0118] Step 1-6: Model inference and application

[0119] After the training process is completed, the model weight with the highest comprehensive performance (average Dice coefficient) on the independent validation set is selected as the final, deployable segmentation model. In the inference stage, for any new image to be tested, it only needs to be input into the trained model, and through one forward propagation, a high-precision segmentation result can be obtained automatically and quickly. This result can be directly used for subsequent clinical quantitative analysis, three-dimensional reconstruction, or as input for other downstream tasks.

[0120] Example Two: Implementation for 3D Left Atrial MR Image Segmentation

[0121] The method of the present application is not limited to 2D images, and its core idea can be seamlessly extended to 3D image segmentation tasks. This embodiment will detail how to apply the present application to semi-supervised segmentation of 3D left atrial (LA) MR images.

[0122] Step 2-1: Data preparation and loading (3D)

[0123] (1) Data acquisition and division: This embodiment uses a LA dataset containing 100 3D left atrial MR images. Similarly, under the 10% annotation rate setting, the data of 8 subjects are randomly selected as the labeled training set DL, and the remaining 72 subjects are used as the unlabeled training set DU.

[0124] (2) Data preprocessing and enhancement: For 3D data, the preprocessing process is similar but the operation dimension is different.

[0125] Random cropping: Due to the larger volume of 3D images, random cropping is usually used to obtain training samples. From the complete 3D image, a fixed-size sub-volume or "block" is randomly cropped.

[0126] Random geometric transformation: Data augmentation operations are also extended to three-dimensional space, for example, random flipping on three axes, or small-angle rotation around an arbitrary axis.

[0127] (3) Data loader: The same double-flow batch sampler strategy is adopted, but this time it is loading 3D data blocks. The batch size can be adjusted according to the GPU memory.

[0128] Step 2-2: Dual-network segmentation model construction (3D)

[0129] For 3D data, the V-Net architecture is a better choice. V-Net is a successful extension of U-Net in the 3D field. Its core components are replaced with 3D versions:

[0130] Step 2-3, 2-4: Data fusion and mutual guidance (3D)

[0131] The core logic is exactly the same as the 2D embodiment, but all operations (such as mask generation, image blending) are performed on three-dimensional tensors. The LCC post-processing of pseudo-labels also uses a 3D connected component analysis algorithm accordingly.

[0132] Step 2-5: Model optimization and joint loss function

[0133] (1) Loss function: The definition and calculation method of the loss function remain consistent with the 2D embodiment, which is a combination of mask-based cross-entropy loss and global Dice loss.

[0134] (2) Learning rate strategy: In order to demonstrate the compatibility of the present invention to different optimization strategies, this embodiment adopts a combined learning rate strategy containing a platform period and a cosine annealing period. Platform period: In the initial stage of training, the learning rate remains at a relatively high constant value, which helps the model to quickly explore and learn in the early stage. Cosine annealing period: After the platform period ends, the learning rate starts to smooth decay according to the cosine function until it reaches a preset minimum learning rate. The mathematical expression is:

[0135]

[0136] where η max is the initial maximum learning rate (i.e. lr bases ), T cur is the current iteration step number in the annealing phase, and T max is the total step number in the annealing phase. This strategy takes into account both the fast convergence in the early stage and the fine tuning in the later stage.

[0137] Implementation effect verification

[0138] To fully and objectively prove the effectiveness and advancement of the technical solutions proposed in the present application, the present embodiment conducts a comprehensive performance comparison with a variety of current advanced semi-supervised segmentation methods on a standardized experimental platform.

[0139] Experimental platform and settings:

[0140] (1) Hardware environment: Intel(R) Xenon(R), 2.6 GHz CPU, 24 GB RAM, and 2 GeForce RTX3090 GPUs.

[0141] (2) Software environment: Ubuntu 20.04, Python 3.9, PyTorch 1.12, and CUDA 11.6.

[0142] (3) Comparison methods: The present application method (denoted as Ours) is compared with a variety of current advanced semi-supervised segmentation methods, including BCP (2023), CrossMatch (2024), ABD (2024), MPER (2025), etc. In addition, two important baselines are set: one is a U-Net or V-Net model trained with only a small amount of labeled data for full supervision, serving as the lower limit of performance, and the other is a U-Net or V-Net model trained with all data for full supervision, serving as the upper limit of performance.

[0143] (4) Evaluation indicators: Four internationally recognized and complementary evaluation indicators are used to comprehensively measure the performance of the model: Dice coefficient (%): an indicator that measures the volume overlap of the predicted segmentation region and the true region, with a higher value being better. Jaccard index (%): also known as the intersection over union (IoU), which also measures the volume overlap, with a higher value being better. 95% Hausdorff distance (mm): a robust indicator that measures the maximum mismatch between two segmentation boundaries, with a lower value being better. Average surface distance (ASD, mm): an indicator that measures the average distance between two segmentation boundaries, with a lower value being better.

[0144] Experimental results and analysis:

[0145] (1) Performance on the ACDC dataset.

[0146] Tables 1 and 2 are the implementation effect verification part of the present application, which respectively show the performance comparison results of the present application method and a variety of existing advanced methods under different labeling rate settings on the 5% and 10% labeled ACDC dataset in table form.

[0147] Table 1

[0148] Table 2

[0149] Table 2

[0150]

[0151] Referring to Table 1 and Table 2, the performance comparison data under the setting of two different annotation rates of 5% and 10% on the ACDC dataset are listed in detail. In the extreme data scarce scenario of 5% annotation rate (3 cases), the method of the present application (Ours) achieves excellent performance of average Dice coefficient 89.05% and 95% Hausdorff distance 1.33mm. This result not only far exceeds the baseline model trained only with labeled data (Dice 47.83%), but also comprehensively surpasses all existing advanced methods for comparison. For example, the Dice coefficient of the MPER method with similar performance is 88.14%, and the 95HD is 4.24mm. The method of the present application leads nearly 1 percentage point in Dice coefficient and shows a huge advantage in the 95HD index measuring boundary quality, which fully proves the excellent ability of the present application in improving the boundary positioning accuracy by using unlabeled data. When the labeled data increases to 10% (7 cases), the performance advantage of the method of the present application is further consolidated and expanded, with an average Dice coefficient of 90.38% and a 95HD of 1.24mm. Similarly, this performance comprehensively leads all comparison methods and further approaches the performance upper limit (Dice 91.44%) of full supervision training using all data. This shows that the method of the present application can continuously and efficiently benefit from the newly added unlabeled data.

[0152] (2) Performance on LA dataset

[0153] Table 3 and Table 4 are the verification part of the implementation effect of the present application, which respectively show the performance comparison results of the method of the present application and various existing advanced methods under different annotation rate settings on the 5% and 10% annotated LA dataset in table form.

[0154] Table 3

[0155]

[0156] Table 4

[0157]

[0158] Referring to Tables 3 and 4, this table presents the performance comparison results on another challenging 3D LA left atrial segmentation dataset. Under extreme data scarcity conditions (5% labeling rate, 4 cases), the method of this invention achieves superior performance on the 3D LA dataset, with an average Dice coefficient of 89.34% and a 95% Hausdorff distance of 6.63 mm. This result not only far surpasses the V-Net baseline model trained only on labeled data (Dice coefficient of only 52.55%), but also comprehensively outperforms all existing state-of-the-art semi-supervised methods compared. For example, the equally high-performing MPER method has a Dice coefficient of 88.40% and a 95HD of 7.49 mm. The method of this invention leads by nearly one percentage point in the Dice coefficient and also demonstrates a significant advantage in the 95HD metric, which measures boundary quality. This fully demonstrates the superior ability of this invention to efficiently utilize unlabeled information and accurately reconstruct the boundaries of complex anatomical structures even with 3D data and extremely limited supervision signals. With a 10% labeling rate (8 labeled cases), the method of this invention achieved an average Dice coefficient of 91.07% and a Hausdorff distance of 5.19 mm (95%). This result not only significantly outperforms the V-Net baseline model trained only on labeled data (Dice 82.74%), but also ranks first among all semi-supervised comparative methods. For example, the high-performance MPER method has a Dice coefficient of 89.96% and CrossMatch has a Dice coefficient of 89.31%. This verifies the good generalization ability and superior performance of the method of this invention on 3D data and different anatomical structures.

[0159] Visualization results analysis:

[0160] (1) Visualization on ACDC: Refer to Figure 3 and Figure 4 This figure visually compares the segmentation results of the method of this invention with several other representative methods on the ACDC dataset. Each row in the figure represents a different test slice, and each column represents a method. By comparing with the ground truth (GT) label in the last column, it can be clearly observed that the segmentation boundary generated by the method of this invention (second to last column) is the smoothest, most continuous, and closest to the real anatomical structure. The outlines of the right ventricle (RV, red), myocardium (MYO, green), and left ventricle (LV, blue) are very clear and accurate. In contrast, other comparison methods have some problems to varying degrees. For example, the BCP method shows small adhesions at the apex, the CrossMatch method does not segment the myocardium completely, and the ABD method shows some missegmentation at the right ventricular boundary. These visual differences intuitively demonstrate the superiority of this invention in improving segmentation detail and boundary accuracy.

[0161] (2) Visualization on LA: Refer to Figure 5 With Figure 6 (Note: corresponding to the picture you uploaded), the figure clearly shows the segmentation results of the same test sample on the LA dataset for the invented method (Ours) and various existing advanced semi-supervised segmentation methods (such as BCP, MLWPL, CrossMatch, MPER, etc.), and the last column of the true label GT as the gold standard. The three-dimensional segmentation results generated by the invented method are the most complete in overall morphology, the most smooth and natural in surface, and the highest consistency with the true label in topological structure. The areas highlighted by the blue circles in the figure are mainly concentrated in the entrance of the left atrium and the pulmonary vein, which are the most complex and most prone to error key parts in anatomy. The pulmonary vein entrance and other detail areas marked by the blue circle can accurately restore the tubular entrance structure, with natural morphology and clear boundaries. In contrast, the performance of various comparison methods in this place has obvious defects: the segmentation results of BCP and CrossMatch appear over-smooth or "mushy" at the pulmonary vein entrance, failing to accurately segment the tubular extension structure of the vein, resulting in loss of key anatomical details. MPER shows under-segmentation, with obvious cracks or holes in the pulmonary vein area, disrupting the topological connectivity of the left atrium as a complete chamber. MLWPL may produce false adhesion or redundant parts, mistakenly including surrounding tissues that do not belong to the left atrium.

[0162] Through the above comprehensive and detailed quantitative and qualitative comparison with various current advanced methods on two standard datasets for 2D and 3D application scenarios, the experimental results strongly prove that the invented "anti-knowledge forgetting mutual guidance semi-supervised medical image segmentation method" can effectively overcome the defects of the prior art, through its innovative implicit disturbance mechanism, knowledge preservation strategy and high-quality pseudo-label generation technology, and can effectively learn robust and strongly discriminative features with only a small amount of labeled data, thereby achieving segmentation accuracy and boundary quality far superior to the prior art.

Claims

1. An anti-knowledge forgetting mutual guidance type semi-supervised medical image segmentation method, characterized in that, The method comprises the following steps: (a) providing a labeled image dataset and an unlabeled image dataset; (b) establishing a dual-network model comprising a first segmentation network and a second segmentation network, wherein the network weight parameters of the first segmentation network and the second segmentation network are set to different initial values; (c) performing a data fusion step to obtain a first image from the labeled image dataset and a second image from the unlabeled image dataset; according to a preset content mask, combining the region derived from the second image with the region derived from the first image to form a mixed image intended to force the dual-network model to simultaneously process the labeled and unlabeled data distribution features in a single iteration; (d) performing a model training step to train the dual-network model; the training step comprises: (d1) using the first segmentation network to predict the second image to generate a first pseudo label; (d2) constructing a mixed supervision signal composed of the true label corresponding to the first image and the first pseudo label, and using the mixed supervision signal to supervise the prediction result of the dual-network model on the mixed image; (e) repeating the model training step according to a preset loss function until the dual-network model converges, and using the obtained model to perform medical image segmentation.

2. The method of claim 1, wherein the method is an anti-knowledge forgetting co-teaching semi-supervised medical image segmentation method. After generating the first pseudo label in step (d1), a connected component analysis step is further performed on the first pseudo label, and the largest connected region is selected to form a structurally purified pseudo label, which is used to provide stable and reliable supervision signal in the mutual guidance learning process of the dual-network model.

3. The method of claim 1, wherein the method is an anti-knowledge forgetting co-teaching semi-supervised medical image segmentation method. Before obtaining the first image and the second image in step (c), a dual-stream batch sampling step is further included to form a mixed batch containing labeled data and unlabeled data, the dual-stream batch sampling step comprising: providing sample index lists for the labeled image dataset and the unlabeled image dataset respectively, and presetting a total batch size and the number of labeled samples in the batch; when generating each training batch, randomly selecting sample indexes from the labeled sample index list and the unlabeled sample index list according to the preset number to obtain two groups of sample indexes; merging the two groups of sample indexes to form the mixed batch, thereby ensuring that the dual-network model is simultaneously provided with labeled and unlabeled data in a single iteration.

4. The method of claim 1, wherein the method is an anti-knowledge forgetting co-teaching semi-supervised medical image segmentation method. The data fusion step (c) specifically comprises: (c1) dividing the labeled images in a batch into a first labeled sub-batch and a second labeled sub-batch, and dividing the unlabeled images in the batch into a first unlabeled sub-batch and a second unlabeled sub-batch; (c2) generating a first mixed image by randomly generating a first binary content mask batch and combining the background region derived from the first unlabeled sub-batch with the content region derived from the first labeled sub-batch based on the mask; (c3) generating a second mixed image in a manner that: a second binary content mask batch is randomly generated, and background regions from the second labeled sub-batch and content regions from the second unlabeled sub-batch are combined based on the mask.

5. The method of claim 4, wherein the method is an anti-knowledge forgetting co-teaching semi-supervised medical image segmentation method. The manner of generating the first binary content mask and the second binary content mask batch comprises: For each group of images to be combined, a rectangular mask is independently and randomly generated; Wherein, the size of the rectangular mask is set to be a preset proportion of the image size, and the position of the rectangular mask in the image is determined by random sampling.

6. The method of claim 4, wherein the method is an anti-knowledge forgetting co-teaching based semi-supervised medical image segmentation method. In step (d2), the mixed supervision signal is constructed in the following manner: Using the content mask, the real label corresponding to the first image is taken as the supervision signal of the background region, and the pseudo label generated after the second image is predicted by another network is taken as the supervision signal of the central content region, and the mixed supervision signal is spliced.

7. The method of claim 6, wherein the method is an anti-knowledge forgetting co-teaching semi-supervised medical image segmentation method. The dual network model is trained in the following mutual guidance manner: Step 11, let the second segmentation network predict the first mixed image and the second mixed image respectively; Step 12, the supervision of the first segmentation network comprises: Step 121, using the second segmentation network to predict the images from the first unlabeled sub-batch to generate the first pseudo label, and predicting the images from the second unlabeled sub-batch to generate the second pseudo label; Step 122, a first mixed supervision signal is constructed by combining the content region derived from the real label of the first labeled sub-batch and the background region of the first pseudo label, and is used to supervise the prediction result of the first mixed image by the first segmentation network; Step 123, a second mixed supervision signal is constructed by combining the background region derived from the real label of the second labeled sub-batch and the content region of the second pseudo label, and is used to supervise the prediction result of the second mixed image by the first segmentation network.

8. The anti-knowledge forgetting co-teaching semi-supervised medical image segmentation method according to claim 6 or 7, characterized in that, The dual network model is trained in the following mutual guidance manner: Step 21, let the first segmentation network predict the first mixed image and the second mixed image respectively; Step 22, the supervision of the second segmentation network comprises: Step 221, using the first segmentation network to predict the images from the first unlabeled sub-batch to generate the first pseudo label, and predicting the images from the second unlabeled sub-batch to generate the second pseudo label; Step 222, a first mixed supervision signal is constructed by combining the content region derived from the real label of the first labeled sub-batch and the background region of the first pseudo label, and is used to supervise the prediction result of the first mixed image by the second segmentation network; Step 223, a second mixed supervision signal is constructed by combining the background region derived from the real label of the second labeled sub-batch and the content region of the second pseudo label, and is used to supervise the prediction result of the second mixed image by the second segmentation network.

9. The method of claim 1, wherein the method is an anti-knowledge forgetting co-teaching semi-supervised medical image segmentation method. In step (e), the preset loss function is a total objective function defined as a weighted sum of a supervised loss and an unsupervised consistency loss; wherein the supervised loss comprises cross-entropy losses respectively calculated based on masks, and the unsupervised consistency loss comprises a Dice loss calculated on the entire mixed image.

10. The method of claim 1, wherein the method is an anti-knowledge forgetting co-teaching semi-supervised medical image segmentation method. In step (e), the model training adopts a stochastic gradient descent optimizer, and a learning rate update strategy thereof adopts a polynomial decay strategy or a combined strategy comprising a plateau and a cosine annealing period.

Images