Semi-supervised medical image segmentation method based on cascaded parallel CNN-transformer and full perturbation

Abstract

The application discloses a semi-supervised medical image segmentation method based on a cascaded parallel CNN-Transformer and full disturbance, and comprises the following steps: step 1, preprocessing an original medical image dataset; step 2, combining an encoder based on a cascaded parallel CNN-Transformer structure and a full disturbance consistency learning strategy, a full disturbance self-integrated framework FPSE based on the cascaded parallel CNN-Transformer is constructed, shallow local features of the medical image are extracted through the CNN first, and then deep local and global features of the medical image are extracted through the CNN and the Transformer double-branch network respectively; step 3, parameters required for training the FPSE are configured, the FPSE is semi-supervisedly trained by using the preprocessed medical image dataset, and the optimal weight and parameters of the FPSE are updated; and step 4, the trained FPSE is used for medical image segmentation, and a prediction result is output. The application improves the accuracy and robustness of medical image segmentation under the condition that the labeled data is very limited.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, specifically a semi-supervised medical image segmentation method based on cascaded parallel CNN-Transformer and full perturbation. Background Technology

[0002] Medical image segmentation technology plays a crucial role in modern medicine, with applications spanning numerous areas such as lesion localization, disease assessment, surgical planning, drug delivery, and real-time surgical navigation. By combining medical image segmentation technology with computer vision and deep learning, doctors can diagnose diseases more accurately, develop treatment plans, and improve the precision and safety of surgeries. However, realizing the full potential of these applications faces a series of significant challenges: First, the complexity of medical images makes segmentation extremely challenging. Medical images are typically highly variable; images acquired from different patients, using different instruments, and under different imaging conditions can be vastly different. This diversity complicates the development of general-purpose medical image segmentation algorithms, as these algorithms need sufficient robustness to handle these variations. Second, the annotation process for medical images requires experienced physicians and is costly and time-consuming. For many diseases, such as tumors or heart disease, physicians need to label each lesion or organ in the image individually, requiring substantial human and material resources. Furthermore, the complexity of medical images introduces subjectivity and uncertainty into the annotation process, potentially leading to inconsistent label data. Third, label data for some diseases requires invasive testing, especially for some cancer cases where a definitive diagnosis can only be made through tissue section analysis, thus limiting the construction of large-scale labeled databases.

[0003] To overcome this problem, semi-supervised learning methods have emerged in the field of medical image segmentation. Semi-supervised learning aims to train models using information from both labeled and unlabeled data, thereby reducing the need for large-scale labeled data. In medical image segmentation, labeled data is typically manually labeled by professional physicians, while unlabeled data consists of unlabeled, unprocessed medical images. This approach provides an innovative solution for medical image segmentation and is expected to play a significant role in improving diagnosis and treatment. Existing semi-supervised medical image segmentation methods have made some progress, with consistency learning becoming an important research direction. Consistency learning aims to improve model performance by imposing consistency constraints on the prediction results of labeled and unlabeled data. For example, Yu et al. proposed a semi-supervised segmentation model with uncertainty-aware MT for left atrial segmentation. Li et al. proposed a semi-supervised segmentation method for skin lesions by forcing consistency between student and teacher models. Chen et al. proposed a cross-pseudo-supervision method based on network perturbation to encourage high consistency in the prediction results of two perturbation networks. This consistency is usually achieved by introducing perturbations or multi-view strategies to increase the diversity and consistency of prediction results. In existing technologies, common perturbation strategies include data perturbation, model perturbation, and feature perturbation. Data perturbation is widely used in methods such as Mean Teacher, FixMatch, and UniMatch. By applying different perturbations to the input data, such as weak perturbations like random rotation, mirroring, cropping, scaling, and adding noise, or strong perturbation methods like CutMix and AutoAugment, the model can observe and understand the data from different perspectives, improving its generalization ability to the input data. Feature perturbation is applied in methods such as CCT and UniMatch. By introducing noise, random cutting, or random dropout into the encoded output features, multiple feature perturbation variants are generated, guiding the model to learn more robust and stable feature representations. Model perturbation involves perturbing the model itself. Methods such as CPS and GCT further enhance the model's diversity and robustness by assigning different weights to the model for initialization or adjusting the model structure.

[0004] While the aforementioned semi-supervised learning methods have achieved good results in medical image segmentation, they still face several challenges: First, the quality and diversity of unlabeled data can negatively impact model performance. Since unlabeled data often contains varying degrees of noise and pseudo-labels, this can lead to inconsistencies that interfere with model training. Second, achieving consistency learning between labeled and unlabeled data is a crucial issue. Consistency learning aims to improve model performance by imposing consistency constraints on predictions between labeled and unlabeled data. This constraint is typically achieved by introducing perturbations or multi-view strategies to increase the diversity and consistency of predictions. However, most current consistency learning methods rely solely on a single perturbation or a simple combination of partial perturbations, failing to fully exploit the potential information in unlabeled medical image data. Therefore, more robust and diverse consistency learning strategies are needed to improve model performance. Furthermore, the mainstream frameworks commonly used in medical image segmentation typically employ cascaded or parallel convolutional neural networks (CNNs) and Transformers. While these models perform well in many computer vision tasks, their performance in medical image segmentation may be limited by the relatively weak local perception capabilities and large number of parameters, especially with limited labeled data.

[0005] In summary, while semi-supervised medical image segmentation is a research area with great potential, it also faces several challenges, specifically in the following aspects: First, the annotation process for medical images requires experienced physicians, is time-consuming, and incurs high labor costs. Second, current mainstream medical image segmentation frameworks employ a cascaded or parallel structure of CNNs and Transformers. However, the cascaded CNN-Transformer structure has relatively many parameters for the inference model, potentially increasing computational and memory overhead. Therefore, this structure has not yet been applied or researched in semi-supervised learning. In the parallel CNN-Transformer structure, due to the lack of local connectivity and position awareness in the Transformer model, it relies on a large amount of labeled data to learn feature patterns to model global dependencies. This makes it difficult to fully leverage its performance advantages when using limited labeled medical image data for semi-supervised learning. Third, in semi-supervised consistency learning, most current methods rely solely on a simple combination of single or partial perturbations, failing to fully exploit the potential information in unlabeled medical image data. This results in slow progress in feature learning from unlabeled data and limited model generalization ability. Therefore, there is an urgent need to innovate semi-supervised medical image segmentation methods to improve the accuracy and efficiency of disease diagnosis and treatment, which has positive and important significance for society. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a semi-supervised medical image segmentation method based on cascaded parallel CNN-Transformer and full perturbation, which improves the accuracy and robustness of medical image segmentation even with very limited labeled data.

[0007] To achieve the above objectives, the present invention employs the following technical solution: A semi-supervised medical image segmentation method based on cascaded parallel CNN-Transformer and full perturbation includes the following steps: Step 1: Preprocess the original medical image dataset; Step 2: Combining the encoder based on the cascaded parallel CNN-Transformer structure and the fully perturbed consistency learning strategy, a fully perturbed self-integration framework FPSE based on the cascaded parallel CNN-Transformer is constructed. First, the shallow local features of the medical image are extracted by the CNN, and then the deep local and global features of the medical image are extracted by the dual-branch network of CNN and Transformer respectively. Step 3: Configure the parameters required for FPSE training, perform semi-supervised training on FPSE using the preprocessed medical image dataset, and update the optimal weights and parameters of FPSE. Step 4: Perform medical image segmentation on the trained FPSE and output the prediction results.

[0008] Furthermore, the medical image dataset in step 1 includes the public datasets ACDC and AbdomenCT-1K; The ACDC dataset contains short-axis cardiac magnetic resonance imaging images of 100 subjects. For each image, segmentation labels for the left ventricle, myocardium, and right ventricle are provided. The short-axis cardiac magnetic resonance imaging images of the 100 subjects are randomly divided into training, validation, and test sets in a ratio of 7:1:2. The AbdomenCT-1K dataset contains 41 annotated cases, with segmentation labels for the liver, kidney, spleen, and pancreas provided for each image. The 41 cases are randomly divided into training, validation, and test sets in a ratio of 30:4:7.

[0009] Furthermore, step 1 preprocesses the medical image dataset by random cropping, random scaling, horizontal flipping, vertical flipping, and adding Gaussian perturbation, thereby expanding the original medical image dataset.

[0010] Furthermore, the random cropping involves cropping the original medical image to a size of 224*224.

[0011] Furthermore, the random scaling is a scaling of the original medical image at a ratio of 0.8 to 1.25.

[0012] Furthermore, the probabilities of horizontal flipping, vertical flipping, and adding Gaussian perturbation are all 0.5.

[0013] Furthermore, the specific process of constructing the FPSE in step 2 is as follows: Step 2.1: Construct a cascaded parallel CNN-Transformer structure CPCT. It first uses the first five layers of the ResNet50 model to extract shallow local features of medical images, and then uses the UNet model and the Swin-Unet model in parallel to extract deep local features and global features of medical images respectively. The Swin-Unet model performs global dependency modeling based on the shallow local features extracted by the ResNet50 model. Step 2.2: Combine data perturbation, model perturbation, and feature perturbation to construct a full perturbation strategy, wherein: the data perturbation refers to simultaneously performing strong and weak enhancement operations on the original data X to obtain strongly enhanced data X. S And weakly augmented data X W Then X S X W The FPSE is trained along with X; the model perturbation refers to using the cascaded parallel CNN-Transformer structure as the encoder, and using two UNet decoders with the same structure but different parameter initializations in the FPSE decoder part to achieve perturbation of the FPSE structure and parameters; the feature perturbation refers to randomly applying Dropout technology to the output features of the encoder CNN branch and the Transformer branch respectively. Step 2.3: Based on the full perturbation strategy, data perturbation consistency, feature-model perturbation consistency, and data-feature-model perturbation consistency are combined to construct a full perturbation consistency learning strategy. Specifically: data perturbation consistency refers to the CNN subnet and Transformer subnet predicting the original data and augmented data respectively, and applying consistency constraints to the prediction results of the same subnet; feature-model perturbation consistency refers to the CNN subnet and Transformer subnet using the prediction results of the original data as pseudo-labels for mutual pseudo-supervised training; and data-feature-model perturbation consistency includes data-feature-model perturbation consistency between the original data and augmented data, as well as data-feature-model perturbation consistency between weakly and strongly augmented data. Step 2.4: Combining the fully perturbation consistency learning strategy, and based on the cascaded parallel CNN-Transformer structure CPCT, a fully perturbation self-integration framework FPSE based on cascaded parallel CNN-Transformer is constructed.

[0014] Furthermore, step 2.1 discards the first two layers of the UNet model and the Swin-Unet model.

[0015] Furthermore, step 2.3, the process of constructing the fully perturbation-consistent learning strategy, is as follows: Step 2.3.1: Establish data perturbation consistency FPSE for input unlabeled data X U Enhanced Data X S And weakly augmented data X W Four different predictions are generated, represented as follows: In the formula, , Representing CNN subnets and Transformer subnet The prediction results of the original data, express Prediction results for strongly enhanced data, express For the prediction results of weakly augmented data, the prediction results generated from the original data are used as pseudo-labels to perform pseudo-supervision on the augmented data, so that the prediction results of the augmented data and the original data tend to be consistent. The generated pseudo-labels are represented as follows: In the formula, , Let these be the pseudo-labels generated from strongly augmented data and weakly augmented data, respectively. The DPC loss can be defined as: In the formula, L dice It is the standard dice loss function, L dpc_C and L dpc_T The DPC loss for the CNN subnet and the Transformer subnet are respectively, L dpc This is the total DPC loss; Step 2.3.2: Construct feature-model perturbation consistency based on and For the prediction of the original data, at this point, and To each and Since pseudo-supervision is performed, the FMPC loss can be defined as: In the formula, L fmpc_C and L fmpc_T The FMPC loss for the CNN subnet and the Transformer subnet are respectively, L fmpc Total FMPC loss; Step 2.3.3: Constructing Data-Feature-Model Perturbation Consistency Step 2.3.3.1: The data-feature-model perturbation consistency between the original data and the augmented data is denoted as the first group. The prediction results of the CNN subnet on the original data are used as pseudo-labels to perform pseudo-supervision on the prediction results of the Transformer subnet on the weakly augmented data. Simultaneously, the prediction results of the Transformer subnet on the original data are used as pseudo-labels for the prediction results of the CNN subnet on the strongly augmented data. The DFMPC loss between the original data and the augmented data can be defined as: In the formula, L dfmpc_C1 and L dfmpc_T1 The first set of DFMPC losses for the CNN subnet and the Transformer subnet are respectively, L dfmpc1 This is the total DFMPC loss for the first group; Step 2.3.3.2: The data-feature-model perturbation consistency between weakly and strongly augmented data is designated as the second group. This group is trained using a CNN subnet and a Transformer subnet, respectively, with mutual pseudo-supervision training on the prediction results of the strongly and weakly augmented data. and For predictions based on augmented data, the generated pseudo-labels are represented as follows: In the formula, , The pseudo-labels generated from strongly augmented data and weakly augmented data are represented respectively. The second set of DFMPC losses can be defined as follows: In the formula, L dfmpc_C2 and L dfmpc_T2 The second set of DFMPC losses for the CNN subnet and the Transformer subnet are respectively, Ldfmpc2 This is the total DFMPC loss for the second group; Step 2.3.3.3, the total DFMPC loss for the first and second groups is expressed as: .

[0016] Furthermore, step 3, which involves semi-supervised training of the FPSE, is as follows: Step 3.1: Input the data into the FPSE, which includes 8 labeled medical image data and 8 unlabeled medical image data; Step 3.2: For labeled data, the traditional supervised training method is used, that is, the FPSE prediction results for labeled data are used to calculate Dice loss and cross-entropy loss with the real labels and implement consistency constraints; for unlabeled data with multiple perturbations, FPSE generates multiple different prediction results, and a full perturbation consistency learning strategy is used to train multiple different prediction results, thereby implementing consistency constraints on multiple different prediction results. Step 3.3: After tens of thousands of training iterations, the SGD optimizer with momentum of 0.9 and weight decay of 0.0001 is adopted, and the learning rate is adjusted by a multivariate learning rate strategy to continuously optimize the weights and parameters of FPSE.

[0017] Compared with the prior art, the present invention has the following technical effects: The cascaded parallel CNN-Transformer structure designed in this invention integrates the advantages of CNN and Transformer, enabling the Transformer to model global dependencies based on low-level feature information extracted by CNN. This captures high-level global dependencies and local and global information in the image, helping to reduce the Transformer's dependence on large amounts of labeled data and improving the model's performance with limited labeled data, thereby improving the medical image segmentation effect. In addition, the fully perturbation consistency learning strategy not only introduces perturbations to data, model, and features simultaneously, enabling the model to fully mine, understand, and learn the potential information in unlabeled medical images from multiple perspectives, thus improving the model's understanding and utilization performance of unlabeled data, but also uses a consistency regularization method to encourage the model to maintain consistent prediction results under fully perturbation conditions, enhancing the model's generalization ability and robustness, and improving the efficiency of the FPSE network framework model in utilizing information in unlabeled medical images. In summary, the FPSE network framework model can achieve excellent medical image segmentation performance with very limited labeled data, surpassing most current state-of-the-art methods. Attached Figure Description

[0018] Figure 1 : Flowchart of the present invention; Figure 2 : A schematic diagram of the FPSE structure of the present invention; Figure 3 Visualization of inference results from this invention and other methods on ACDC test sets labeled with 5% and 10%; Figure 4 Visualization of inference results using the AbdomenCT-1K test set with 5% and 10% labeling, compared with other methods; Figure 5 : Schematic diagram of different combinations of CNN-Transformer structures; Figure 6 : A schematic diagram of the structure of the fully perturbation-based consensus learning strategy of the present invention. Detailed Implementation

[0019] The specific content of the present invention will be further explained in detail below with reference to the embodiments.

[0020] like Figure 1 As shown, the semi-supervised medical image segmentation method based on cascaded parallel CNN-Transformer and full perturbation includes the following steps: Step 1: Preprocess the original medical image dataset; The original medical image datasets primarily consist of the public datasets ACDC and AbdomenCT-1K. The ACDC dataset contains short-axis cardiac magnetic resonance imaging images from 100 subjects, with segmentation labels for the left ventricle (LV), myocardium (Myo), and right ventricle (RV) provided for each image. 70, 10, and 20 subject cases were randomly selected as training, validation, and test sets, respectively. The AbdomenCT-1K dataset contains 41 annotated cases, with segmentation labels for the liver, kidney, spleen, and pancreas provided for each image. The data were randomly divided into 30, 4, and 7 cases for training, validation, and test sets, respectively. In the case of a small amount of labeled data and a large amount of unlabeled data, in order to improve the generalization ability and robustness of FPSE, this embodiment performs the following preprocessing on the original medical image dataset to achieve data augmentation: ① preprocess the original images to a size of The process involves: ① random cropping of the image; ② random scaling of the original image, with a scaling ratio between 0.8 and 1.25; ③ horizontal flipping with a probability of 0.5; ④ vertical flipping with a probability of 0.5; ⑤ adding Gaussian perturbation with a probability of 0.5. Simultaneously, the corresponding manually labeled mask undergoes the same data augmentation operation. These operations are performed to randomly augment the images in the training dataset in real time during the model's training process to enhance the model's generalization performance. However, no data augmentation is performed on the test dataset. Furthermore, data normalization is applied to ensure that the image pixel values ​​are within an appropriate range, and window width and level are adjusted to improve image contrast. Step 2: Combining the encoder based on the cascaded parallel CNN-Transformer structure and the fully perturbed consistency learning strategy, construct a system as follows: Figure 2 The fully perturbation self-integration framework FPSE based on cascaded parallel CNN-Transformer is shown. To fully utilize both limited labeled medical images and a large number of unlabeled medical images for semi-supervised medical image segmentation, this embodiment proposes a fully perturbed self-integration framework (FPSE) based on cascaded parallel CNN-Transformer. The construction process of FPSE is as follows: Step 2.1: Construct the cascaded parallel CNN-Transformer network structure CPCT. The structure of CPCT is as follows: Figure 5 As shown in (c), the blue in the figure represents the CNN model and the red represents the Transformer model. CPCT can give full play to the role of the Transformer network under limited labeled data. Unlike the parallel CNN-Transformer structure, which directly uses the CNN model and the Transformer model to encode medical images, the CPCT structure first uses the first five layers of the ResNet50 (CNN) model to extract shallow local features such as edges, textures and shapes in medical images. Then, on this basis, the UNet (CNN) model and the Swin-Unet (Transformer) model are used in parallel to extract deep local and global features of medical images, respectively. CPCT overcomes the performance limitation of traditional concatenated models on limited labeled data, while retaining the advantages of the parallel structure of CNN and Transformer, and can better handle semi-supervised medical image segmentation tasks. CPCT enables the Swin-Unet model to model global dependencies based on shallow local features extracted by the ResNet50 model. Specifically, the Swin-Unet model pays high attention to globally similar local features and low attention to irrelevant local features, without relying on a large amount of labeled medical image data to learn feature patterns to model global dependencies. At the same time, CPCT can use CNN subnets for model inference independently without introducing additional parameters, and retains the advantages of parallel CNN-Transformer structures, allowing FPSE to learn and represent local and global features at different levels simultaneously. To ensure fairness, the first two layers of UNet and Swin-Unet are discarded in the encoder of FPSE. Step 2.2: To fully extract the rich information from unlabeled data from multiple perspectives, a full perturbation strategy is designed, combining data perturbation, model perturbation, and feature perturbation. Specifically: ① Data perturbation: Strong and weak augmentation operations are simultaneously performed on the original data X to obtain strongly augmented data X. S And weakly augmented data X W Then X S X W X is input into FPSE for training; since the CNN subnetwork structure is simple and easy to optimize, X is used... S In the input CNN subnet, the Transformer subnet has a relatively complex structure and is difficult to optimize, therefore X is... W The data is fed into the Transformer subnet for prediction. Notably, data augmentation strategies are applied not only to unlabeled data but also to labeled data, thereby increasing the amount of labeled data. ② Model perturbation: The cascaded parallel CNN-Transformer structure is used as the encoder, and two UNet decoders with the same structure but different parameter initializations are used in the FPSE decoder part to achieve perturbation of the FPSE structure and parameters. ③ Feature perturbation: To encourage FPSE to learn more robust feature representations, Dropout technology is randomly applied to the output features of the CNN branch and the Transformer branch of the encoder to achieve feature perturbation. Step 2.3: Based on the full perturbation strategy, a full perturbation consistency learning strategy was designed to improve the efficiency of the network model in utilizing information in unlabeled medical images. The full perturbation consistency learning strategy mainly covers three aspects: data perturbation consistency, feature-model perturbation consistency, and data-feature-model perturbation consistency. It fully mines the rich information of unlabeled data from multiple perspectives, thereby enhancing the robustness and generalization ability of the model. The design process of the full perturbation consistency learning strategy is as follows: Step 2.3.1: Construct data perturbation consistency. Use CNN subnets and Transformer subnets to predict the original data and augmented data respectively, and apply consistency constraints to the prediction results of the same subnet. FPSE applies the input unlabeled data X. U Enhanced Data X S And weakly augmented data X W Four different predictions are generated, represented as follows: In the formula, , Representing CNN subnets and Transformer subnet The prediction results of the original data, express Prediction results for strongly enhanced data, express For the prediction results of weakly augmented data, the prediction results generated from the original data are used as pseudo-labels to perform pseudo-supervision on the augmented data, so that the prediction results of the augmented data and the original data tend to be consistent. The generated pseudo-labels are represented as follows: In the formula, , Let these be the pseudo-labels generated from strongly augmented data and weakly augmented data, respectively. The DPC loss can be defined as: In the formula, L dice It is the standard dice loss function, L dpc_C and L dpc_T L represents the DPC loss for the CNN subnet and the Transformer subnet, respectively. dpc This is the total DPC loss; Step 2.3.2: Construct feature-model perturbation consistency. The CNN subnet and the Transformer subnet use the prediction results of the original data as pseudo-labels and perform pseudo-supervised training on each other. and For the prediction of the original data, at this point, and Let each represent a pseudo-label generated for the other. Therefore, the FMPC loss can be defined as: In the formula, L fmpc_C and L fmpc_T The FMPC loss for the CNN subnet and the Transformer subnet are respectively, L fmpc Total FMPC loss; Step 2.3.3: Construct data-feature-model perturbation consistency, which includes two sets of consistency strategies based on data-feature-model perturbation. The first set is the data-feature-model perturbation consistency between the original data and the augmented data, and the second set is the data-feature-model perturbation consistency between the weakly augmented data and the strongly augmented data. Step 2.3.3.1: The first group uses the prediction results of the CNN subnet on the original data as pseudo-labels to perform pseudo-supervision on the prediction results of the Transformer subnet on the weakly augmented data; simultaneously, the prediction results of the Transformer subnet on the original data are used as pseudo-labels for the prediction results of the CNN subnet on the strongly augmented data. The DFMPC loss between the original data and the augmented data can be defined as: In the formula, L dfmpc_C1 and L dfmpc_T1 The first set of DFMPC losses for the CNN subnet and the Transformer subnet are respectively, L dfmpc1 This is the total DFMPC loss for the first group; Step 2.3.3.2: The second group uses CNN subnets and Transformer subnets to perform mutual pseudo-supervised training on the prediction results of strongly enhanced and weakly enhanced data, respectively, based on... and For predictions based on augmented data, the generated pseudo-labels are represented as follows: In the formula, , The pseudo-labels generated from strongly augmented data and weakly augmented data are represented respectively. The second set of DFMPC losses can be defined as follows: In the formula, L dfmpc_C2 and L dfmpc_T2 The second set of DFMPC losses for the CNN subnet and the Transformer subnet are respectively, L dfmpc2 This is the total DFMPC loss for the second group; Step 2.3.3.3, the total DFMPC loss for the first and second groups is expressed as: The fully perturbation self-integration framework FPSE based on cascaded parallel CNN-Transformer constructed in this embodiment achieves excellent medical image segmentation performance with very limited labeled data, surpassing most of the current state-of-the-art methods. Step 3: Configure the parameters required for FPSE training, perform semi-supervised training on FPSE using the preprocessed medical image dataset, and update the optimal weights and parameters of FPSE. Step 3.1: After preprocessing, half of the original medical images are labeled and the other half are unlabeled. The data is input into the FPSE in batches of 16, that is, 8 labeled medical image data and 8 unlabeled medical image data are input into the FPSE. Step 3.2: For labeled data, this embodiment adopts the traditional supervised training method, which implements consistency constraints on the Dice loss and cross-entropy loss calculated by FPSE on the predicted results of labeled data and the true labels. This uses label information to guide the learning process of FPSE, ensuring that FPSE can accurately segment the target structure in the image. This part of the data provides a strong supervision signal for the performance of FPSE. For unlabeled data, this embodiment adopts a full-perturbation consistency learning strategy. This means that this embodiment applies multiple perturbations to these unlabeled data, including data perturbation, model perturbation, and feature perturbation, to generate multiple slightly different prediction results. This embodiment trains FPSE by encouraging the consistency of prediction results under these multiple perturbations, thereby constraining the consistency of multiple different prediction results. This helps FPSE understand and learn data from different perspectives, improving its robustness and generalization ability. Step 3.3: After 30k iterations of training, FPSE continuously optimizes the model's weights and parameters to gradually improve its performance on medical image segmentation tasks, thereby enabling it to better handle limited labeled and unlabeled data and achieve superior performance in semi-supervised medical image segmentation tasks. Step 4: Perform medical image segmentation on the trained FPSE and output the prediction results.

[0021] To verify the accuracy and superiority of the fully perturbed self-integration framework FPSE based on cascaded parallel CNN-Transformer proposed in this embodiment, the following hardware and software experimental platforms were used: ① In terms of hardware, this embodiment used an Intel Core i9-9900X 3.5GHz CPU and an NVIDIA GeForce RTX 3090Ti GPU with 24GB of video memory; ② In terms of software, this embodiment used open-source software and frameworks such as PyTorch, Python, OpenCV, and NumPy.

[0022] During the training phase of FPSE, the data loading method in this embodiment is as follows: the batch size of 2D data is set to 16. First, 8 labeled NIFTI images and their label masks are scaled. Then, another 8 unlabeled NIFTI images are scaled. Next, the medical images are horizontally flipped and vertically flipped with a probability of 0.5. Then, random center cropping is performed according to a certain size. Then, contrast-limited histogram equalization and Gaussian perturbation are performed with a probability of 0.5. Finally, the mean and variance of the images are standardized.

[0023] This embodiment uses a simple Stochastic Gradient Descent (SGD) optimization algorithm to train the FPSE, where the momentum is set to 0.9, the weight_decay is set to 0.001, the initial learning rate is 0.0001, and the learning rate is adjusted exponentially every 30,000 iterations with a decay rate of 0.1. To evaluate the algorithm's performance, this embodiment calculates the following metrics: In the formula, Represents set similarity. Indicates the degree of similarity between two sets of points. This represents the segmentation result of the algorithm. This represents the segmentation results manually annotated by experts. This indicates the boundaries manually marked by experts.

[0024] To verify the superiority of the proposed fully perturbed self-integration framework FPSE based on cascaded parallel CNN-Transformer in this embodiment, a comprehensive comparison was conducted with nine state-of-the-art semi-supervised methods under two different settings. These nine semi-supervised methods include: MT, CAA-ViT, UA-MT, URPC, CPS, CCT, MC-Net+, and CTCT. For ease of comparison, all comparison methods used UNet as the backbone network. Table 1 shows the experimental results of each method performing inference on the test set with only 5% and 10% labeled ACDC test set data. Here, Mean represents the average of the segmentation indices of the three organs LV, Myo, and RV. As shown in Table 1, FPSE exhibits superior performance with only 5% and 10% labeled data. Its overall index is significantly better than other state-of-the-art methods. In particular, when trained with only 5% labeled data, FPSE's Dice index is 11.5% higher than the current best method, and even higher than MT, CAA-ViT, UA-MT, URPC, CPS, and CCT methods trained with 10% labeled data by 3.9%, 8.5%, 4.8%, 1.6%, 1.1%, and 3.3%, respectively. Compared to training data with 5% labeled data, other methods showed significant performance improvements when using 10% labeled data. For example, CPS and CCT methods improved by 23.5% and 23% respectively, indicating that they are highly dependent on labeled data. In contrast, FPSE showed only a 3.4% performance improvement, which means that FPSE can fully extract effective information from 95% of unlabeled data when trained with 5% labeled data, rather than relying solely on information from labeled data. This further proves the ability of the FPSE model to effectively utilize unlabeled data in semi-supervised learning.

[0025] Table 1. Comparison of inference results between the method of this embodiment and nine existing methods on the ACDC dataset with 5% and 10% labeled data, respectively. Figure 3 The visualization shows the results of inference using 5% labeled data and 10% labeled ACDC test set data. It can be clearly observed from the figure that the FPSE method presents better segmentation results. Compared with other segmentation methods, FPSE has fewer false predictions and false negatives, and the segmentation edges are closer to the labels. In particular, when using only 5% labeled data, other methods have a large number of false predictions and false negatives on the segmentation target, but the FPSE method is still able to maintain relatively good segmentation results.

[0026] Table 2 shows the inference results of the method in this embodiment and nine existing methods on the AbdomenCT-1K test set data with 3% and 10% labeling. As can be seen from Table 2, FPSE achieves the best results under both proportions of labeled data. In particular, when trained with only 3% labeled data, its comprehensive index is improved by 5.5% and -2.2mm respectively compared with other segmentation methods. It is even higher than MT, CAA-ViT, CPS, CCT, CTCT and DSTP methods trained with 10% labeled data by 1.3%, 1.8%, 0.2%, 0.6%, 1.8% and 1.3% respectively, further verifying the superiority and structural advantages of FPSE.

[0027] Table 2. Comparison of inference results of the method of this embodiment with nine existing methods on the AbdomenCT-1K dataset labeled with 3% and 10% labels. Figure 4 The visualization results of various methods on AbdomenCT-1K training data with 3% and 10% labels are shown in the figure. As can be seen from the figure, other comparative methods often miss the liver and pancreas, or misidentify the spleen as the liver or the spleen as the kidney. In contrast, FPSE shows more reliable prediction results on both 3% and 10% labeled data and better preserves the anatomical morphology, which fully demonstrates the effectiveness of the FPSE method.

[0028] To verify the superiority of the CPCT structure, this embodiment uses the CPS framework as a benchmark and designs experiments with three different CNN-Transformer combination structures in its encoder section. See [link to relevant documentation]. Figure 5 (a) Parallel CNN-Transformer structure, Figure 5 (b) cascaded CNN-Transformer architecture and Figure 5(c) The CPCT structure was designed with four different baseline network combinations for CNN and Transformer to ensure sufficient comparison. Different CNN-Transformer combinations were compared on 5% labeled ACDC data. The following metrics were calculated on the test dataset: Dice similarity coefficient (Dice) and 95% Hausdorff Distance. (95HD) The experimental results are shown in Table 3. As can be seen from Table 3, the cascaded CNN-Transformer structure has the worst segmentation performance. In the combined network with UNet and Swin-Unet as the baseline, its Dice index is only 72.7%, which is 0.9% and 4.1% lower than the parallel CNN-Transformer structure and CPCT structure, respectively. This is because the cascaded structure limits the model's ability to extract local features and model global dependencies at the same time, thus negatively affecting the model's performance. Although the parallel CNN-Transformer structure has a slight advantage over the cascaded structure, its performance is limited in finitely labeled medical image data. In the four baseline network combinations, the Dice index of this structure is 3.2%, 3.9%, 2.6%, and 4.3% lower than that of the CPCT structure, respectively. This means that the CPCT structure can still give full play to the feature learning capabilities of CNN and Transformer in finitely labeled medical image data and achieve better segmentation results.

[0029] Table 3 compares different CNN-Transformer combinations on the 5% labeled ACDC data. Figure 6 Here is a simplified diagram of the fully perturbation-consistent learning strategy, where L dpc L represents the DPC loss. fmpc L represents FMPC loss dfmpc1 L represents the DFMPC loss between the original data and the augmented data. dfmpc2 The DFMPC loss represents the difference between weakly and strongly augmented data. To evaluate the effectiveness of the fully perturbation-based consistency learning strategy, this embodiment conducted a series of ablation experiments on 5% labeled ACDC training samples, calculating the Dice similarity coefficient (Dice) and 95% Hausdorff Distance (95HD). The results were used to explore the contributions of DPC, FMPC, and the two sets of DFMPC to the model performance, as shown in Table 4. In the table, "√" indicates that the corresponding loss was used to train FPSE. By observing the 15 sets of experimental results in Table 4, it can be seen that as L... dpc L fmpc Ldfmpc1 L dfmpc2 With the addition of [L], the model's performance metrics were improved to some extent; by comparing experiments 1, 2, 3, and 4, it can be found that L […]. dfmpc1 The contribution of [the component name] is the most significant, with its Dice index reaching 79.3%, followed by [the component name]. fmpc L dfmpc2 The Dice index reached 78.8% and 78.4% respectively, compared to L... dpc The improvement in model performance is minimal, likely because FPSE is only affected by data perturbation and cannot fully extract effective information from unlabeled data. By comparing the results of experiments 11, 12, 13, and 14 with those of experiment 15, it can be observed that missing L... dfmpc1 This will lead to a 1.1% decrease in model performance metrics, which is a significant impact; while missing L... fmpc This will cause a 0.9% decrease in the model's Dice metric, which is also a significant impact. In comparison, L... dpc and L dfmpc2 The absence of L had a relatively small impact on the model's performance, decreasing by only 0.3%. Based on the above analysis, L... dpc L fmpc L dfmpc1 L dfmpc2 All of them contribute to the performance metrics of FPSE, among which L dfmpc1 The largest contribution to model performance is made by L. fmpc and L dfmpc2 Relatively speaking, L dpc Its contribution was the smallest.

[0030] Table 4 Ablation experiments of the fully perturbation-based consensus learning strategy on 5% labeled ACDC training samples. To address the performance issues of Transformer in semi-supervised learning on limited labeled medical image data, this embodiment designs a CPCT structure. This structure allows Transformer to model global dependencies based on local features extracted by CNN, thereby reducing its reliance on large amounts of labeled medical image data. Secondly, to fully utilize the rich information contained in unlabeled data, a fully perturbation-based consistency learning strategy is designed. The core idea is to introduce multiple perturbation strategies during training, allowing FPSE to explore the data from different perspectives and making the perturbed predictions more consistent, thus improving FPSE's generalization ability. Experiments demonstrate that the proposed semi-supervised medical image segmentation framework FPSE based on cascaded parallel CNN-Transformer achieves excellent results on different datasets and tasks, especially when the amount of labeled data is very limited, fully validating the effectiveness of FPSE.

Claims

1. A semi-supervised medical image segmentation method based on cascaded parallel CNN-Transformer and fully perturbation, characterized in that, Includes the following steps: Step 1: Preprocess the original medical image dataset; Step 2: Combining the encoder based on the cascaded parallel CNN-Transformer structure and the fully perturbed consistency learning strategy, a fully perturbed self-integration framework FPSE based on the cascaded parallel CNN-Transformer is constructed. First, the shallow local features of the medical image are extracted through CNN, and then the deep local and global features of the medical image are extracted through the dual-branch network of CNN and Transformer, respectively. The specific process is as follows: Step 2.1: Construct a cascaded parallel CNN-Transformer structure CPCT. It first uses the first five layers of the ResNet50 model to extract shallow local features of medical images, and then uses the UNet model and the Swin-Unet model in parallel to extract deep local features and global features of medical images respectively. The Swin-Unet model performs global dependency modeling based on the shallow local features extracted by the ResNet50 model. Step 2.2: Combine data perturbation, model perturbation, and feature perturbation to construct a full perturbation strategy, wherein: the data perturbation refers to simultaneously performing strong and weak enhancement operations on the original data X to obtain strongly enhanced data X. S And weakly augmented data X W Then X S X W The FPSE is trained along with X; the model perturbation refers to using the cascaded parallel CNN-Transformer structure as the encoder, and using two UNet decoders with the same structure but different parameter initializations in the FPSE decoder part to achieve perturbation of the FPSE structure and parameters; the feature perturbation refers to randomly applying Dropout technology to the output features of the encoder CNN branch and the Transformer branch respectively. Step 2.3: Based on the full perturbation strategy, data perturbation consistency, feature-model perturbation consistency, and data-feature-model perturbation consistency are combined to construct a full perturbation consistency learning strategy. Specifically: data perturbation consistency refers to the CNN subnet and Transformer subnet predicting the original data and augmented data respectively, and applying consistency constraints to the prediction results of the same subnet; feature-model perturbation consistency refers to the CNN subnet and Transformer subnet using the prediction results of the original data as pseudo-labels for mutual pseudo-supervised training; and data-feature-model perturbation consistency includes data-feature-model perturbation consistency between the original data and augmented data, as well as data-feature-model perturbation consistency between weakly and strongly augmented data. Step 2.4: Combining the fully perturbation consistency learning strategy, and based on the cascaded parallel CNN-Transformer structure CPCT, a fully perturbation self-integration framework FPSE based on cascaded parallel CNN-Transformer is constructed. Step 3: Configure the parameters required for FPSE training, perform semi-supervised training on FPSE using the preprocessed medical image dataset, and update the optimal weights and parameters of FPSE. Step 4: Perform medical image segmentation on the trained FPSE and output the prediction results.

2. The semi-supervised medical image segmentation method based on cascaded parallel CNN-Transformer and full perturbation as described in claim 1, characterized in that, The medical image dataset in step 1 includes the public datasets ACDC and AbdomenCT-1K; The ACDC dataset contains short-axis cardiac magnetic resonance imaging images of 100 subjects. For each image, segmentation labels for the left ventricle, myocardium, and right ventricle are provided. The short-axis cardiac magnetic resonance imaging images of the 100 subjects are randomly divided into training, validation, and test sets in a ratio of 7:1:

2. The AbdomenCT-1K dataset contains 41 annotated cases, with segmentation labels for the liver, kidney, spleen, and pancreas provided for each image. The 41 cases are randomly divided into training, validation, and test sets in a ratio of 30:4:

7.

3. The semi-supervised medical image segmentation method based on cascaded parallel CNN-Transformer and full perturbation as described in claim 2, characterized in that, Step 1 preprocesses the medical image dataset by random cropping, random scaling, horizontal flipping, vertical flipping, and adding Gaussian perturbation, thereby expanding the original medical image dataset.

4. The semi-supervised medical image segmentation method based on cascaded parallel CNN-Transformer and full perturbation as described in claim 3, characterized in that, The random cropping involves cropping the original medical image to a size of 224*224.

5. The semi-supervised medical image segmentation method based on cascaded parallel CNN-Transformer and full perturbation as described in claim 3, characterized in that, The random scaling refers to scaling the original medical image by a ratio of 0.8 to 1.

25.

6. The semi-supervised medical image segmentation method based on cascaded parallel CNN-Transformer and full perturbation as described in claim 3, characterized in that, The probabilities of horizontal flipping, vertical flipping, and adding Gaussian perturbation are all 0.

5.

7. The semi-supervised medical image segmentation method based on cascaded parallel CNN-Transformer and full perturbation as described in claim 1, characterized in that, Step 2.1 discards the first two layers of the UNet and Swin-Unet models.

8. The semi-supervised medical image segmentation method based on cascaded parallel CNN-Transformer and full perturbation as described in claim 1, characterized in that, The process of constructing the fully perturbation-consistent learning strategy in step 2.3 is as follows: Step 2.3.1: Establish data perturbation consistency FPSE for input unlabeled data X U Enhanced Data X S And weakly augmented data X W Four different predictions are generated, represented as follows: In the formula, , Representing CNN subnets and Transformer subnet The prediction results of the original data, express Prediction results for strongly enhanced data, express For the prediction results of weakly augmented data, the prediction results generated from the original data are used as pseudo-labels to perform pseudo-supervision on the augmented data, so that the prediction results of the augmented data and the original data tend to be consistent. The generated pseudo-labels are represented as follows: In the formula, , Let these be the pseudo-labels generated from strongly augmented data and weakly augmented data, respectively. The DPC loss can be defined as: In the formula, L dice It is the standard dice loss function, L dpc_C and L dpc_T The DPC loss for the CNN subnet and the Transformer subnet are respectively, L dpc This is the total DPC loss; Step 2.3.2: Construct feature-model perturbation consistency based on and For the prediction of the original data, at this point, and To each and Since pseudo-supervision is performed, the FMPC loss can be defined as: In the formula, L fmpc_C and L fmpc_T The FMPC loss for the CNN subnet and the Transformer subnet are respectively, L fmpc Total FMPC loss; Step 2.3.3: Constructing Data-Feature-Model Perturbation Consistency Step 2.3.3.1: The data-feature-model perturbation consistency between the original data and the augmented data is denoted as the first group. The prediction results of the CNN subnet on the original data are used as pseudo-labels to perform pseudo-supervision on the prediction results of the Transformer subnet on the weakly augmented data. Simultaneously, the prediction results of the Transformer subnet on the original data are used as pseudo-labels for the prediction results of the CNN subnet on the strongly augmented data. The DFMPC loss between the original data and the augmented data can be defined as: In the formula, L dfmpc_C1 and L dfmpc_T1 The first set of DFMPC losses for the CNN subnet and the Transformer subnet are respectively, L dfmpc1 This is the total DFMPC loss for the first group; Step 2.3.3.2: The data-feature-model perturbation consistency between weakly and strongly enhanced data is designated as the second group. This group is trained using a CNN subnet and a Transformer subnet, respectively, with mutual pseudo-supervision training on the prediction results of the strongly and weakly enhanced data. and For predictions based on augmented data, the generated pseudo-labels are represented as follows: In the formula, , The pseudo-labels generated from strongly augmented data and weakly augmented data are represented respectively. The second set of DFMPC losses can be defined as follows: In the formula, L dfmpc_C2 and L dfmpc_T2 The second set of DFMPC losses for the CNN subnet and the Transformer subnet are respectively, L dfmpc2 This is the total DFMPC loss for the second group; Step 2.3.3.3, the total DFMPC loss for the first and second groups is expressed as: 。 9. The semi-supervised medical image segmentation method based on cascaded parallel CNN-Transformer and full perturbation as described in claim 1, characterized in that, Step 3, which involves semi-supervised training of the FPSE, is as follows: Step 3.1: Input the data into the FPSE, which includes 8 labeled medical image data and 8 unlabeled medical image data; Step 3.2: For labeled data, the traditional supervised training method is used, that is, the FPSE prediction results for labeled data are used to calculate Dice loss and cross-entropy loss with the real labels and implement consistency constraints; for unlabeled data with multiple perturbations, FPSE generates multiple different prediction results, and a full perturbation consistency learning strategy is used to train multiple different prediction results, thereby implementing consistency constraints on multiple different prediction results. Step 3.3: After tens of thousands of training iterations, the SGD optimizer with momentum of 0.9 and weight decay of 0.0001 is adopted, and the learning rate is adjusted by a multivariate learning rate strategy to continuously optimize the weights and parameters of FPSE.

Images