Semi-supervised medical image segmentation method based on data and model collaborative learning

Abstract

The invention discloses a semi-supervised medical image segmentation method based on data and model collaborative learning. The method comprises the steps of preprocessing medical image data and dividing the medical image data into a training set, a verification set and a test set; constructing a data and model collaborative learning model, and inputting the training set and the verification set into the data and model collaborative learning model for training and verification to obtain an optimal weight; inputting the test set into the data and model collaborative learning model with the optimal weight, and outputting a medical image segmentation result; the data collaborative learning module performs bidirectional copying and pasting on the labeled data and the unlabeled data by using an N-fold local copying and pasting technology so as to generate mixed data; and a model collaborative learning module is constructed to generate reliable pseudo labels for the unlabeled data through pixel-by-pixel competition. The problems that in an existing semi-supervised medical image segmentation method, noise of label-free data interferes with model training, and empirical distribution between the label-free data and the label data is not matched are solved.

Description

Technical Field

[0001] The present invention relates to the interdisciplinary field of computer science, medical image processing and machine learning, and in particular to a semi-supervised medical image segmentation method for collaborative learning of data and models. Background Art

[0002] Medical image segmentation technology plays a crucial role in modern medicine, with applications spanning a wide range of fields, including lesion localization, condition assessment, surgical planning, drug delivery, and real-time surgical navigation. By combining medical image segmentation with computer vision and deep learning, doctors can more accurately diagnose diseases, formulate treatment plans, and improve surgical precision and safety. However, current medical image segmentation technology still faces the following challenges. First, the complexity of medical images makes the segmentation task extremely challenging. Medical images are often highly variable, and images acquired from different patients, instruments, and imaging conditions can differ significantly. This diversity complicates the development of universal medical image segmentation algorithms, as algorithms must be robust enough to cope with these variations. Second, the annotation process for medical images requires experienced physicians, which is expensive and time-consuming. Furthermore, due to the complexity of medical images, the annotation process is subject to subjectivity and uncertainty, which can lead to inconsistent labeled data. Third, labeled data for some diseases is difficult to obtain. Some diseases require invasive testing or surgery to obtain accurate labeled information.

[0003] To overcome this problem, semi-supervised learning methods have emerged in the field of medical image segmentation. Semi-supervised learning aims to leverage information from both labeled and unlabeled data to train models, reducing the need for large amounts of labeled data. In medical image segmentation, labeled data is typically manually labeled by professional physicians, while unlabeled data consists of unlabeled, unprocessed medical images. This approach offers an innovative solution for medical image segmentation and is expected to play a significant role in improving diagnosis and treatment.

[0004] Currently, many semi-supervised methods have been proposed and applied to medical image segmentation. For example, Bai et al. proposed a method called Semi-FCN in their paper to self-train cardiac magnetic resonance image segmentation by integrating labeled and unlabeled data during training. Yu et al. proposed a semi-supervised segmentation model with uncertainty-aware MT for left atrium segmentation. Li et al. proposed a semi-supervised segmentation method for skin lesions by enforcing consistency between the student and teacher models. Chen et al. proposed a cross-pseudo-supervision method based on network perturbations to encourage high consistency in the prediction results of the two perturbed networks. Sun et al. developed a hierarchical attention module in which the teacher model outputs pseudo-labels and the student model is jointly trained using the true and pseudo-labels. Li et al.'s MT framework optimizes the network through a weighted combination of conventional supervision loss and regularization loss. In addition, Li et al. further strengthened the shape constraint and introduced a signed distance map (SDM) to improve performance. Luo et al. proposed a dual-task consistency semi-supervised framework that combines the predicted segmentation map with geometry-aware level set regularization constraints and studied multi-scale consistency semi-supervised models for medical volume data segmentation. Furthermore, Xia et al. employed a multi-view co-training strategy for ensemble learning of 3D medical image segmentation. In this study, the authors employed a joint training framework to enhance each sub-model for kidney tumor segmentation and applied adversarial training to the network to constrain the model's output to remain unchanged under different perturbations of the input data. Similar to joint training, Peng et al. proposed a method based on deep segmentation network ensembles, training different models using corresponding subsets of labeled data and exchanging internal information between sub-models using unlabeled images. Xie et al. used an attention mechanism to extract pairwise relationships between labeled and unlabeled data to further address the overfitting problem caused by limited labeled data. Xia et al. developed a unified uncertainty-aware modeling framework for unsupervised domain adaptation and semi-supervised segmentation tasks, integrating mutual information-based regularization and transformation consistency for semi-supervised segmentation. In recent work by Luo et al., the Transformer was introduced for semi-supervised medical image segmentation. These methods all combine labeled and unlabeled data to train robust CNN medical image segmentation models.

[0005] Although the aforementioned semi-supervised methods have achieved good results in the field of medical image segmentation, they still face some challenges. First, the quality and diversity of unlabeled data may have a negative impact on the performance of the model. Unlabeled data usually has varying degrees of noise and pseudo-labels, which may cause the model to be disturbed by inconsistencies during training. Therefore, it is necessary to develop effective methods to process unlabeled data to ensure the robustness of the model. Second, there is an empirical distribution mismatch between the labeled and unlabeled data distributions, which results in the unlabeled data being unable to fully learn common semantic information from the foreground and background of the labeled data. Summary of the Invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to propose a semi-supervised medical image segmentation method for collaborative learning of data and model, which solves the problems of the noise of unlabeled data interfering with model training and the mismatch of empirical distribution between unlabeled data and labeled data in the existing semi-supervised medical image segmentation methods.

[0007] The technical solution adopted in the present invention is:

[0008] A semi-supervised medical image segmentation method for collaborative learning of data and model includes the following steps:

[0009] Preprocess the medical image data and divide it into training set, validation set and test set;

[0010] Constructing a data and model collaborative learning model, inputting the training set and the validation set into the data and model collaborative learning model for training and validation, and obtaining a trained data and model collaborative learning model;

[0011] Inputting the test set into the trained data and model collaborative learning model to output the medical image segmentation result;

[0012] The data and model collaborative learning model includes a data collaborative learning module and a model collaborative learning module, specifically:

[0013] The data collaborative learning module inputs labeled data and unlabeled data, and uses N-fold local copy and paste technology to perform bidirectional copy and paste on the labeled data and unlabeled data to generate mixed data, so that the unlabeled data learns common semantic information from the labeled data, and obtains the prediction results of the mixed data;

[0014] The model collaborative learning module inputs unlabeled data and generates reliable pseudo labels for the unlabeled data through pixel-by-pixel competition; the pseudo labels obtained by the model collaborative learning module are mixed with the real labels of the labeled data using N-fold local copy and paste technology to generate mixed labels, and the mixed labels are used to perform pixel-level semi-supervised training on the prediction results of the mixed data to obtain a trained data and model collaborative learning model.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] The data collaborative learning strategy designed by the present invention utilizes N-LCP technology to generate mixed data. Compared with the traditional method of only mixing labeled-labeled or unlabeled-unlabeled data and being limited to continuous region mixing, there is an empirical distribution difference between labeled and unlabeled data, and it is difficult for unlabeled data to fully learn the semantic information of labeled data. The N-LCP of the present invention can simultaneously mix multiple scattered regions, more effectively reducing the distribution difference between labeled data and newly generated mixed data, so that unlabeled data can more fully learn common semantic information from labeled data, and enhance the generalization ability of the model on unlabeled data. At the same time, the existing pseudo-label generation methods mostly rely on single network model predictions, which are prone to produce unreliable pseudo-labels interfered by noise. The model collaborative learning strategy in the present invention combines the prediction results of two different sub-networks (CNN sub-network and Transformer sub-network), evaluates uncertainty through Monte Carlo random dropout technology, selects low-uncertainty pixels by pixel competition, filters high-uncertainty predictions, and generates more reliable pseudo-labels, thereby improving the overall quality and credibility of pseudo-labels.

[0017] Experiments on multiple medical image datasets (ACDC, AbdomenCT-1K, ISIC2018) show that compared with 16 other semi-supervised methods, the method of the present invention achieves excellent segmentation performance with very limited labeled data, with better performance parameters such as the Dice index, fewer mispredictions and missed detection areas, and segmentation edges that are closer to the true labels, demonstrating the effectiveness and superiority of the method of the present invention in semi-supervised medical image segmentation.

[0018] The present invention has broad application prospects in the field of semi-supervised medical image segmentation in medical image analysis. The method of the present invention not only performs well when labeled data is limited, but also has potential in a wider range of medical image segmentation tasks, helping to improve the accuracy and efficiency of medical diagnosis and treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of the method of the present invention.

[0020] Figure 2This is the overall architecture diagram of the data and model collaborative learning model proposed in this invention, where the blue module represents the CNN model and the red module represents the Transformer model.

[0021] Figure 3 This is a comparison chart between the N-LCP of the present invention and the traditional data mixing method, where Figure 3 (a) is the traditional data mixing method, Figure 3 (b) is the N-LCP technology of the present invention.

[0022] Figure 4 This is the structure diagram of the collaborative learning module of the model of the present invention, where blue represents the CNN model and red represents the Transformer model.

[0023] Figure 5 It is a visual comparison of the results of the ACDC training data with 5% and 10% markings between the present invention and different comparison methods.

[0024] Figure 6 This is a visual comparison of the results of the present invention and different comparison methods on the AbdomenCT-1K training data with 5% labeling and 10% labeling. DETAILED DESCRIPTION

[0025] The present invention will be described in detail below with reference to the accompanying drawings.

[0026] A semi-supervised medical image segmentation method based on collaborative data and model learning utilizes N-fold local copy and paste (N-LCP) technology to generate mixed data and generates reliable pseudo-labels through a collaborative model learning strategy. This method focuses on improving the accuracy and generalization of medical image segmentation in scenarios where labeled data is scarce, addressing the current challenge of disparate distributions between labeled and unlabeled data in medical image segmentation. By integrating knowledge from multiple fields, including deep learning, computer vision, and medical image analysis, this method has broad potential for application in practical medical scenarios, such as medical imaging diagnosis, disease progression tracking, and treatment planning.

[0027] Figure 1 This is a flowchart of the implementation steps of the present invention. To address the distribution mismatch between labeled and unlabeled data and the unreliable pseudo-labeling of unlabeled data, a collaborative learning method on data and model (CoLDM) is proposed. The method of the present invention is described in detail as follows:

[0028] (a) Data collaborative learning strategy for generating mixed data

[0029] Most existing semi-supervised methods use a small amount of labeled data and a large amount of unlabeled data as training data. In the context of semi-supervised medical image segmentation, this data composition model faces numerous challenges. Since the amount of unlabeled data often far exceeds the amount of labeled data, the distribution difference between labeled and unlabeled data is often significant. From the perspective of semantic information transfer, this large distribution difference prevents the model from effectively transferring the semantic information learned from labeled data to unlabeled data. This results in the model being unable to fully extract the potential information from the unlabeled data, thereby limiting the model's learning ability.

[0030] In order to solve this problem, the present invention designs a data collaborative learning strategy, the core idea of ​​which is to fully combine labeled data and unlabeled data to generate mixed data, and then replace the unlabeled data with the mixed data as training data. Compared with unlabeled data, mixed data contains relevant information of labeled data, so the distribution of mixed data is closer to the labeled data. Therefore, the semantic information learned by the model from the labeled data can be more smoothly transferred to the mixed data, thereby effectively enhancing the model's ability to mine potential information from unlabeled data. Specifically, in the training phase for mixed data, the model can more selectively learn and understand important feature information from unlabeled data under the guidance of the semantic information of the labeled data. This enables the model to focus more on capturing various patterns and structures that are closely related to the task, thereby significantly improving the learning efficiency on unlabeled data.

[0031] like Figure 3 As shown, the key to realizing this data collaborative learning strategy lies in the N-LCP technology designed by the present invention, which can efficiently mix labeled data and unlabeled data to generate mixed data. Figure 3 (a) Traditional data blending methods usually focus on blending operations between labeled data and labeled data, or between unlabeled data and unlabeled data, and are limited to continuous areas of the blended data. This method limits the full information interaction between labeled data and unlabeled data, resulting in the failure to effectively integrate and fully utilize the data advantages of both. The N-LCP technology proposed in this invention, such as Figure 3 As shown in (b), it is directly applied between labeled data and unlabeled data, and can support the simultaneous mixing of multiple scattered regions. This mixing method enables the model to fully integrate labeled data and unlabeled data, and then comprehensively absorb and learn diverse valuable information from both.

[0032] The N-LCP technique generates N zero-centered masks of size (H / 3, W / 3) on the unlabeled data, where H and W represent the spatial height and width of the unlabeled data, respectively. These masks are then aggregated into a single mask through a union operation, which identifies whether each pixel comes from labeled data or unlabeled data. Next, based on this mask, the N-LCP technique mixes the labeled data with the unlabeled data to generate mixed data. The specific process is as follows:

[0033]

[0034] The above formula Indicates the multiplication of corresponding pixels. X l 、X unl and M represent the labeled information, unlabeled information, and mask, respectively. Note that the area with a mask value of 0 corresponds to the unlabeled data area, while the area with a mask value of 1 corresponds to the labeled data area.

[0035] like Figure 2 As shown, before using N-LCP technology to mix the labeled data and the unlabeled data, the present invention firstly mixes the unlabeled data X U Perform strong enhancement and weak enhancement operations similar to FixMatch to obtain strong enhancement data X U_S and weakly enhanced data X U_W Among them, strong enhancement operations include methods such as CutMix and AutoAugment. These methods will perform large-scale transformations on the original data, thereby increasing the diversity and complexity of the data. In this way, the model can learn more different data patterns, thereby improving the generalization ability of the model. Weak enhancement operations are relatively mild, such as random rotation, mirroring, cropping, etc., which introduce certain changes without changing the main features of the data, helping the model to better adapt to slight changes in the data.

[0036] Then, the N-LCP technique is applied to the unlabeled data X U , strong enhanced data X U_S and weakly enhanced data X U_W , so that the unlabeled data can be copied and pasted into the labeled data X L In this paper, we get three different mixed data, and get the mixed data CM U , Strongly Enhanced Mixed Data CM U_S and weakly enhanced mixed data CM U_W These three types of mixed data correspond to the original unlabeled data, the strongly enhanced unlabeled data, and the result of mixing the weakly enhanced unlabeled data with the labeled data. The specific process is as follows:

[0037] CM U =fN-LCP (X L ,X U ,M U ) (2)

[0038] CM U_S =f N-LCP (X L ,X U_S ,M U_S ) (3)

[0039] CM U_W =f N-LCP (X L ,X U_W ,M U_W ) (4)

[0040] Where M U 、M U_S and M U_W In the unlabeled data X U , strong enhanced data X U_S and weakly enhanced data X U_W The mask generated on

[0041] Mixed data CM U The data is input into both the CNN sub-network and the Transformer sub-network for processing. The CNN sub-network can effectively extract local features and capture detailed information in the data; while the Transformer sub-network captures long-range correlation information by modeling global dependencies. The combination of the two sub-networks enables the model to analyze the data from two different perspectives, local and global, and ultimately generates the following two prediction results: one is the prediction result P based on local features. U _C , the other is the prediction result P based on the global relationship U_T The specific process is as follows:

[0042]

[0043] Since the CNN sub-network structure is simple and easy to optimize, the CM after strong enhancement processing U_S is input into the CNN sub-network for processing. On the contrary, since the Transformer branch has a complex network structure and is difficult to optimize, the CM processed by the weak enhancement operation U_W It is input into the Transformer sub-network, and the specific prediction process can be described as follows:

[0044]

[0045] Among them, P U_S and P U_WRepresents the CNN sub-network to CM U_S The prediction results of the Transformer sub-network and the CM U_W prediction results.

[0046] During the training process, the true label Y of the labeled data and the pseudo label Y of the unlabeled data PL With the help of masks, they are mixed in a manner similar to that of mixed data generation. This process fully exploits the precise information contained in the labeled data and the pseudo-label information with specific reference value generated after processing the unlabeled data, and generates mixed labels Y U 、Y U_S and Y U_W In this way, a more comprehensive and targeted supervision mechanism can be built for the training of mixed data, effectively improving the learning effect and accuracy of the model during training to cope with complex application scenarios. The generation process of mixed labels can be expressed as follows:

[0047] Y U =f N-LCP (Y,Y PL ,M U ) (7)

[0048] Y U_S =f N-LCP (Y,Y PL ,M U_S ) (8)

[0049] Y U_W =f N-LCP (Y,Y PL ,M U_W ) (9)

[0050] Where Y PL represents the pseudo label generated by unlabeled data, Y represents the real label, and Y U Represented as mixed data CM U Generated mixed label; Y U_S Represented as strongly enhanced mixed data CM U_S Generated mixed label; Y U_W Represented as weakly enhanced mixed data CM U_W Generated mixed labels. It is worth noting that with the mixed data CM U CM U_S and CM U_W The corresponding true labels and pseudo labels are the same. Therefore, the mixed label Y U 、Y U_S and Y U_W The only difference in the generation process is the mask.

[0051] YU 、Y U_S and Y U_W For mixed data CM U CM U_S and CM U_W During the training process, because the mixed data used for model training contains both labeled and unlabeled data, the model can not only learn common semantic information from the labeled data, but also better transfer this information to the unlabeled data, thereby improving the learning effect of the unlabeled data.

[0052] (b) Model collaborative learning strategy for generating pseudo labels

[0053] Existing pseudo-label generation methods face a pressing problem: the generated pseudo-labels are often susceptible to noise, leading to unreliability. This is primarily due to the fact that traditional methods typically rely directly on the prediction results of a single network model to generate pseudo-labels, ignoring the uncertainty factors closely related to the prediction results. This one-sided approach makes it difficult to effectively guarantee the quality of the generated pseudo-labels, thereby affecting the model's training effectiveness.

[0054] In order to solve this problem, the present invention designs a model collaborative learning strategy for generating pseudo labels, which generates pseudo labels by combining the prediction results of two different sub-networks. Specifically, the strategy uses Monte Carlo Dropout technology to evaluate the uncertainty of the prediction results from the CNN sub-network and the Transformer sub-network. These two different sub-networks each have unique structures and advantages, and analyze and process the data from different angles. Therefore, the obtained prediction results also contain rich and diverse information. After calculating the prediction uncertainty of the two sub-networks, the pixel competition link is entered. By comparing the uncertainty pixel by pixel, the pixels with lower uncertainty are selected as the winner pixels, and the pixels with higher uncertainty are marked as loser pixels. Finally, the winner pixels of both sides are selected for integration to eventually form a new prediction result.

[0055] However, during the model collaboration process, although the winner pixels have been selected from the predictions of the two sub-networks through pixel competition, the actual situation is not satisfactory. Some winner pixels still have high uncertainty and may introduce noise and errors, thereby affecting the performance of the entire model and the quality of the segmentation results.

[0056] In order to effectively solve this potential problem, the present invention introduces a threshold value, and predicts and identifies pixels with uncertainty higher than the threshold value as key pixels. Then, the cosine similarity between each key pixel and its adjacent pixels with uncertainty lower than the threshold value in the k×k window is calculated. As an effective indicator to measure the similarity between vectors, cosine similarity can accurately reflect the similarity relationship between the key pixel and its adjacent pixels in the feature space. Then, based on the calculated cosine similarity result, the pixel with the highest cosine similarity to the key pixel is selected to replace the key pixel, thereby obtaining the final pseudo-label. This replacement operation can largely eliminate factors that may cause interference due to high uncertainty, thereby effectively improving the reliability and stability of the pseudo-label.

[0057] Unlabeled data X U They are input into the CNN sub-network and the Transformer sub-network respectively, producing two prediction results as shown below:

[0058]

[0059] Monte Carlo Dropout is used to calculate P C and P T The uncertainties of , in order to assess their reliability, are as follows:

[0060]

[0061] Among them, n represents the number of categories, U C and U T They are P C and P T It is worth noting that since the uncertainty is calculated pixel by pixel, U C 、U T 、P C and P T have the same dimensions. Here, U C and U T Higher values ​​of indicate higher uncertainty. Therefore, we expect U C and U T The smaller the value of , the higher the confidence in the prediction of these pixels.

[0062] The present invention designs a pixel competition method to obtain more reliable pseudo labels. C and U T The pixels with lower uncertainty are selected as winner pixels, while the pixels with higher uncertainty are selected as loser pixels. Finally, these winner pixels guide P C and PT Form a high confidence pixel prediction. Specifically, it can be expressed as:

[0063]

[0064]

[0065] Among them, i∈[0,H], j∈[0,W], H and W represent the spatial height and width of the unlabeled data respectively. Indicated by P C and P T The winner pixel constitutes a new pixel prediction. express uncertainty.

[0066] A threshold following a Gaussian ramp-up paradigm is introduced, which starts from 3 / 4U max Increase to U max , where U max Represents the maximum uncertainty value, Each pixel whose uncertainty is higher than the threshold is predicted and marked as a key pixel. The specific process is as follows:

[0067]

[0068] C represents the set of all key pixels;

[0069] For each key pixel (i, j)∈C, find the pixels with uncertainty lower than the threshold in the corresponding (k×k) window, calculate the cosine similarity between the pixels with uncertainty lower than the threshold and the key pixel, and thus find the pixel with the highest similarity. The specific process is as follows:

[0070]

[0071] in represents a (k×k) window centered at pixel (i, j), S i,j is the set of pixels that meet the conditions within the window;

[0072]

[0073] Where, (i * ,j * ) is the pixel coordinate with the highest cosine similarity to the key pixel (i, j), and the pseudo label P generated by the unlabeled data is obtained. PL , as shown below:

[0074]

[0075] Y PL=argmax(P PL ). (18)

[0076] Generally speaking, the true labels of labeled data are more reliable than the pseudo labels of unlabeled data. Therefore, the coefficient α is used to control the contribution of unlabeled data to the loss function. The semi-supervised loss consists of the following four parts.

[0077]

[0078]

[0079]

[0080]

[0081] Where, L seg It is a linear combination of Dice loss and CE loss, L U_C Indicates mixed label Y U CM with CNN sub-network for mixed data U The prediction result P U_C The semi-supervised loss between U_C Select P U_C The predicted part of the unlabeled data is used to control the proportion of this part in the loss function; L U_C Indicates mixed label Y U CM with CNN sub-network for mixed data U The prediction made P U_C The semi-supervised loss between U_C Will choose P U_C The predicted part of the unlabeled data is used to control the proportion of this part in the loss function; L U_T Indicates mixed label Y U CM with Transformer subnetwork for mixed data U The prediction made P U_T The semi-supervised loss between U_S Indicates mixed label Y U_S CM with CNN sub-network for strongly enhanced mixed data U_S The prediction made P U_S The semi-supervised loss between U_W Indicates mixed label Y U_W CM with Transformer subnetwork for weakly enhanced mixed data U_W The prediction made P U_W The semi-supervised loss between .

[0082] The effects of the present invention can be further illustrated by the following experiments.

[0083] In order to verify the accuracy and superiority of the data and model collaborative learning strategy proposed in the present invention, the present invention uses the following hardware and software experimental platform: in terms of hardware, the present invention adopts an Intel Core i9-9900X 3.5GHz CPU and a NVIDIAGeForce RTX 3090Ti GPU with a video memory of 24GB; in terms of software, the present invention uses open source software and frameworks such as PyTorch, Python, OpenCV, and NumPy. In the training phase of the model, the data loading method of the present invention is as follows: the Batch size of the 2D data is set to 16, and horizontal and vertical flips are performed with a probability of 0.5, followed by random center cropping according to a certain size. After that, the present invention performs contrast-limited histogram equalization and Gaussian perturbation with a probability of 0.5, and finally performs image standardization according to the mean and variance of the image. The present invention uses a simple stochastic gradient descent SGD (Stochastic Gradient Descent, SGD) optimization algorithm to train the model, wherein the momentum is set to 0.9, the weight_decay is set to 0.001, the initial learning rate is 0.0001, and the decay rate is set to 0.1.

[0084] In order to verify the superiority of the proposed method, a comprehensive comparison was conducted with 15 mainstream semi-supervised medical image segmentation methods under two different settings, including MT, UA-MT, CCT, CPS, URPC, CAA-ViT, MC-Net+, CTCT, DSTP, MCF, 3D-ViT, AC-MT, BCP, EVIL and CroCT. For ease of comparison, all comparison methods use UNet as the backbone network. Table 1 shows the results of the comparison of the proposed method with other methods on the ACDC dataset using 5% and 10% labeled data respectively. The following two indicator parameters were calculated on the test dataset: Dice similarity coefficient (Dice) and 95% Hausdorff Distance (95HD).

[0085] Table 1

[0086]

[0087]

[0088] Table 1 shows the experimental results of reasoning on the test set with only 5% and 10% labeled data. As shown in Table 1, compared with the other 15 comparison methods, the proposed CoLDM achieves the best performance with both 5% and 10% labeled data. In particular, when trained with only 5% labeled data, CoLDM's Dice metric is 8.2%, 9.1%, 7.6%, 5.4%, 5.9%, 12.8%, 4.0%, 2.9%, 2.4%, and 1% higher than the MT, UA-MT, CCT, CPS, URPC, CAA-ViT, MC-Net+, CTCT, DSTP, and BCP methods trained with 10% labeled data, respectively. Compared with the training data with 5% labeled data, the performance of CPS and CCT improved by 23.5% and 23% respectively when using 10% labeled data, while CoLDM only improved by 3.4%. This shows that CoLDM has fully mined the effective information of unlabeled data when using 5% labeled data, rather than relying solely on labeled data. This further proves that the CoLDM model can effectively utilize unlabeled data in semi-supervised medical image segmentation. In addition, Figure 5 Visualizations of inference results on the test set after training with 5% labeled data and 10% labeled data are shown. These images clearly show that the CoLDM method achieves superior segmentation results. Compared to other methods, CoLDM exhibits fewer mispredictions and missed detections, and its segmentation edges align more closely with the labels. Notably, when trained with only 5% labeled data, other methods exhibit a significant number of mispredictions and missed detections, while CoLDM maintains relatively good segmentation results.

[0089] Table 2 shows the comparison results of the present invention with other methods on the AbdomenCT-1K dataset using 3% and 10% labeled data, respectively. The following two index parameters are calculated on the test dataset: Dice similarity coefficient (Dice) and 95% Hausdorff Distance (95HD).

[0090] Table 2

[0091]

[0092] As shown in Table 2, CoLDM also achieves the best results under both ratios of labeled training data. In particular, when only 5% of the labeled data is used for training, the indicators are improved by 1.7% and 1.5% respectively compared with the other best methods. It is even 6.0%, 4.4%, 5.3%, 4.9%, 4.1%, 6.5%, 4.6%, 6.5%, 6.0%, and 0.2% higher than the MT, UA-MT, CCT, CPS, URPC, CAA-ViT, MC-Net+, CTCT, DSTP, and BCP methods trained with 10% of the labeled data, which further verifies the superiority of CoLDM. Figure 6 The following visualizations show the results of various methods on the 3% and 10% labeled AbdomenCT-1K training data. As can be seen, the other comparison methods often miss the liver and pancreas, or misidentify the spleen as the liver or the spleen as the kidney. In contrast, CoLDM demonstrates more reliable predictions on both the 3% and 10% labeled data, and better preserves anatomical morphology, fully demonstrating the effectiveness of the CoLDM method.

[0093] As shown in Table 3, to verify the effectiveness of the N-LCP technology, the present invention replaces the N-LCP technology with mixup, PatchUp, and FMix respectively to conduct ablation studies as shown in Table 3. Table 3 shows the results of the ablation experiment on N-LCP on 5% labeled data. The following two indicator parameters are calculated on the test dataset: Dice similarity coefficient (Dice) and 95% Hausdorff Distance (95HD).

[0094] As shown in Table 3, the proposed N-LCP technique achieves the best performance of 0.892 and 2.2 on Dice and 95% HD, respectively. This demonstrates that, compared to other hybrid data generation strategies, the N-LCP technique has advantages in improving segmentation quality and preserving details, providing an effective hybrid data generation method for semi-supervised medical image segmentation.

[0095] Table 3

[0096]

[0097] To verify the effectiveness of the model collaborative learning strategy in generating pseudo labels, the present invention designed an ablation experiment as shown in Table 4. The present invention compares the model collaborative learning strategy with other different pseudo label generation strategies, including using the CNN sub-network's prediction results PC as pseudo labels, using the Transformer sub-network's prediction results PT as pseudo labels, and using the average of the prediction results of the CNN and Transformer sub-networks ((PC + PT) / 2) as pseudo labels. Table 4 shows the ablation experiment results of the model collaborative learning strategy on 5% labeled training samples. The following two indicator parameters are calculated on the test dataset: Dice similarity coefficient (Dice) and 95% Hausdorff Distance (95HD).

[0098] As can be seen from Table 4, the pseudo labels generated by the model collaborative learning strategy proposed in this paper show the best performance, proving its effectiveness.

[0099] Table 4

[0100]

Claims

1. A semi-supervised medical image segmentation method based on collaborative learning of data and model, characterized in that: The following steps are involved: Preprocess the medical image data and divide it into training set, validation set and test set; Constructing a data and model collaborative learning model, inputting the training set and the validation set into the data and model collaborative learning model for training and validation, and obtaining a trained data and model collaborative learning model; Inputting the test set into the trained data and model collaborative learning model to output the medical image segmentation result; The data and model collaborative learning model includes a data collaborative learning module and a model collaborative learning module, specifically: The data collaborative learning module inputs labeled data and unlabeled data, and uses N-fold local copy and paste technology to perform bidirectional copy and paste on the labeled data and unlabeled data to generate mixed data, so that the unlabeled data learns common semantic information from the labeled data, and obtains the prediction results of the mixed data; The collaborative learning module of the model inputs unlabeled data and generates reliable pseudo labels for the unlabeled data through pixel-by-pixel competition; The pseudo labels obtained by the model collaborative learning module are mixed with the real labels of the labeled data using the N-fold local copy and paste technology to generate mixed labels, and the mixed labels are used to perform pixel-level semi-supervised training on the prediction results of the mixed data to obtain a trained data and model collaborative learning model.

2. A semi-supervised medical image segmentation method for collaborative learning of data and model according to claim 1, characterized in that: The N-fold local copy-paste technique is specifically as follows: N masks of size H / 3 and W / 3 and centered on zero are generated on the unlabeled data, where H and W represent the spatial height and width of the unlabeled data, respectively; all masks are aggregated into a total mask through a union operation, and the mask is used to identify whether each pixel comes from labeled data or unlabeled data; based on the mask, the labeled data is mixed with the unlabeled data to generate mixed data, and the process is as follows: Where, Indicates the multiplication of corresponding pixels, X l 、X unl and M represent the labeled information, unlabeled information, and mask, respectively. The area with a mask value of 0 corresponds to the unlabeled data area, and the area with a mask value of 1 corresponds to the labeled data area.

3. The semi-supervised medical image segmentation method for collaborative learning of data and model according to claim 2, characterized in that: The specific steps of the data collaborative learning module are as follows: For unlabeled data X U Perform strong enhancement and weak enhancement operations to obtain strong enhancement data X U_S and weakly enhanced data X U_W ; The unlabeled data X will be copied and pasted using the N-fold local copy and paste technique U , strong enhanced data X U_S and weakly enhanced data X U_W To the labeled data X L In the equation, we get the mixed data CM U , Strongly Enhanced Mixed Data CM U_S and weakly enhanced mixed data CM U_W , as follows: CM U =f N-LCP (X L ,X U ,M U ) (2) CM U_S =f N-LCP (X L ,X U_S ,M U_S ) (3) CM U_W =f N-LCP (X L ,X U_W ,M U_W ) (4) Where M U 、M U_S and M U_W In the unlabeled data X U , strong enhanced data X U_S and weakly enhanced data X U_W The mask generated on Mixed data CM U At the same time, it is input into the CNN sub-network and the Transformer sub-network for processing to obtain the prediction result P based on local features. U_C And the prediction result P based on global relationship U_T , as follows: Strongly enhance the mixed data CM U_S Input into the CNN sub-network for processing, and the weakly enhanced mixed data CM U_W Input into the Transformer sub-network for processing to obtain the prediction result P U_S and P U_W , as follows:

4. The semi-supervised medical image segmentation method for collaborative learning of data and model according to claim 3, characterized in that: The specific steps of the model collaborative learning module are as follows: The unlabeled data X U Input them into the CNN sub-network and Transformer sub-network respectively, and get the CNN sub-network prediction result P C and Transformer subnetwork P T , as follows: Monte Carlo random dropout is used to calculate the CNN sub-network prediction result P C and Transformer subnetwork P T The uncertainties are as follows: In the formula, n represents the number of categories, U C and U T They are P C and P T Uncertainty map of U C and U T Higher values ​​of indicate higher uncertainty; in U C and U T A pixel-by-pixel comparison is performed between the two pixels, and the pixel with lower uncertainty is selected as the winner pixel, and the pixel with higher uncertainty is selected as the loser pixel. The winner pixel guides P C and P T Forming a high-confidence pixel prediction can be specifically expressed as: Where i∈[0,H], j∈[0,W], H and W represent the spatial height and width of the unlabeled data respectively. Indicated by P C and P T The winner pixel constitutes a new pixel prediction, express uncertainty; A threshold following a Gaussian ramp-up paradigm is introduced, which starts from 3 / 4U max Increase to U max , where U max Represents the maximum uncertainty value, Each pixel whose uncertainty is higher than the threshold is predicted and marked as a key pixel. The specific process is as follows: C represents the set of all key pixels; For each key pixel (i, j)∈C, find the pixels with uncertainty lower than the threshold in the corresponding (k×k) window, calculate the cosine similarity between the pixels with uncertainty lower than the threshold and the key pixel, and thus find the pixel with the highest similarity. The specific process is as follows: in represents a (k×k) window centered at pixel (i, j), S i,j is the set of pixels that meet the conditions within the window; Where, (i * ,j * ) is the pixel coordinate with the highest cosine similarity to the key pixel (i, j), and the pseudo label P generated by the unlabeled data is obtained. PL , as shown below: Y PL =argmax(P PL )。 (19) 5. The semi-supervised medical image segmentation method for collaborative learning of data and model according to claim 4, characterized in that: According to the N-fold local copy and paste technique, the generation process of mixed labels can be expressed as follows: Y U =f N-LCP (Y,Y PL ,M U ) (20) Y U_S =f N-LCP (Y,Y PL ,M U_S ) (21) Y U_W =f N-LCP (Y,Y PL ,M U_W ) (22)In the formula, Y PL represents the pseudo label generated by unlabeled data, Y represents the real label, and Y U Represented as mixed data CM U Generated mixed label; Y U_S Represented as strongly enhanced mixed data CM U_S Generated mixed label; Y U_W Represented as weakly enhanced mixed data CM U_W The generated mixed labels.

6. The semi-supervised medical image segmentation method for collaborative learning of data and model according to claim 5, characterized in that: The pixel-level semi-supervised training of the prediction results of the mixed data using the mixed labels is specifically as follows: The coefficient α is used to control the contribution of unlabeled data to the loss function. The semi-supervised loss consists of the following four parts. Where, L seg It is a linear combination of Dice loss and CE loss, L U_C Indicates mixed label Y U CM with CNN sub-network for mixed data U The prediction result P U_C The semi-supervised loss between U_C Select P U_C The predicted part of the unlabeled data is used to control the proportion of this part in the loss function; L U_C Indicates mixed label Y U CM with CNN sub-network for mixed data U The prediction made P U_C The semi-supervised loss between U_C Will choose P U_C The predicted part of the unlabeled data is used to control the proportion of this part in the loss function; L U_T Indicates mixed label Y U CM with Transformer subnetwork for mixed data U The prediction made P U_T The semi-supervised loss between U_S Indicates mixed label Y U_S CM with CNN sub-network for strongly enhanced mixed data U_S The prediction made P U_S The semi-supervised loss between U_W Indicates mixed label Y U_W CM with Transformer subnetwork for weakly enhanced mixed data U_W The prediction made P U_W The semi-supervised loss between .

Images