A heterogeneous perception-based mutual learning semi-supervised 3D medical image segmentation method

Abstract

The application discloses a kind of based on heterogeneous perception's mutual learning semi-supervised 3D medical image segmentation method, to more fully exploit and utilize unlabeled data, to realize the promotion of segmentation performance. First, the embedded uncertain perception model, in single-step forward propagation process real-time evaluation potential error prediction area, and by constructing difference mask and optimizing correction loss function, improve the understanding and judgment ability of model to uncertain area.Second, the algorithm designs multi-view contrast reconstruction model, the module includes two groups of different sub-networks.The two groups of sub-networks accurately locate and focus on optimizing those areas that may exist misclassification by comparing the differences between their respective prediction results.Finally, a mutual consistency learning network model is constructed, including entropy increase guidance, cross-pseudosupervision, and fine directional consistency optimization for fuzzy areas and decision boundaries, effectively utilizing the large amount of information contained in unlabeled data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning of computer vision processing, and particularly relates to a mutual learning semi-supervised 3D medical image segmentation method based on heterogeneous perception. BACKGROUND

[0002] With the in-depth application of deep learning in the field of medical image segmentation, the method based on deep learning has made significant progress. However, in practical application, the fine annotation of medical images is a difficult and time-consuming task, which limits the effective use of a large amount of unlabeled data resources. To overcome this bottleneck, semi-supervised segmentation technology is widely used, which can improve model performance with unlabeled data, although the labels generated by semi-supervised learning may be noisy, thereby affecting the accuracy of the final segmentation result. In view of this problem, the consistency regularization method emerges as the times require, which ensures that the model prediction remains internally consistent even after various augmentations are performed on the unlabeled samples, thereby improving the segmentation effect. However, the existing consistency regularization method is often limited by the data augmentation scheme customized for a specific field. For example, when solving the problem caused by the lack of labeled samples in medical image segmentation, the existing method attempts to use unlabeled data joint training to cope with the data shortage condition, but these methods excessively rely on the consistency constraint mechanism between model predictions, and fail to fully focus on and handle the key problem of cognitive uncertainty. SUMMARY

[0003] The purpose of the present application is to overcome the deficiencies in the prior art and provide a mutual learning semi-supervised 3D medical image segmentation method based on heterogeneous perception.

[0004] The purpose of the present application is achieved by the following technical solutions:

[0005] The mutual learning semi-supervised 3D medical image segmentation method based on heterogeneous perception comprises:

[0006] S1. Construct an uncertainty perception evaluation model; specifically comprising:

[0007] S101. Design a computing unit containing four classifiers, which share the same encoder but use different loss functions, including cross-entropy loss, focal loss, Dice loss and IoU loss;

[0008] S102. Through the above classifiers, the certainty region and the uncertainty region are obtained by predicting each voxel, and a weight map with cognitive uncertainty is generated to reduce the additional computational cost.

[0009] S103. For the certainty region, a pseudo-label strategy is used, and the corresponding pseudo-label loss is calculated The consistency loss between the prediction of the student network and the prediction of the teacher network is obtained by predicting the uncertainty region by using the student network and the teacher network

[0010] S104. Defining a total unsupervised loss function An uncertainty perception evaluation model is obtained, and the uncertainty perception evaluation model is optimized for predicting the uncertainty region by combining the pseudo-label loss and the consistency loss;

[0011] S2. Constructing a multi-view contrast reconstruction model, specifically including:

[0012] S201. Performing a binary exclusive-OR operation on the outputs of the student network and the teacher network to obtain a difference mask M diff ;

[0013] S202. Using the difference mask M diff to extract the potential error prediction region of each voxel, and calculating a correction loss function of the potential error prediction region

[0014] S203. Integrating the basic segmentation loss and the correction loss function to obtain a multi-view contrast reconstruction model, and defining a supervised learning loss of the multi-view contrast reconstruction model to improve the performance of the multi-view contrast reconstruction model.

[0015] S3. Constructing and optimizing a mutual consistency learning network model, specifically including:

[0016] S301. Using a cross-pseudo-supervision method to calculate an inter-class similarity penalty loss of the student network and the teacher network and a cross-pseudo-supervision loss to supervise the learning of the teacher and student networks;

[0017] S302. Using a distillation method to calculate a bidirectional KL loss of the student network and the teacher network and a distillation loss obtained by a Dice loss to promote the information exchange between the student network and the teacher network;

[0018] S303. Defining an entropy loss based on confidence combining the cross-pseudo-supervision loss, the bidirectional KL loss, and the distillation loss to optimize the prediction consistency of the student network and the teacher network;

[0019] S304. Decoupling the inconsistent part of the prediction results of the student network and the teacher network into unreliable data and guided data, calculating a directional consistency loss of the student network and the teacher network and optimizing the consistency loss across the pixel space The mutual consistency learning network model is obtained, and the prediction accuracy of the mutual consistency learning network model for pixels around the decision boundary is improved.

[0020] The uncertainty perception evaluation model, the multi-view contrast reconstruction model and the mutual consistency learning network model are combined to mine the uncertainty knowledge of the input data of the student network and the teacher network.

[0021] Further, the total unsupervised loss function in step S104 is The pseudo-label loss and the consistency loss are combined:

[0022]

[0023] Wherein, M c is a binary mask of the region, the voxel with a value of 1 corresponds to a feature part, J is a full one tensor, is a hyperparameter for adjusting the consistency regularization strength, represents the element-wise multiplication operation, and the Mean(·) function is used to calculate the average value of non-zero elements to obtain a scalar result; is the pseudo-label loss.

[0024] Further, in step S201, the softmax output of the unlabeled data part of the student network and the teacher network is subjected to a binary XOR operation, and the difference mask M diff The calculation formula is as follows:

[0025]

[0026] Wherein, BINA(·) represents a binary function, is an XOR operation;

[0027] The error prediction region is extracted; the difference mask M diff is used to extract the error prediction region, and the correction loss function

[0028]

[0029] Wherein, represents the true value of the labeled data, and MASK(·) represents the predicted value identified by the difference mask M diff represents the potential error prediction region.

[0030] Further, in step S203, the supervised learning loss The calculation formula is as follows:

[0031] ​

[0032] wherein, Y denotes the ground truth of labeled data, L Y denotes the network prediction result of labeled data, represents the potential error prediction area, and β is used to balance the importance of the reconstruction loss and other segmentation loss terms; the Dice loss and the cross-entropy CE loss jointly constitute the basic segmentation loss, and the correction loss function is which helps to improve the recognition ability of the multi-view contrast reconstruction model for potential error prediction.

[0033] Further, the inter-class similarity penalty loss in step S3 is The calculation formula is as follows:

[0034]

[0035] wherein, Xi denotes the i-th data of the data set, is a cross-entropy loss function, and p 1i and p 2i respectively represent the prediction results of the student network and the teacher network, and represent the ground truth label, and WxHxD respectively represent the width, height and depth of the input image;

[0036] By applying the cross-pseudo supervision loss in both directions to improve the learning effect of the student network and the teacher network on unlabeled data;

[0037] Calculate the bidirectional KL loss and the distillation loss; for the student network and the teacher network Their original predictions at the same pixel point x i are denoted as The probability distribution is calculated by the softmax function:

[0038]

[0039] Calculate the bidirectional KL loss and the distillation loss obtained by the Dice loss

[0040]

[0041]

[0042] wherein, p1 and p2 respectively represent the posterior probability prediction of the student and the teacher network; the Dice loss is used to replace the bidirectional KL divergence loss to alleviate the class imbalance problem, and the distillation loss is formed, wherein T represents the parameter temperature setting;

[0043] define the entropy loss based on confidence

[0044]

[0045] By decoupling the inconsistent part of the student network and the teacher network prediction result as unreliable data and guide data, the direction consistency loss is calculated

[0046]

[0047] Wherein, sp is the sharpening operation, sp 1 and sp 2 Comparison, get high confidence part (hsp 1 , hsp 2 ) and low confidence part (lsp 1 , lsp 2 ); Only the low confidence part is applied to the mean square error L2 loss as the optimization target, detach does not backpropagate the gradient of the high confidence part;

[0048] Optimize the consistency loss across the pixel space To improve the prediction accuracy of the mutual consistency learning network model for the pixels around the decision boundary;

[0049]

[0050] Wherein, PL1 and PL2 represent the corresponding pseudo label respectively, f1(x i ) and f2(x i ) represent the prediction output results of the student and teacher network, and CE represents the cross entropy loss.

[0051] The application also provides a mutual learning semi-supervised 3D medical image segmentation system based on heterogeneous perception, comprising:

[0052] The uncertainty perception evaluation module is used for evaluating the potential error prediction area in real time in the single-step forward propagation process, and improving the understanding and judgment ability of the uncertain area by constructing the difference mask and optimizing the correction loss function.

[0053] The multi-view contrast reconstruction module comprises two groups of subnetworks with different structures, and the two groups of subnetworks locate and optimize the region that may exist in the misclassification by comparing the difference between the respective prediction results.

[0054] The mutual consistency learning module is based on entropy loss, cross pseudo supervision loss, bidirectional KL loss and distillation loss to optimize the prediction consistency of the student network and the teacher network.

[0055] The application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for mutual learning semi-supervised 3D medical image segmentation based on heterogeneous perception when executing the program.

[0056] The application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method for mutual learning semi-supervised 3D medical image segmentation based on heterogeneous perception.

[0057] Compared with the prior art, the application effectively solves the problem of insufficient attention to cognitive uncertainty when existing semi-supervised learning methods process unlabeled data by introducing a heterogeneous perception and mutual learning mechanism, thereby significantly improving the accuracy and robustness of medical image segmentation. The specific beneficial effects brought by the technical scheme of the application are as follows:

[0058] 1. Improved segmentation accuracy: by introducing a heterogeneous perception mechanism, the application can effectively identify and process noisy labels in data, significantly reducing segmentation errors caused by noisy labels and improving segmentation accuracy.

[0059] 2. Enhanced model robustness: using a mutual learning mechanism, the application exchanges information and mutually supervises between different models, enhancing the adaptability of the model to different types of data and improving the robustness of the overall model.

[0060] 3. Make full use of unlabeled data: the semi-supervised learning algorithm of the application can effectively utilize a large amount of unlabeled data, further improving the performance of the model in processing unlabeled data through consistency regularization and data augmentation strategies.

[0061] 4. Reduce the cost of manual annotation: since the algorithm can effectively learn on unlabeled data, the need for a large amount of fine annotation data is reduced, thereby significantly reducing the cost and time investment of manual annotation.

[0062] 5. Improve the consistency of the model: through the consistency regularization method, the model prediction remains internally consistent even after various augmentations on unlabeled samples, improving the reliability and stability of the segmentation results.

[0063] 6. Adapt to different domain data augmentation: the method of the application is no longer limited to domain-specific data augmentation schemes, and has strong universality and adaptability, and can be widely applied to different types of medical image segmentation tasks.

[0064] Based on the above technical means, the present application significantly improves the segmentation effect when solving the problem caused by the lack of labeled samples in medical image segmentation, and provides a more efficient and reliable solution for medical image analysis. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 A flowchart of the method of the present application.

[0066] Figure 2 A general framework diagram of the present application based on heterogeneous perception mutual learning semi-supervised medical image segmentation algorithm. DETAILED DESCRIPTION

[0067] The present application will be further described in detail below in conjunction with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0068] As shown in Figure 1 , the present application provides a mutual learning semi-supervised 3D medical image segmentation method based on heterogeneous perception, comprising:

[0069] S1. Construct an uncertainty perception evaluation model; specifically comprising:

[0070] S101. Design a computing unit containing four classifiers, which share the same encoder but use different loss functions, including cross-entropy loss, focal loss, Dice loss and IoU loss;

[0071] S102. Through the above classifier, the certainty region and the uncertainty region are obtained by predicting each voxel, and a weight map with cognitive uncertainty is generated to reduce the additional computational cost.

[0072] S103. For the certainty region, use the pseudo-label strategy and calculate the corresponding pseudo-label loss Adopting student network and teacher network to predict the uncertainty region, the consistency loss between student network and teacher network prediction is obtained

[0073] S104. Define the total unsupervised loss function Get the uncertainty perception evaluation model, and combine the pseudo-label loss and the consistency loss to optimize the prediction of the uncertainty perception evaluation model on the uncertainty region;

[0074] S2. Construct a multi-view contrast reconstruction model, specifically comprising:

[0075] S201. Perform binary XOR operation on the output of the student network and the teacher network to obtain the difference mask M diff ;

[0076] S202. Extracting potential misprediction regions of each voxel using the difference mask M diff and calculating the correction loss function of the potential misprediction regions

[0077] S203. Integrating the base segmentation loss and the correction loss function to obtain a multi-view contrastive reconstruction model, and defining a supervised learning loss of the multi-view contrastive reconstruction model to improve the performance of the multi-view contrastive reconstruction model.

[0078] S3. Constructing and optimizing the mutual consistency learning network model, specifically including:

[0079] S301. Using a cross-pseudo-supervised method to calculate the inter-class similarity penalty loss of the student network and the teacher network and the cross-pseudo-supervised loss to supervise the learning of the teacher and student networks;

[0080] S302. Using a distillation method to calculate the bidirectional KL loss of the student network and the teacher network and the distillation loss obtained by the Dice loss to promote the information exchange between the student network and the teacher network;

[0081] S303. Defining an entropy loss based on confidence combining the cross-pseudo-supervised loss, bidirectional KL loss, and distillation loss to optimize the prediction consistency of the student network and the teacher network;

[0082] S304. By decoupling the inconsistent part of the prediction results of the student network and the teacher network into unreliable data and guided data, calculating the directional consistency loss of the student network and the teacher network and optimizing the consistency loss across the pixel space to obtain a mutual consistency learning network model and improve the prediction accuracy of the mutual consistency learning network model for pixels around the decision boundary.

[0083] Integrating the uncertainty perception evaluation model, the multi-view contrastive reconstruction model, and the mutual consistency learning network model to mine the uncertainty knowledge of the input data of the student network and the teacher network.

[0084] Specifically, in the semi-supervised segmentation task, cognitive uncertainty is used to guide learning. Early research usually relies on the Monte Carlo dropout method to generate uncertainty weight maps. However, this method requires multiple forward propagation calculations, greatly increasing the computational cost. To alleviate this burden, the embodiment uses an alternative solution: using a set of four classifiers with a shared encoder with different loss functions. Unlike the method of achieving diversity only by weighting the cross-entropy loss, the four-class classifier of the embodiment shares the underlying encoder structure, but uses different loss functions respectively, including cross-entropy loss, focal loss, dice loss and IoU loss. These differentiated loss functions guide the model to produce prediction results from different optimization perspectives in a single forward propagation process, thereby obtaining pixel-level and geometric-level cognitive uncertainty without increasing computational cost.

[0085] When processing the uncertainty perception evaluation model, the embodiment treats the deterministic region and the uncertainty region differently. The deterministic region is directly trained using pseudo labels, and the prediction consistency between the student network and the teacher network is strengthened in the uncertainty region. Specifically, in the deterministic region, the pseudo label strategy is applied, and the loss tensor generated thereby is denoted as with dimensions HxWxD. For the uncertainty region, a consistency loss is introduced The calculation formula is where represents the prediction value of the student network for the i-th voxel in the j-th classification, and is the corresponding prediction value of the teacher network.

[0086] Based on the above analysis, the total unsupervised loss function based on uncertainty cognitive guidance is defined as follows:

[0087]

[0088] In this expression, M c represents the binary mask of a specific region, where the voxel with a value of 1 corresponds to the specific part; J is an all-1 tensor; is a hyperparameter that adjusts the strength of consistency regularization; ⊙ represents element-wise multiplication operation; The Mean(·) function is used to calculate the average value of non-zero elements to obtain a scalar result.

[0089] In summary, the improved scheme not only reduces the computational overhead, but also effectively captures and utilizes the pixel-level and structure-level cognitive uncertainty in the image, while ensuring that the deterministic region makes full use of the pseudo label information and improving the consistency of the model prediction in the uncertainty region.

[0090] The multi-view contrastive reconstruction model includes two equally-performing subnetworks (i.e., student network and teacher network), which are marked as subnetwork A (denoted as f A (·)) and subnetwork B (denoted as f B (·)), respectively. For the input dataset, the original labeled data (X L , Y L ) and unlabeled data X U will be simultaneously sent into the two subnetworks for processing:

[0091]

[0092]

[0093] In the binary classification scenario, if the same voxel is predicted differently by subnetwork A and B, at least one of the predictions is wrong. Therefore, the embodiment regards the area where the two network prediction results are inconsistent as a potential wrong prediction area, and guides the network to focus on these areas. According to the method in semi-supervised mutual correction learning, the difference mask M diff is obtained by calculating the binary exclusive-OR operation between the softmax outputs of subnetwork A and B for the unlabeled data part , and the expression is as follows:

[0094]

[0095] where BINA(·) represents the binary function, is the exclusive-OR operation. Then, the difference mask is used to reveal the potential wrong prediction area:

[0096]

[0097] where Y L represents the true value of the labeled data, MASK(·) represents the extracted predicted value identified by the difference mask M diff , and M 1i represents the potential wrong prediction area.

[0098] In order to further optimize the multi-view contrastive reconstruction model's awareness of the possible wrong prediction area, a correction loss function L

[0099]

[0100] Here, MSE(·) represents the mean square error loss function, and Y 2i corresponds to the true label of those areas, represents the potential wrong prediction area.

[0101] Finally, the entire supervised learning loss includes the basic segmentation loss and the correction loss, and the form is as follows:

[0102]

[0103] wherein, denotes the ground truth of labeled data, Y L denotes the network prediction of labeled data, denotes the potential false prediction region, and the hyper-parameter β is used to balance the importance of reconstruction loss and other segmentation loss terms. The Dice loss and the cross-entropy loss jointly constitute the basic segmentation loss, and the rectification loss helps to improve the recognition ability of the multi-view contrast reconstruction model for potential false predictions.

[0104] In the process of training the mutual consistency learning network model, the entropy guide network gradually increases to extract more valuable information from the input data. In cross-pseudo supervision, the hard pseudo segmentation map is used as a guide signal to supervise the learning of other segmentation networks. The inter-class similarity penalty loss is calculated as follows:

[0105]

[0106] wherein, denotes the lth data of the data set, is the cross-entropy loss function, p 1i and p 2i denote the prediction results of the network, and represent the ground truth label, and WxHxD are the width, height and depth of the input image, respectively;

[0107] Cross-pseudo supervision is bidirectional, that is, the pixel-level single-label mapping Y1 of one network f(θ1) is used to supervise the pixel-level confidence mapping P2 of another network f(θ2), and vice versa.

[0108] For student and teacher networks Their original predictions at the same pixel point x i are denoted as probability distribution is calculated by the softmax function:

[0109]

[0110] Naturally, bidirectional KL loss can be introduced into mutual learning:

[0111]

[0112] wherein, p1 and p2 represent the posterior probability predictions of the respective networks.

[0113] However, KL divergence loss is difficult to handle the class imbalance problem between foreground and background pixels. Therefore, Dice loss is used to replace the bidirectional KL divergence loss to alleviate this problem, thus forming the distillation loss:

[0114]

[0115] where the temperature parameter T is set to be greater than 1 to expand the deep knowledge distillation, and the distillation probability is calculated:

[0116]

[0117] With the increase of temperature parameter T, the network can exchange more abundant information through high entropy probability, especially in the fuzzy area. The distillation process makes the output p j，T become a soft label, which helps to reduce cognitive bias in the training process.

[0118] Then, the confidence-based entropy loss can be obtained by jointly training this part of unsupervised loss, and its definition can be expressed as:

[0119]

[0120] Finally, this embodiment decouples the inconsistent part into unreliable data that may appear in the decision boundary and guiding data that is more likely to be located near the high-density area. These two parts share the same index information, but the guiding data has higher confidence than the unreliable data. According to the smoothing assumption, it is believed that the outputs of the two should be consistent and close to the high-density area. Therefore, more attention should be paid to optimizing the pixels around the decision boundary to make them closer to the high-density area. First, sharpening operation is used to improve the confidence of these pixels, and the formula is as follows:

[0121]

[0122] where f(x i ) represents the model output, T∈(0, 1) is the sharpening temperature, and T=0.5 is set in the experiment.

[0123] By comparing sp 1 and sp 2 obtained by sharpening operation, high-confidence parts (hsp 1 , hsp 2 ) and low-confidence parts (lsp 1 , lsp 2 ) are obtained. Only the L2 loss is applied to the low-confidence part as the optimization target, and detach means that the gradient is not backpropagated to the high-confidence part. Thus, the direction consistency loss can be expressed as:

[0124]

[0125] Meanwhile, the cross pseudo-supervision method is adopted to optimize the consistency part loss in the cross-pixel space. PL1 and PL2 represent the corresponding pseudo labels, respectively.

[0126]

[0127] where f1(x i ) and f2(x i ) represent the prediction output results of the student and teacher networks, and CE represents the cross-entropy loss.

[0128] Referring to Figure 2 , the overall framework of the embodiment is divided into three parts.

[0129] First, the method of the present application embeds an uncertainty perception evaluation model, which evaluates the potential error prediction area in real time during the single-step forward propagation process, and improves the understanding and judgment ability of the model for uncertain areas through the construction of difference masks and the optimization of correction loss function. Secondly, the method designs a multi-view comparison reconstruction model, which includes two groups of sub-networks with different structures. The two groups of sub-networks accurately locate and focus on optimizing the areas that may have missegmentation by comparing the differences between their respective prediction results. Finally, the mutual consistency learning network model adopts a series of strategies, including entropy increase guidance, cross pseudo-supervision, and fine directional consistency optimization for fuzzy areas and decision boundaries, thereby effectively utilizing the large amount of information contained in the unlabeled data.

[0130] Specifically, in the training link, the left atrium and brain tumor dataset is set to a batch size of 4, each batch contains two labeled data and two unlabeled data, and data augmentation strategies such as random flipping and rotation are used to alleviate the problem of overfitting. For 3D data, the present embodiment selects VNet as the basic network architecture, inserts a dropout layer between the encoder and the decoder, and sets the dropout rate to 0.5. The hyperparameters Follow the time-dependent Gaussian warm-up function, the formula is represented as where t i represents the current training iteration number, and t max is the maximum iteration number.

[0131] On 2D datasets, samples are also standardized to zero mean and unit variance, and random rotation and flipping are adopted to enhance data diversity. Randomly cropped image patches of size 256x256 are used for training, with a batch size of 24, containing 12 labeled and 12 unlabeled samples per batch. In the testing stage, scans are resized to 256x256 as input and upscaled to the original size to obtain the final segmentation results. The 2D model is built based on UNet. All settings on the ACDC dataset follow the public benchmark. To ensure fair comparison, all comparative and ablation experiments use the same experimental configuration. During training, the SGD optimizer with an initial learning rate of 0.01 is used to update the weights with a multi-learning rate strategy.

[0132] To evaluate the semi-supervised learning method proposed in this chapter, this embodiment selects three open-source datasets for experimental verification: left atrial dataset, brain tumor dataset, and automatic cardiac diagnosis challenge (ACDC) dataset.

[0133] For the left atrial dataset, it contains 100 three-dimensional gadolinium-enhanced MRI images with a voxel resolution of 0.625x0.625x0.625mm 3 , with corresponding accurate annotations. The dataset is divided into five groups, each containing 20 cases. The preprocessing stage includes normalizing all volumes to zero mean and unit variance, and cropping and expanding the edge for the target. During training, random cropping is performed to 112x112x80 as the model input; while in the inference stage, a sliding window of the same size is used to generate the segmentation results with a step size of 18x18x4.

[0134] The brain tumor dataset is from the 2019 multi-modal brain tumor segmentation challenge, containing 335 scans from glioma patients and annotations, with a resolution of 1x1x1mm 3 . According to previous research, 250 scans are randomly divided for training, 25 for validation, and the remaining 60 for testing. During training, random cropping is performed to 96x96x96 as the input image. Among them, 10% (i.e. 25 scans) and 20% (i.e. 50 scans) of the labeled data, and the rest are unlabeled data.

[0135] The ACDC dataset is obtained from real clinical examinations at the University Hospital of Lausanne, containing 100 cardiac MRI samples from patients. Fixed data segmentation is performed at the patient level, i.e. 70 patients for training, 10 for validation, and 20 for testing. Unlike the previous two 3D binary segmentation tasks, the ACDC dataset is expanded to a 2D multi-class segmentation task, aiming to segment the myocardium, left ventricle, and right ventricle from 2D MRI slices.

[0136] Finally, the quantitative evaluation is performed by Dice coefficient, Iaccard index, average surface distance (ASD) and 95% Hausdorff distance (95HD). In addition, the model complexity is also calculated, including the number of parameters (Para) and multiply-accumulate operations (MAC).

[0137] From the data analysis in Table 1, it can be seen that all methods have improved after introducing unlabeled data, but the gain of UA-MT is the smallest, and it shows larger standard deviation and poor stability. In contrast, AC-MT achieves the best results in Dice coefficient and Jaccard index, and the performance is stable; while MC-Net+ is significantly better than other comparative methods in 95HD and average surface distance (ASD), which are two edge-sensitive indicators, showing the advantage of cycle pseudo-label consistency learning in improving boundary segmentation. It is particularly worth noting that the model of the embodiment outperforms the current state-of-the-art method in all evaluation indicators, especially in 95HD and ASD which measure the edge segmentation accuracy. Specifically, compared with UA-MT, the model of the embodiment significantly reduces the 95HD index from 18.71 to 7.34 and the ASD from 4.63 to 1.66 under the condition of only 10% labeled data, not only improving the segmentation accuracy, but also improving the overall stability.

[0138] Table 1 Quantitative comparison results on left atrium dataset

[0139]

[0140]

[0141] On the brain tumor dataset, the embodiment performs a detailed performance comparison analysis of different methods through Table 2. The table records the performance of various methods under the same labeled proportion (i.e. 10% or 25%, and 20% or 50% of the scanned images as the training set), and the first three rows of bolded values represent the best performance indicators achieved under the fully supervised setting. The framework of the embodiment is quantitatively compared with a series of classic and latest semi-supervised methods, and significant improvements are achieved in both region segmentation accuracy and boundary fitting degree.

[0142] When only 10% of the labeled data is used for training, although the AC-MT method temporarily leads other existing methods with a Dice similarity coefficient of 83.03%, the method of the present application achieves a 1.67% Dice precision improvement on this basis, thereby successfully surpassing AC-MT. As the annotation ratio increases to 20%, the performance of the model of the present application on the test set further highlights the advantages. Especially worth mentioning is that compared with other semi-supervised techniques and baseline models, the model of the present application not only outperforms in overall segmentation accuracy, but also effectively reduces over-segmentation and under-segmentation, ensuring more accurate and continuous tumor region segmentation effect.

[0143] The model of the present application is further extended for two-dimensional multi-class segmentation tasks. The results of Table 3 are the average performance of the myocardium, left ventricle, and right ventricle three segmentation targets on the ACDC dataset. The results show that compared with other methods, the model of the present application obtains the highest Dice, Jaccard, 95HD and ASD in each semi-supervised setting; by effectively utilizing unlabeled data, the model of the present application almost produces an average 9.78% or 3.47% Dice gain than the fully supervised UNet model trained using 10% or 20% labeled data. It can be seen that whether using 10% or 20% labeled data for training, the model of the present application has achieved very large performance improvement in each class of 2D medical image segmentation.

[0144] Overall, based on the results of the three datasets, the model of the present application shows superior performance to other SOTA methods in semi-supervised medical image segmentation. Moreover, the model of the present application is not dependent on a specific backbone and can be applied to various medical tasks in 2D or 3D segmentation.

[0145] Table 2 Quantitative comparison results on brain tumor dataset

[0146]

[0147] Table 3 Quantitative comparison results on cardiac automatic diagnosis dataset

[0148]

[0149]

[0150] The experimental results show that, on three public medical image datasets of left atrium, brain tumor and automatic diagnosis of heart, the algorithm proposed in the application shows obvious performance improvement compared with existing semi-supervised learning methods in the context of limited labeled data, especially in the aspect of edge segmentation accuracy. Specifically, on the left atrium dataset using only 20% labeled data, the method of the application achieves a low difference level of 5.98 on the 95HD index, and a low error level of 1.52 on the ASD index.

[0151] Preferably, the embodiment also provides a heterogeneous perception-based mutual learning semi-supervised 3D medical image segmentation system, comprising:

[0152] An uncertainty perception evaluation module is configured to evaluate potential error prediction areas in real time in a single-step forward propagation process, and improve the understanding and judgment ability of uncertain areas by constructing a difference mask and optimizing a correction loss function.

[0153] A multi-view contrast reconstruction module includes two groups of sub-networks with different structures, which locate and optimize the areas that may exist in the segmentation by comparing the differences between their respective prediction results.

[0154] A mutual consistency learning module is based on entropy loss, cross pseudo-supervised loss, bidirectional KL loss and distillation loss to optimize the prediction consistency of student network and teacher network.

[0155] Preferably, the embodiment of the application also provides a specific implementation of an electronic device capable of implementing all steps of the heterogeneous perception-based mutual learning semi-supervised 3D medical image segmentation method in the above-mentioned embodiments, and the electronic device specifically includes the following contents:

[0156] A processor, a memory, a communications interface and a bus;

[0157] The processor, the memory and the communications interface complete mutual communication through the bus; the communications interface is configured to realize information transmission between the server-side device, the metering device and the user-side device and other related devices.

[0158] The processor is configured to call the computer program in the memory, and the processor executes the computer program to realize all steps of the heterogeneous perception-based mutual learning semi-supervised 3D medical image segmentation method in the above-mentioned embodiments.

[0159] The embodiment of the present application also provides a computer readable storage medium capable of realizing all steps in the heterogeneous perception based mutual learning semi-supervised 3D medical image segmentation method in the above-mentioned embodiment. The computer readable storage medium has a computer program stored thereon. When the computer program is executed by a processor, all steps of the heterogeneous perception based mutual learning semi-supervised 3D medical image segmentation method in the above-mentioned embodiment are realized.

[0160] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous or possible.

[0161] Although the present application provides method operation steps as embodiments or flowcharts, more or less operation steps can be included based on routine or non-creative labor. The order of steps listed in the embodiments is only one of the many execution orders of the steps, and does not represent the only execution order. When the device or client product in the actual implementation is executed, the method order shown in the embodiments or the drawings can be executed in sequence or in parallel (for example, in the environment of parallel processors or multi-threaded processing).

[0162] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) having computer usable program code embodied therein.

[0163] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which realizes the flowchart Figure 1 The functions specified in one or more flows or one or more blocks. Figure 1 The functions specified in one or more flows or one or more blocks.

[0164] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus, so that a series of operation steps are executed on the computer or other programmable device to produce a computer implemented process, so that the instructions executed on the computer or other programmable device provide a means for realizing the flowchart Figure 1one or more processes and / or blocks Figure 1 the steps of a function specified in one or more blocks.

[0165] The present application is not limited to the embodiments described above. The above description of specific embodiments is intended to describe and illustrate the technical solutions of the present application, and the specific embodiments described above are merely illustrative and are not restrictive. Without departing from the purpose of the present application and the scope protected by the claims, those of ordinary skill in the art can make many forms of specific changes under the inspiration of the present application, and these all belong to the protection scope of the present application.

Claims

1. A method for semi-supervised 3D medical image segmentation based on heterogeneous-aware mutual learning, characterized in that, The method comprises the following steps: S1. Constructing an uncertainty perception evaluation model, specifically comprising: S101. Designing a computing unit containing four classifiers, which share the same encoder but use different loss functions, including cross-entropy loss, focal loss, Dice loss and IoU loss; S102. Obtaining the deterministic region and the uncertainty region by predicting each voxel through the above-mentioned classifiers, and generating a weight map with cognitive uncertainty to reduce additional computational cost; S103. For the deterministic region, a pseudo-label strategy is used, and a corresponding pseudo-label loss is calculated A student network and a teacher network are used to predict the uncertainty region, and a consistency loss between the student network and the teacher network prediction is obtained S104. define a total unsupervised loss function An uncertainty-aware evaluation model is obtained, and combined with the pseudo-label loss and the consistency loss to optimize the prediction of the uncertainty-aware evaluation model on the uncertainty region. S2. Constructing a multi-view contrast reconstruction model, specifically comprising: S201. Perform a binary XOR operation on the outputs of the student network and the teacher network to obtain a difference mask M diff ; S202. Use the difference mask M diff to extract the potential misprediction region for each voxel, and compute the correction loss function for the potential misprediction region S203. Synthesize the base segmentation loss and the correction loss function Obtain the multi-view contrastive reconstruction model, define the supervised learning loss of the multi-view contrastive reconstruction model To improve the performance of the multi-view contrastive reconstruction model; S3. Constructing and optimizing a mutual consistency learning network model, specifically comprising: S301. Calculate the inter-class similarity penalty loss of the student network and the teacher network by using the cross pseudo-supervision method and the cross pseudo-supervision loss to supervise the teacher and student network learning; S302. Calculate the bidirectional KL loss of the student network and the teacher network by using the distillation method and the distillation loss obtained by the Dice loss to facilitate the information exchange between the student network and the teacher network; S303. define the entropy loss based on the confidence The cross-pseudo-supervised loss, the bidirectional KL loss and the distillation loss are combined to optimize the prediction consistency of the student network and the teacher network. S304. Calculate the directional consistency loss of the student network and the teacher network by decoupling the inconsistent part of the prediction results of the student network and the teacher network as unreliable data and guiding data and optimize the consistency loss across the pixel space Obtain the mutual consistency learning network model, and improve the prediction accuracy of the mutual consistency learning network model for the pixels around the decision boundary; The uncertainty perception evaluation model, the multi-view contrast reconstruction model and the mutual consistency learning network model are combined to realize the mining of the uncertainty knowledge of the input data of the student network and the teacher network.

2. The method of claim 1, wherein the method is based on a heterogeneous perception-based co-learning semi-supervised 3D medical image segmentation method. Total unsupervised loss function in step S104 Combining pseudo-label loss and consistency loss: where M c is a binary mask of the region, where voxels with value 1 correspond to the foreground, J is an all-one tensor, is a hyper-parameter to adjust the strength of consistency regularization, denotes the element-wise multiplication operation, and the Mean(·) function is used to compute the average value of non-zero elements to get a scalar result; is the pseudo-label loss.

3. The method of claim 1, wherein the method further comprises: In step S201, the softmax outputs of the student network and the teacher network for the unlabeled data part of the student network The binary XOR operation is performed on the difference mask M diff The calculation formula is as follows: where BINA( ) represents a binarization function, is an XOR operation; extracting the error prediction region; using a difference mask M diff to extract the error prediction region and to compute the correction loss function wherein, denotes the true value of the labeled data, MASK(·) denotes extracting the difference mask M diff identified predicted values, denotes the potentially erroneous prediction region.

4. The method of claim 1, wherein, In step S203, the supervised learning loss The calculation is as follows: wherein, Y denotes the ground truth of the labeled data, L Y denotes the network prediction result of the labeled data, β is used to balance the importance of the reconstruction loss and other segmentation loss terms; the Dice loss and the cross-entropy (CE) loss jointly constitute the basic segmentation loss, and the loss function is modified as which helps to improve the recognition ability of the multi-view contrast reconstruction model for potential erroneous predictions.

5. The method of claim 1, wherein, The inter-class similarity penalty loss in step S3 The calculation formula is as follows: wherein, represents the i-th data of the dataset, is a cross-entropy loss function, p 1i and p 2i represent the prediction results of the student network and the teacher network, respectively, and denotes the true value label, and WxHxD are the width, height, and depth of the input image, respectively; By bidirectional application of cross pseudo-supervised loss To improve the learning effect of student network and teacher network on unlabeled data; Compute bidirectional KL loss and distillation loss; for student and teacher networks They are denoted as i Probability distribution computed by the softmax function:​ Computing bidirectional KL loss and distillation loss by Dice loss Wherein, p1 and p2 represent the posterior probability prediction of the student network and the teacher network respectively; Dice loss is used to replace bidirectional KL divergence loss to alleviate the class imbalance problem, and form a distillation loss, wherein T represents the parameter temperature setting; Defining a confidence-based entropy loss By decoupling the student network and the teacher network, the inconsistent part of the prediction result is predicted as unreliable data and guide data, and the direction consistency loss is calculated where sp is a sharpening operation, sp is obtained by applying the sharpening operation to sp 1 and sp 2 Comparing, obtaining high-confidence part (hsp 1 , hsp 2 ) and low-confidence part (lsp 1 , lsp 2 ); only applying mean square error L2 loss as an optimization goal to the low-confidence part, detach does not backpropagate gradients to the high-confidence part; Optimizing consistency loss across pixel space To improve the prediction accuracy of pixels around the decision boundary by the mutually consistent learning network model; wherein PL1 and PL2 represent corresponding pseudo labels, f1(x i ) and f2(x i ) represent prediction output results of student and teacher networks, and CE represents cross-entropy loss.

6. A heterogeneous perception based co-learning semi-supervised 3D medical image segmentation system, characterized in that, The method comprises the following steps: The uncertainty perception evaluation module is used for real-time evaluation of potential error prediction regions in the single-step forward propagation process, and the understanding and judgment ability for the uncertainty region is improved through the construction of difference mask and the optimization of correction loss function; The multi-view contrast reconstruction module contains two groups of sub-networks with different structures, which locate and optimize the region that may exist in the missegmentation by comparing the differences between the prediction results of the two groups of sub-networks; The mutual consistency learning module is based on entropy loss, cross-pseudo-supervised loss, bidirectional KL loss and distillation loss to optimize the prediction consistency of the student network and the teacher network.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the steps of the heterogeneous perception-based mutual learning semi-supervised 3D medical image segmentation method of any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the heterogeneous perception-based mutual learning semi-supervised 3D medical image segmentation method of any one of claims 1-5.

Images