A multi-organ semi-supervised image segmentation method based on balanced sub-category mining

Abstract

The application discloses a multi-organ semi-supervised image segmentation method based on balanced sub-category mining, comprising the following steps. S10, a balanced sub-category mining stage, further dividing the original target category with bias, and generating sub-category data with balanced categories from the original multiple segmentation targets; S20, a semi-supervised model training stage based on sub-category regularization, using the sub-category data with balanced categories to train a sub-category segmentation auxiliary task, and using the auxiliary task to regularize and constrain the original multi-organ segmentation, so as to promote the unbiased knowledge in the sub-category network to flow into the target multi-organ segmentation network, thereby relieving the influence of the class imbalance problem on the network. The scheme provided by the application is simple and effective, the class-balanced sub-category is mined to assist the learning of the segmentation task, thereby providing class-unbiased knowledge for the multi-organ segmentation task, and solving the foreground class imbalance problem in the multi-organ semi-supervised segmentation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image processing methods, and particularly relates to a multi-organ semi-supervised image segmentation method based on balanced sub-class mining. BACKGROUND

[0002] Semi-supervised learning can utilize unlabeled data to improve model performance, alleviate the dependence of the network on labeled data, and reduce labeling costs. Therefore, a large number of researchers have invested in the research of medical image semi-supervised segmentation problems and have made great progress. At present, the popular research methods in semi-supervised learning are almost all aimed at a single target in medical images, such as a certain organ or a certain type of disease lesion area. However, in clinical practice, doctors usually have multiple targets that need to be paid attention to at the same time, such as when performing radiotherapy, doctors not only need to pay attention to the tumor target area, but also need to understand the situation of multiple organs at risk. Therefore, it is of great research significance and value to study the multi-organ medical image semi-supervised segmentation algorithm.

[0003] Compared with single-target segmentation, multi-organ medical image segmentation faces more challenges. First, different organs have significant differences in morphology and structure, which makes it difficult for the model to learn discriminative features. More importantly, multi-organ segmentation also has the problem of foreground class imbalance. Because of the large size difference between different organs, the sample amount of different classes in the training data is biased, which eventually causes the model class bias problem. This problem is usually manifested as the network model having good generalization performance for large targets, but it is easy to cause prediction bias for small targets. In the semi-supervised learning scenario, this prediction bias for small targets will cause the model to continuously introduce external noise in the training on unlabeled data, thereby exacerbating the difficulty of unbalanced target feature extraction. The existing multi-organ semi-supervised segmentation method simply applies the single-target semi-supervised segmentation algorithm to the multi-organ segmentation scene, and does not deeply explore and research the possible difficulties of multi-organ semi-supervised segmentation, thereby limiting the performance of the algorithm. SUMMARY

[0004] To solve the above problems, the application provides a multi-organ semi-supervised image segmentation method based on balanced sub-class mining, which is a simple and effective solution that mines class-balanced sub-classes to assist in learning the segmentation task, thereby providing class-unbiased knowledge for the multi-organ segmentation task and solving the foreground class imbalance problem in multi-organ semi-supervised segmentation.

[0005] To achieve the above purpose, the technical scheme adopted by the application is as follows: a multi-organ semi-supervised image segmentation method based on balanced sub-class mining, comprising the following steps:

[0006] S10, a balanced sub-category mining stage, further divides the original target categories with bias to generate sub-category data with balanced categories from the original multiple segmentation targets;

[0007] S20, a semi-supervised model training stage based on sub-category regularization, uses the sub-category data with balanced categories to train a sub-category segmentation auxiliary task, and uses the auxiliary task to regularize and constrain the original multi-organ segmentation, so as to promote the unbiased knowledge in the sub-category network to the target multi-organ segmentation network, thereby alleviating the influence of the class imbalance problem on the network.

[0008] Further, in the balanced sub-category mining stage, the steps include:

[0009] S11, using the labeled data set D l The multi-organ segmentation model is pre-trained, and the multi-organ segmentation model uses a U-net based on an encoder-decoder structure as a network framework; after pre-training, the multi-organ segmentation model has the ability to perceive different targets and the ability to encode semantic features;

[0010] S12, by shielding the output layer, using the pre-trained model to map all original images X l in the label data set D i into pixel-level semantic features f to form a feature map, each feature vector in the feature map corresponds to a pixel in the original input image;

[0011] S13, performing feature clustering operation on the feature vectors of each category respectively, similar feature vectors are aggregated into the same cluster as a corresponding sub-category of the category;

[0012] S14, numbering different sub-categories, and labeling the pixel points corresponding to the feature vectors according to the numbering to obtain sub-category data.

[0013] Further, considering the problem of class imbalance, the clustering of different categories is determined by the pixel ratio of the category in the label data, that is, the larger the target, the more pixel points it occupies, and the category with less pixel point ratio will be divided into fewer sub-categories, so as to ensure that the sub-categories divided by different categories are more balanced.

[0014] Further, an equalization clustering algorithm is used for sub-category clustering, and the clustering method regularizes the number of samples in each cluster in the optimization process, so as to ensure that all clusters have the same proportion of sample number; in this way, the balanced sub-category data set with balanced categories can be generated

[0015] Further, in the sub-category regularization-based semi-supervised learning stage, the unbiased knowledge in the balanced sub-category data l and the unlabeled data D u is introduced to assist the training of the semi-supervised network, and the unbiased knowledge in the balanced sub-category data is fully utilized by the sub-category regularization-based semi-supervised segmentation model.

[0016] Further, the sub-category regularization-based semi-supervised segmentation model adopts a Mean Teacher-based architecture, including a student network and a teacher network, both of which have the same network structure; the student network updates the parameters through the back propagation of the loss, while the teacher model updates according to the exponential moving average of the student model weights, and the expression is as follows:

[0017] θ′ t =τθ′ t-1 +(1-τ)θ;

[0018] where θ′ t represents the model parameters of the teacher network at the t th iteration, θ represents the model parameters of the student network, and τ is the EMA decay coefficient for controlling the update rate of the teacher network.

[0019] For the original image X i , weak augmentation is first performed before inputting into the teacher network, that is, Gaussian noise is added to the input image; while strong augmentation is performed before inputting into the student network, that is, rotation, color conversion and strong noise disturbance are adopted.

[0020] Further, the U-net is used as the basic network structure of the student model and the teacher model; the original multi-organ segmentation task and the balanced sub-category task are embedded into the same model by using the multi-task strategy, and the shared encoder weight is adopted to share the encoding feature information between the two tasks.

[0021] Further, the sub-category regularization-based semi-supervised segmentation model is trained based on the labeled data set D l , and the unlabeled data set D u ; for the labeled data image, the difference between the reduced label and the model prediction is used to update the model parameters; while for the unlabeled data, the teacher model and the balanced sub-category auxiliary task are used to generate the supervision signal, and the multi-organ segmentation task is supervised together, so as to improve the segmentation performance.

[0022] According to the labeled data and the unlabeled data, the loss function is established, and thus the sub-category regularization-based semi-supervised segmentation model objective function is constructed.

[0023] ​Furthermore, for labeled data, constraints are directly applied based on the difference between the labels and the target segmentation network's predictions, combining cross-entropy loss and Dice loss as the loss function for labeled data:

[0024] L sup =L seg (Y,p)+L seg (Y sub ,p sub );

[0025] Among them, L seg (Y,p)=L ce (Y,p)+μL dice (Y,p);

[0026] Cross-entropy loss:

[0027] Dice loss:

[0028] Among them, Y ic p represents the label of category c in sample i; c Y represents the network's prediction result for category c; C refers to the number of categories included; μ is the harmonic hyperparameter; sub Indicates subcategory label; p sub This indicates the prediction result for the sub-category task.

[0029] Furthermore, for unlabeled data, two types of consistency loss are included: model consistency loss and task consistency loss.

[0030] (1) Model consistency loss is applied to the same original input X. i To ensure consistency between the student model's and teacher model's predictions, the teacher network provides supervision for the student model. Mean squared error loss is used to constrain prediction consistency, with the following formula:

[0031]

[0032] Where n represents the number of pixels in the image; p s p represents the prediction result of the student model. t p represents the prediction result of the teacher model. s,i p represents the prediction result of the model for the i-th sample student. t,i This represents the prediction result of the model for the i-th sample teacher;

[0033] Similarly, the model consistency loss also includes two parts: the multi-organ segmentation task and the sub-class segmentation task, and the formula is:

[0034]

[0035] The sub-class prediction result representing the student model, The sub-class prediction result representing the teacher model, L mse Representing the MSE loss;

[0036] The consistency loss based on uncertainty perception is introduced to evaluate the uncertainty of the network prediction result, identify the error-prone area in the teacher network, so that the student network pays more attention to the reliable prediction generated by the teacher network for learning, and the consistency loss formula based on uncertainty is:

[0037]

[0038] Wherein,

[0039] Wherein, N represents the number of samples; and represents the product of the corresponding position elements of two matrices.

[0040] The uncertainty of the network is estimated by using the prediction entropy, when the probability distribution of the prediction result is more uniform, the prediction entropy is higher, and the uncertainty is higher; on the contrary, when the prediction probability of a certain class is very high and the prediction probability of other classes is very low, the prediction entropy is lower, and the uncertainty of the network is lower.

[0041] (2) The task consistency loss aims to constrain the semantic information in the sub-class and the original target class to be consistent; the sub-class prediction result is reversely mapped into the original class result, and is used as a pseudo-label multi-organ segmentation task for regularization, so as to introduce the class-unbiased knowledge into the original target segmentation model, and the formula is:

[0042]

[0043] Wherein, y' represents the pseudo-label of the original target, Map represents a mapping function; The sub-class prediction result representing the student model, p s The prediction result of the student model; L ce (y', p s ) represents the cross-entropy loss of p s and the pseudo-label, and mu represents a balance parameter, L dice (y', p s ) represents the Dice loss of p s and the pseudo-label.

[0044] The beneficial effects of the technical solution are as follows:

[0045] This invention proposes a semi-supervised multi-organ segmentation algorithm for medical images based on balanced subclass mining, which solves the class imbalance problem encountered in multi-organ segmentation and greatly improves the segmentation quality of semi-supervised models for imbalanced targets. Specifically, the algorithm proposed in this invention includes a balanced subclass mining stage and a semi-supervised model training stage based on subclass regularization. The balanced subclass mining stage mines class-balanced subclass data based on manually labeled data. The semi-supervised model training stage based on subclass regularization uses the mined balanced subclass data to assist in the training of the semi-supervised model, thereby helping the multi-object segmentation network extract knowledge of class balance.

[0046] Because the original multi-organ targets suffer from class imbalance, using them for model training can easily lead to class bias, thus failing to accurately capture all target knowledge. To mitigate the impact of class imbalance, this invention utilizes a balanced subclass mining method. By further dividing the original classes, a class-balanced subclass dataset is generated, thereby providing unbiased knowledge for subsequent model training and improving the model's ability to perceive small targets.

[0047] To introduce class-unbiased features into multi-object segmentation tasks, this invention proposes a semi-supervised learning framework based on a Mean Teacher architecture with subclass regularization. Specifically, this model integrates subclass segmentation and multi-organ segmentation tasks, leveraging the correlation between the two tasks to transfer class-unbiased knowledge from subclass segmentation to multi-organ segmentation. Furthermore, to prevent prediction biases in the semi-supervised learning process from introducing noise into model training, this invention also introduces uncertainty assessment, enabling the network to learn more reliable and accurate knowledge. Attached Figure Description

[0048] Figure 1 This is a schematic diagram illustrating the principle of a multi-organ semi-supervised image segmentation method based on balanced sub-class mining according to the present invention.

[0049] Figure 2 This is a schematic diagram of the visualization prediction results of the cervical cancer dataset comparison experiment in an embodiment of the present invention;

[0050] Figure 3 This is a visual comparison diagram of the original label sub-category labels in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings.

[0052] In this embodiment, see Figure 1 As shown, this invention proposes a multi-organ semi-supervised image segmentation method based on balanced subclass mining, including the following steps:

[0053] S10, a balanced sub-category mining stage, further dividing the original target categories with bias to generate sub-category data balanced in category from the original multiple segmentation targets;

[0054] S20, a semi-supervised model training stage based on sub-category regularization, using the sub-category data balanced in category to train a sub-category segmentation auxiliary task, and using the auxiliary task to regularize and constrain the original multi-organ segmentation, so as to promote the unbiased knowledge in the sub-category network to flow into the target multi-organ segmentation network, thereby alleviating the influence of the category imbalance problem on the network.

[0055] As an optimization scheme of the above embodiment, in the balanced sub-category mining stage, the step of:

[0056] S11, using the labeled data set D l The multi-organ segmentation model is pre-trained, and the multi-organ segmentation model uses a U-net based on an encoder-decoder structure as a network framework; after pre-training, the multi-organ segmentation model has the perception ability and semantic feature encoding ability for different targets;

[0057] S12, by shielding the output layer, using the pre-trained model to map all original images X l in the label data set D i into pixel-level semantic features f to form a feature map, each feature vector in the feature map corresponding to a pixel in the original input image; the purpose of this step is to map all pixel points in the original image into feature vectors with rich semantic information;

[0058] S13, performing feature clustering operation on the feature vectors of each category respectively, similar feature vectors are aggregated into the same cluster as a corresponding sub-category of the category;

[0059] S14, numbering different sub-categories, and labeling the pixel points corresponding to the feature vectors according to the number to obtain sub-category data.

[0060] Considering the problem of category imbalance, the clustering of different categories is determined by the pixel ratio of the category in the label data, that is, the larger the target, the more pixel points it occupies, and the category will be divided into more sub-categories, while the category with less pixel point ratio will be divided less times, so as to ensure that the sub-categories divided by different categories are more balanced.

[0061] In addition, traditional clustering methods, such as the k-means clustering algorithm, cannot ensure balanced sample numbers in different clusters. To ensure that the generated sub-category dataset is class-balanced and can assist the target multi-organ segmentation network in learning class-unbiased knowledge, an equalization clustering algorithm is used for sub-category clustering. This clustering method regularizes the sample number of each cluster during optimization, thereby ensuring that all clusters have the same proportion of sample numbers. In this way, a class-balanced equalization sub-category dataset can be generated

[0062] As an optimization scheme of the above embodiment, in the semi-supervised learning stage based on sub-category regularization, the original labeled data D l and unlabeled data D u are used as the basis to introduce unbiased knowledge in the equalization sub-category data to assist the training of the semi-supervised network, and the semi-supervised segmentation model based on sub-category regularization fully utilizes the unbiased knowledge in the equalization sub-category data.

[0063] Preferably, the semi-supervised segmentation model based on sub-category regularization adopts a Mean Teacher-based architecture, including a student network and a teacher network, both of which have the same network structure. The student network updates the parameters through the backpropagation of the loss, while the teacher model updates according to the exponential moving average of the student model weights, and the expression is as follows:

[0064] θ′ t =τθ′ t-1 +(1-τ)θ;

[0065] wherein θ′ t represents the model parameters of the teacher network at the t-th iteration, θ represents the model parameters of the student network, and τ is the EMA decay coefficient for controlling the update rate of the teacher network.

[0066] For the original image X i , weak augmentation is first performed before inputting into the teacher network, that is, Gaussian noise is added to the input image; while strong augmentation is performed before inputting into the student network, that is, rotation, color conversion, and strong noise disturbance are used.

[0067] U-net is used as the base network structure for both the student and teacher models. Considering that subclass data is generated from the original multi-object segmentation, containing a large amount of semantic information identical to the original target, this invention employs a multi-task strategy to embed the original multi-organ segmentation task and the subclassing task into the same model, and uses shared encoder weights to allow the two tasks to share encoded feature information. In this way, the target region extracted by the original multi-organ task can define the range of its subclasses to a certain extent, simplifying the learning difficulty of the subclassing task; simultaneously, the unbiased features learned by the subclassing task can also be directly used by the multi-organ segmentation task, thereby alleviating the class imbalance problem to some extent.

[0068] A semi-supervised segmentation model based on subclass regularization is used to label the dataset D. l and and unlabeled dataset D u Training is performed on labeled data images; model parameters are updated by reducing the difference between the labels and model predictions; and for unlabeled data, supervision signals are generated by using a teacher model and a balanced subclass auxiliary task to jointly supervise the multi-organ segmentation task, thereby improving segmentation performance.

[0069] A loss function is established based on labeled and unlabeled data, thereby constructing the objective function of a semi-supervised segmentation model based on subclass regularization.

[0070] For labeled data, since it has accurate manual segmentation labels and generated sub-category labels, the constraint is directly based on the difference between the labels and the predictions of the target segmentation network. The loss function for labeled data combines cross-entropy loss and Dice loss:

[0071] L sup =L seg (Y,p)+L seg (Y sub ,p sub );

[0072] Among them, L seg (Y,p)=L ce (Y,p)+μL dice (Y,p);

[0073] Cross-entropy loss:

[0074] Dice loss:

[0075] Among them, Y ic p represents the label of category C in sample i; cThe prediction result of the representative network for the category c; C refers to the number of included categories; mu is a harmonic hyperparameter; Y sub represents a subcategory label; p sub represents the prediction result of the subcategory task.

[0076] For unlabeled data, since there is a lack of real human labeling as supervision, it is necessary to train the model by using consistency constraints. Two kinds of consistency losses are included, which are model consistency loss and task consistency loss, respectively;

[0077] (1) The model consistency loss is for the same original input X i , which constrains the prediction result of the student model to be consistent with the prediction result of the teacher model, so that the teacher network provides supervision for the student model; the mean square error loss is used to constrain the prediction consistency, and the formula is:

[0078]

[0079] Where n represents the number of pixels in the image; p s represents the prediction result of the student model; p t represents the prediction result of the teacher model, p s,i represents the prediction result of the student model of the i-th sample, p t,i represents the prediction result of the teacher model of the i-th sample;

[0080] Similarly, the model consistency loss also includes two parts of the multi-organ segmentation task and the subcategory segmentation task, and the formula is:

[0081]

[0082] represents the subcategory prediction result of the student model, represents the subcategory prediction result of the teacher model, L mse represents the MSE loss;

[0083] Since the model is trained on unlabeled data only by the supervision signal generated by the teacher network, and the prediction result of the teacher network inevitably contains prediction errors, if these error noises are directly learned by the student network, it will cause the error knowledge to be iteratively amplified. In order to prevent these prediction noises from interfering with the training of the network, the present application introduces a consistency loss based on uncertainty perception to evaluate the uncertainty of the network prediction result, and identifies the error-prone areas in the teacher network, so that the student network pays more attention to the reliable prediction generated by the teacher network for learning. The formula of the consistency loss based on uncertainty is:

[0084]

[0085] Wherein,

[0086] where N represents the number of samples; and represents the product of the elements in the corresponding positions of two matrices;

[0087] The uncertainty of the network is estimated by using prediction entropy. When the probability distribution of the prediction result is more uniform, the prediction entropy is higher, and the uncertainty is higher. On the contrary, when the prediction probability of a certain class is very high and the prediction probability of other classes is very low, the prediction entropy is lower, and the network uncertainty is lower.

[0088] (3) Task consistency loss aims to constrain the semantic information in the subcategory and the original target category to be consistent. Since the subcategory is generated by subdividing the original target, it contains the same semantic information as the original target. At the same time, the model trained by balancing the subcategory does not suffer from the class imbalance problem and contains a large number of class-unbiased features. The subcategory prediction result is reversely mapped to the original category result and used as a pseudo-label multi-organ segmentation task for regularization, thereby introducing class-unbiased knowledge into the original target segmentation model. The formula is:

[0089]

[0090] where y' represents the pseudo-label of the original target, map represents a mapping function; represents the subcategory prediction result of the student model, p s represents the prediction result of the student model; L ce (y', p s ) represents the cross-entropy loss of p s and the pseudo-label, μ represents a balancing parameter, L dice (y', p s ) represents the Dice loss of p s and the pseudo-label.

[0091] Finally, the semi-supervised segmentation model objective function based on subcategory regularization is constructed as:

[0092]

[0093] where λ1 and λ2 are balancing coefficients.

[0094] To evaluate the effectiveness of the proposed method, the proposed method is compared with several state-of-the-art semi-supervised medical image segmentation methods, including U-net, EM, MT, UAMT, ICT, UTPC, and the proposed method. The bold font in all tables represents the optimal result, and the underlined font represents the suboptimal result. In addition, the paired t-test is performed on the average indicators, and the * in the table represents that the p-value calculated by the proposed method relative to the method is less than 0.05, indicating that the proposed method is significantly better than the method.

[0095] Comparison experiments and results: To prove the superiority of the proposed method, the proposed method is compared with several commonly used or advanced semi-supervised medical image segmentation methods, including U-net, EM, MT, UAMT, ICT, UTPC, and the proposed method. The bold font in all tables represents the optimal result, and the underlined font represents the suboptimal result. In addition, the paired t-test is performed on the average indicators, and the * in the table represents that the p-value calculated by the proposed method relative to the method is less than 0.05, indicating that the proposed method is significantly better than the method.

[0096] Comparison results of the cervical cancer dataset:

[0097] Table 1 Comparison results of the cervical cancer dataset in the indicator Dice

[0098]

[0099] Table 2 Comparison results of the cervical cancer dataset in the indicator JI

[0100]

[0101] Table 3 Comparison results of the cervical cancer dataset in the indicator ASD

[0102]

[0103] Tables 1, 2 and 3 show the quantitative comparison results of all methods on the cervical cancer dataset in different training set divisions in terms of Dice, JI and ASD. From the table, it can be observed that for all labeling ratios, the method proposed in the present application achieves the best results in all indicators. For the case of a labeling ratio of 10%, the performance of the present application relative to all comparison methods is significantly improved. Compared with the fully supervised method U-net, the average values of the three indicators of the present method are all greater than those of U-net in all labeling ratio cases. When the labeling data ratio is only 10%, the present application improves the Dice, JI and ASD indicators by 15.93%, 21.08% and 2.3, respectively, relative to U-net, proving that the present method can effectively utilize unlabeled data to improve the performance and generalization ability of the model. Compared with the suboptimal advanced semi-supervised segmentation algorithm ICT, the present method improves the Dice by 3.88%, 1.29% and 0.62% when the labeling ratios are 10%, 20% and 30%, respectively. Especially when the labeling data ratio is small, the present work can significantly improve the segmentation performance of multiple targets, proving the superiority of the method proposed in the present application. As the number of labeled samples increases, the performance of all methods gradually improves, and the performance gap between fully supervised algorithms and semi-supervised algorithms gradually decreases. Relative to other methods, the segmentation performance of the present method for these organs is generally better in all data division cases. In particular, when the labeling ratio is 10%, the best segmentation performance is achieved for the three unbalanced organs, proving that the present method can effectively alleviate the influence of class imbalance problems. In addition, it can also be observed from the table that when the amount of labeled data is small, the MeanTeacher algorithm is not as good as the fully supervised algorithm in the segmentation quality of unbalanced classes. This also shows that when semi-supervised learning is used on class-biased datasets, too much noise is easily introduced into the learning of these unbalanced classes, resulting in a decrease in the segmentation performance of the model for them.

[0104] In addition to quantitative index comparison, the present application also visualizes the segmentation prediction results of all methods on the cervical cancer dataset in FIGS. 6, 7 and 8. Figure 2In general, the prediction results of the proposed method are closer to the true labels, and the multiple targets can be more accurately captured and segmented. For the left femur, most semi-supervised methods cannot accurately obtain the information of the target, resulting in a large number of missegmentations, and the segmentation effect of some methods is even lower than that of the fully supervised method. When the labeling ratio is 10% and 20%, almost only the segmentation results of the method in this paper can completely and accurately segment the left femur region, which also proves that by using the balanced sub-class for regularization, the method in this paper can more accurately capture the features of these small targets, thereby obtaining better segmentation performance. With the increase of the number of labels, the segmentation quality of each method is continuously improved, but the method in this paper still has obvious performance advantages, proving that the method in this paper has superior performance.

[0105] To verify the effectiveness of each key module in this paper, a series of ablation experiments are also conducted on the cervical cancer dataset, and the experimental results are summarized in Table 4, where ① represents the uncertainty constraint, ② and ③ represent the task consistency constraint and the balanced sub-class segmentation task respectively. In addition, I ~ V are the numbers of the models, model I is the basic architecture of this work, i.e. MeanTeacher, and model V is the complete method proposed in this paper. Specifically, this paper explores the effectiveness of the uncertainty constraint module, the multi-task strategy, the task consistency constraint and the balanced sub-class under the labeling ratio of 10%. In addition, this paper also conducts experiments on the selection of clustering algorithms.

[0106] Table 4 Key module ablation experiment results on the cervical cancer dataset

[0107]

[0108] (1) Effectiveness of uncertainty constraint: To explore the effectiveness of the uncertainty constraint, this experiment adds the uncertainty constraint to the MeanTeacher (model I), as shown in model II in Table 4. After adding the uncertainty constraint module, the average performance on the two indicators increases by 0.9% and 2.57% respectively. This also shows that the uncertainty constraint module can effectively guide the network to learn more reliable information and reduce the noise introduced in the training process of unlabeled samples.

[0109] (2) The contribution of multi-task strategy: To verify the effectiveness of the multi-task strategy, this paper compares model II with model III which adds a sub-category auxiliary task. Table 4 summarizes the experimental results, which show that the Dice index and JI index are improved by 2.65% and 2.16% respectively by integrating the two models together using the multi-task strategy. In addition, the performance of the left femur, right femur and rectum organs is improved by integrating the sub-category segmentation task through the multi-task strategy, with an improvement of 4.87%, 2.58% and 1.60% in Dice respectively. This result shows that by integrating the two tasks together, the original multi-organ segmentation task can perceive the unbiased knowledge of the sub-category through the shared encoder, thereby alleviating the class imbalance problem and improving the performance of the model, proving the effectiveness of the multi-task strategy.

[0110] (3) The contribution of task consistency constraint: The task consistency loss utilizes the correlation between multiple tasks and uses the sub-category prediction results as the supervision signal for the original target segmentation network to allow the network to continue training. To verify the contribution of the task consistency constraint, this paper adds the task consistency loss to model III (i.e. model V) and conducts experiments. From the experimental results, it can be seen that after adding the task consistency regularization loss term, model V improves the Dice and JI indices by 3.72% and 4.37% respectively compared to model III. This proves that the sub-category consistency loss term plays a positive role in the training of the semi-supervised network, and by adding the regularization constraint, the network can make full use of the unlabeled data for learning and improve the model performance.

[0111] (4) The effectiveness of balancing sub-categories: To explore the effectiveness of sub-category balancing, this paper replaces the sub-category segmentation auxiliary task in model V with the original target segmentation task to obtain model IV, keeping other structures consistent to avoid the impact of parameter differences. According to Table 4.7, the Dice and JI indices are improved by 4.36% and 3.24% respectively by using the sub-category auxiliary task, which proves that the sub-category auxiliary can effectively assist in multi-organ segmentation. In addition, model V has significantly improved the performance of the left femur, right femur and bladder, which are imbalanced targets, indicating that by introducing the sub-category task, the network has mined more class unbiased knowledge, effectively alleviating the class imbalance problem.

[0112] Figure 3 The sub-category labeling illustration on the cervical cancer data is shown. Comparing the original multi-organ labeling with the generated sub-category labeling, it can be seen that the large rectum and bladder are divided into more detailed and balanced sub-categories. In addition, from the Figure 3As can be seen in the original class distribution on the top right, there is a serious class imbalance between the original multi-organ, and the rectum organ accounts for more than half of the foreground class pixels. After balanced sub-class clustering, the sub-class becomes more balanced, so that the model can better mine the characteristics of small targets and learn class-unbiased knowledge.

[0113] To solve the class imbalance problem of multi-organ semi-supervised segmentation, the present application proposes a multi-organ semi-supervised segmentation model based on balanced sub-class mining. The model can effectively alleviate the impact of foreground imbalance in medical images. The model is divided into two stages, which are the balanced sub-class mining stage and the semi-supervised learning stage based on sub-class regularization. In the balanced sub-class mining stage, this paper uses a balanced clustering algorithm to cluster the image semantic features, so as to divide the unbiased sub-class label from the original class. Then in the second stage, the unbiased information provided by the sub-class label is used to assist the training of the target segmentation network, so that the network learns more accurate multi-organ features from unlabeled data, thereby alleviating the impact of data class imbalance. In addition, the semi-supervised segmentation model in the second stage adopts the MeanTeacher architecture, introduces the sub-class segmentation task and the multi-organ segmentation task, and uses the sub-class segmentation task to regularize the multi-organ segmentation. In addition, in order to prevent the prediction generated by the teacher model on the unmarked data from containing a large amount of noise and interfering with the training of the student network, this paper also introduces an uncertainty estimation strategy to guide the student network to learn more reliable knowledge.

[0114] The above shows and describes the basic principles and main features of the present application and the advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A semi-supervised image segmentation method for multiple organs based on balanced subclass mining, characterized in that, Including the following steps: S10, the balanced subclass mining stage, further divides the biased original target categories to generate balanced subclass data from the original multiple segmentation targets; S20, the semi-supervised model training stage based on subclass regularization, uses class-balanced subclass data to train the subclass segmentation auxiliary task, and uses this auxiliary task to apply regularization constraints to the original multi-organ segmentation, so as to promote the flow of unbiased knowledge in the subclass network to the target multi-organ segmentation network, thereby mitigating the impact of class imbalance on the network. The semi-supervised segmentation model based on subclass regularization adopts a MeanTeacher-based architecture, consisting of a student network and a teacher network with identical network structures. The student network updates its parameters through backpropagation of the loss, while the teacher model updates its parameters based on the exponential moving average of the student model weights, expressed by the following formula: I will t =tθ' t-1 +(1-τ)θ; Where, θ' t Let θ represent the model parameters of the teacher network in the t-th iteration, θ represent the model parameters of the student network, and τ be the EMA decay coefficient, which is used to control the update rate of the teacher network. For the original image X i Before being input into the teacher network, the image undergoes weak enhancement, which involves adding Gaussian noise to the input image; while before being input into the student network, it undergoes strong enhancement, which involves rotation, color conversion, and strong noise perturbation.

2. The multi-organ semi-supervised image segmentation method based on balanced subclass mining according to claim 1, characterized in that, The balanced sub-category mining stage includes the following steps: S11, using the labeled dataset D l The multi-organ segmentation model was pre-trained, and the U-net based on the encoder-decoder structure was used as the network framework. After pre-training, the multi-organ segmentation model has the ability to perceive different targets and encode semantic features. S12, by masking the output layer, uses a pre-trained model to process the labeled dataset D. l All original images X i The semantic features f are mapped to the pixel level, and each feature vector in the feature map corresponds to a pixel in the original input image. S13, Perform feature clustering operation on the feature vector of each category, and similar feature vectors are aggregated into the same cluster as a corresponding subcategory of that category; S14 assigns numbers to different subcategories and labels the pixels corresponding to the feature vectors based on the assigned numbers, thus obtaining subcategory data.

3. The multi-organ semi-supervised image segmentation method based on balanced sub-class mining according to claim 2, characterized in that, To address the issue of class imbalance, the clusters of different classes are determined by the proportion of pixels of that class in the label data. That is, the larger the target and the more pixels the class occupies, the more subclasses it will be divided into, while the class with fewer pixels will be divided less often, thus ensuring that the subclasses divided from different classes are more balanced.

4. A semi-supervised image segmentation method for multiple organs based on balanced subclass mining according to claim 2 or 3, characterized in that, A balanced clustering algorithm is used for sub-category clustering. This clustering method regularizes the number of samples in each cluster during the optimization process, thereby ensuring that all clusters have the same proportion of samples. In this way, a balanced sub-category dataset with class balance can be generated.

5. The multi-organ semi-supervised image segmentation method based on balanced sub-class mining according to claim 1, characterized in that, In the semi-supervised learning stage based on subclass regularization, the original labeled data D is used. l And unlabeled data D u Based on this, balanced sub-category data is introduced. The unbiased knowledge in the data is used to assist in the training of the semi-supervised network. The unbiased knowledge in the balanced subclass data is fully utilized by the semi-supervised segmentation model based on subclass regularization.

6. The multi-organ semi-supervised image segmentation method based on balanced subclass mining according to claim 1, characterized in that, U-net is used as the basic network structure for both the student and teacher models. A multi-task strategy is employed to embed the original multi-organ segmentation task and the balanced sub-class task into the same model, and the two tasks share encoded feature information by using shared encoder weights.

7. A multi-organ semi-supervised image segmentation method based on balanced sub-class mining according to any one of claims 5-6, characterized in that, A semi-supervised segmentation model based on subclass regularization is used to label the dataset D. l and and unlabeled dataset D u Training is performed on labeled data images; model parameters are updated by reducing the difference between the labels and model predictions; and for unlabeled data, supervision signals are generated by using a teacher model and a balanced subclass auxiliary task to jointly supervise the multi-organ segmentation task, thereby improving segmentation performance. A loss function is established based on labeled and unlabeled data, thereby constructing the objective function of a semi-supervised segmentation model based on subclass regularization.

8. A multi-organ semi-supervised image segmentation method based on balanced sub-class mining according to claim 7, characterized in that, For labeled data, the difference between the labels and the target segmentation network predictions is used as a constraint, combining cross-entropy loss and Dice loss as the loss function for labeled data: L sup =L seg (Y,p)+L seg (Y sub ,p sub ); Among them, L seg (Y,p)=L ce (Y,p)+μL dice (Y,p); Cross-entropy loss: Dice loss: Among them, Y ic p represents the label of category C in sample i; c Y represents the network's prediction result for category C; C refers to the number of categories included; μ is the harmonic hyperparameter; sub Indicates subcategory label; p sub This indicates the prediction result for the sub-category task.

9. A multi-organ semi-supervised image segmentation method based on balanced sub-class mining according to claim 7, characterized in that, For unlabeled data, there are two types of consistency loss: model consistency loss and task consistency loss. (1) Model consistency loss is applied to the same original input X. i This constrains the student model's predictions to be consistent with the teacher model's predictions, thus allowing the teacher network to provide supervision for the student model; mean squared error loss is used to constrain prediction consistency, and the formula is: Where n represents the number of pixels in the image; p s p represents the prediction result of the student model. t p represents the prediction result of the teacher model. s,i p represents the prediction result of the model for the i-th sample student. t,i This represents the prediction result of the model for the i-th sample teacher; Similarly, the model consistency loss also includes two parts: the multi-organ segmentation task and the sub-class segmentation task, and the formula is: The subclass prediction results represent the student model. L represents the subclass prediction results of the teacher model. mse Represents MSE loss; An uncertainty-aware consistency loss is introduced to assess the uncertainty of network predictions, identify error-prone areas in the teacher network, and thus encourage the student network to focus on learning from reliable predictions generated by the teacher network. The formula for the uncertainty-based consistency loss is: in, Where N represents the number of samples; ⊙ represents the product of corresponding elements of the two matrices; The uncertainty of the network is estimated using prediction entropy. The more uniform the probability distribution of the prediction results, the higher the prediction entropy and the higher the uncertainty. Conversely, when the prediction probability of a certain category is very high while the prediction probabilities of other categories are very low, the prediction entropy is lower and the network uncertainty is lower. (2) The task consistency loss aims to ensure that the semantic information in the sub-category and the original target category remains consistent; the sub-category prediction results are back-mapped to the original category results and used as pseudo-labels for regularization in the multi-organ segmentation task, thereby introducing unbiased knowledge into the original target segmentation model. The formula is as follows: Where y' represents the pseudo-label of the original target. map stands for mapping function; p represents the subclass prediction result of the student model. s The prediction result represents the student model; L ce (y',p s ) represents p s Cross-entropy loss with pseudo-labels, μ represents the balance parameter, L dice (y',p s ) represents p s Dice loss with pseudo-labels.

Images