A 2D mean teacher model using difference information

Abstract

The present invention relates to a 2D mean teacher model using difference information, which is improved on the basis of the mean teacher framework. The mean teacher framework is a relatively popular semi-supervised learning framework, and many excellent semi-supervised frameworks are modifications of it. Based on the basic mean teacher, the present invention extracts the difference information between the prediction of the student model and the prediction of the teacher model, and then ensures the consistency of the prediction between the student model and the teacher model by constraining the extracted difference information, ensuring that the two can learn from each other. At the same time, in order to further ensure the consistency between the student model and the teacher model, thereby improving the model performance, the present invention uses the extracted difference information to improve the consistency loss between the models, so that it can pay more attention to the challenging areas of the image.

Description

Technical Field

[0001] The present invention relates to the field of medical image segmentation, and in particular to a 2D meanteacher model using difference information. Background Art

[0002] For semi-supervised left atrial medical image segmentation, most of the methods in the semi-supervised left atrial medical image segmentation task are based on the consistency criterion. However, some existing semi-supervised methods that follow this principle usually calculate the uncertainty of the image in order to ensure the accuracy of training, and remove the areas with large uncertainty during the training process. For example Figure 1 As shown in the figure, Yu et al. use Monte Carlo Dropout in the 3D mean teacher model to calculate the uncertainty areas predicted by the teacher model. These uncertainty areas are challenging areas. In order to allow the student model to learn more accurate data, the areas with high uncertainty are removed for the student model to learn.

[0003] However, these large uncertain regions calculated by Monte Carlo Dropout represent challenging parts of the image and contain difficult-to-recognize features such as Figure 2 As shown in the figure, these features are also important and should not be discarded. Blindly discarding them will cause the model to be unable to obtain some useful features from these areas for optimization, which is not conducive to improving model performance.

[0004] There are other methods that recognize the importance of these large uncertainty areas, that is, challenging areas. For example Figure 3 As shown in

[15] , a cyclic pseudo-labeling scheme is used to transform the prediction difference of the two decoders into an unsupervised loss, so that the overall framework can capture certain features from challenging regions in unlabeled images, thereby reducing the prediction difference of the two decoders and making the two decoders have consistent and accurate predictions.

[0005] This method uses the differences between models as uncertainty. Regions with large differences represent areas of high uncertainty, and therefore challenging areas. However, while this method takes into account the differences between models, it does not extract and further utilize them. Instead, it simply converts them into an unsupervised loss. This does not guarantee that the model will pay attention to these areas and extract features from them. Moreover, it only considers the differences between the final model outputs, not the intermediate outputs, which may result in incomplete information about the differences. Summary of the Invention

[0006] Based on this, in the left atrium segmentation task, it is necessary to address the technical problems of the existing 3D model's imperfect performance, large memory usage, and long training time. The present invention provides a 2D mean teacher model using difference information.

[0007] The present invention is implemented by the following technical solution: a 2D mean teacher model using difference information, which is a 2D model for left atrial image segmentation. The 2D mean teacher model includes the following steps during segmentation:

[0008] Step 1: performing data preprocessing on the obtained left atrial data set, where the left atrial data set includes a plurality of left atrial images;

[0009] Step 2: Divide the processed left atrial data set into a training set and a test set. The data in the training set is used for model training of the 2D mean teacher model. After multiple trainings, a 2D mean teacher model optimized on the training set is obtained.

[0010] Step 3: Use the test set to verify the model obtained from the training set and further evaluate the performance of the model;

[0011] In step 1, the 3D data in the left atrial dataset needs to be sliced first, and then the obtained slices are uniformly converted into the size required by the 2D mean teacher model;

[0012] In step 2, the training set and test set are generally divided into 80% for the training set and 20% for the test set. Then, the slices in the training set are divided into labeled slices and unlabeled slices, and the number of unlabeled slices is guaranteed to be greater than that of labeled slices.

[0013] In step three, the 2D mean teacher model includes a student model and a teacher model. All labeled slices in the training set enter the student model for processing, while unlabeled slices enter the teacher model and the student model for processing. Both the student model and the teacher model are a basic 2D Unet network, but an additional 1×1 convolution layer and a sigmoid layer are added to the last upsampling layer of the Unet network decoder, and an additional upsampling layer, a 1×1 convolution layer, and a sigmoid layer are added to the second-to-last upsampling layer, so as to obtain an intermediate prediction image of the same size as the final prediction image of the 2D Unet network.

[0014] As a further improvement to the above solution, in step 1, all 3D data are converted into 2D slices along the Z axis, and then the images of darker targets in these 2D slices are brightened. At the same time, 2D slices without segmented targets, that is, 2D slices that do not contain the left atrium, are removed.

[0015] Furthermore, brightening an image is achieved by overall amplifying the pixel values of the image.

[0016] As a further improvement to the above solution, the size required for the 2D mean teacher model is: the size of the 2D slice is transformed to 128×128, and the corresponding label is also transformed to 128×128.

[0017] As a further improvement of the above scheme, in the 2D mean teacher model, the student model and the teacher model have the same network architecture, and the weight of the teacher model Use the weights of the student model The exponential moving average EMA is updated, and the teacher model weight at step t will be updated to ,in It is used to control the update rate of EMA, that is, the proportion of the student model weight at step t and the teacher model weight at step t-1 in the weight of the teacher model at step t.

[0018] Furthermore, when the slice enters the student model, both the final and the middle of the student model will get the corresponding prediction results. Then, for the labeled slice, a supervised loss will be calculated. , this is a supervised loss It is calculated based on the prediction result generated by the student model after the labeled slice passes through the student model and the label corresponding to the labeled slice, as shown in the following formula:

[0019]

[0020] in, represents the dice loss function, which is used to evaluate the network prediction quality on labeled slice input;

[0021] i Indicates the sequence number of the labeled slice, with a value of 1, 2, ..., L, where L is a positive integer representing the number of labeled slices;

[0022] represents the segmentation network of the student model;

[0023] is the network weight of the student model;

[0024] For the i-th labeled slice;

[0025] The label of the i-th labeled slice.

[0026] Furthermore, different interferences are added to the unlabeled slices entering the student model and the teacher model respectively. The interference information added to the unlabeled slices entering the student model and the unlabeled slices entering the teacher model is different:

[0027] First, we need to calculate the difference between the predictions of the student model and the teacher model for the same unlabeled slice;

[0028] To calculate this difference, the following information is needed: the final prediction result of the student model for this unlabeled slice, the intermediate prediction result of the student model for this unlabeled slice, and the final prediction result of the teacher model for this unlabeled slice; the intermediate prediction result includes the prediction result generated by the last upsampling layer of the student model and the prediction result generated by the second to last upsampling layer of the student model;

[0029] The difference information D is obtained by calculating the difference between the final prediction result of the teacher model for this unlabeled slice and the final prediction result and the intermediate prediction result of the student model for this unlabeled slice. The specific calculation method of the difference is to calculate the pixel difference between the images.

[0030] After obtaining the difference information D, it can be supervised: use an image with all zero values and the extracted difference information D to calculate the difference loss ; This difference loss It is used to constrain the prediction consistency between the student model and the teacher model, and encourage the student model and the teacher model to learn from each other.

[0031] Preferably, the difference information D and the difference loss The specific calculation formula is:

[0032] ;

[0033] ;

[0034] Where U represents the number of unlabeled slices, 、 and represents the segmentation network of the student model, The output of is the final output of the student model segmentation network. The output of is the output of the last upsampling layer of the student model segmentation network. The output of is the output of the penultimate upsampling layer of the student model segmentation network, represents the segmentation network of the teacher model, is the mean square error loss, is the network weight of the student model, is the network weight of the teacher model, is the network perturbation of the student model, is the network perturbation of the teacher model, a, b, c are all constant coefficients, and Black is an image of all zero values of the same size as D.

[0035] Furthermore, based on the extracted difference information D, a new consistency loss function is designed for the student model and the teacher model. ,for:

[0036]

[0037] in, It is an indicator function that plays a role in judgment. H is the threshold coefficient of the selected area. and All are constant coefficients;

[0038] Afterwards, comprehensive D and Get the unsupervised loss , as shown below:

[0039]

[0040] Where p is the threshold coefficient.

[0041] Then combine the supervised loss obtained by the slices entering the student model , get the final loss function of the 2D mean teacher model , as shown below:

[0042]

[0043] Where m is the threshold coefficient.

[0044] As a further improvement to the above solution, in step 3, we directly feed the slices in the test set into the trained model to obtain the prediction results. Then, we compare the predicted results of the slices with their corresponding labels and calculate some indicators that represent the performance of the trained model. These indicators are as follows:

[0045] Dice similarity coefficient: ;

[0046] Jaccard similarity coefficient: ;

[0047] Surface average distance: ,

[0048] ;

[0049] Hausdorff distance_95: ;

[0050] Where TP, TN, FP, and FN refer to true positives, true negatives, false positives, and false negatives, respectively; X and Y represent the boundaries of the prediction and label, respectively; d(X,Y) represents the maximum distance from a point on X to the nearest point on Y; x represents the coordinates of a point on the boundary of the prediction result, and y represents the coordinates of a point on the boundary of the label.

[0051] The present invention solves the following technical problems:

[0052] First, to address the problems of 3D models taking up large amounts of memory, long training times, and limited training data, our method framework is based on 2D models, so it takes up less memory and is faster to train. Moreover, 2D models use slices of 3D volume images, and each 3D volume image can obtain a large number of 2D slices. Therefore, in datasets with fewer 3D volume images, the 2D model can achieve performance comparable to that of the 3D model by virtue of the advantage of the amount of training data.

[0053] Second, with regard to the problem that some methods remove areas with high calculated uncertainty in order to improve the overall accuracy of the model and thus improve performance, the present invention focuses on the importance of these areas and the important information they contain. It is able to mine information from them and improve the performance of the model without discarding these areas with high uncertainty, which are challenging areas.

[0054] Third, some methods pay attention to the importance of difference information between models, and believe that the challenging areas represented by areas with large difference information contain important information that cannot be discarded, but do not fully extract and utilize the difference information. When calculating the difference information between models, the present invention not only takes into account the differences between the final outputs of the models, but also takes into account the differences between the intermediate outputs of the models. In this way, the difference information obtained is more complete, and the present invention also reuses the obtained difference information to optimize the consistency loss of the present invention and further improve the performance of the overall model.

[0055] The present invention has the following beneficial effects:

[0056] 1. Reduced demand for labeled data, alleviating the burden of annotation for doctors: For medical images, obtaining labeled images is generally difficult. This is because it requires a large number of medical personnel with professional knowledge and a lot of time to complete accurate annotation, which greatly increases the cost of obtaining labeled data and also greatly increases the burden on medical personnel. Left atrial images also face this problem. The method framework of the present invention is a semi-supervised method that can use unlabeled data to improve model performance. This can reduce the demand for labeled data and alleviate the burden of annotation for doctors to a certain extent.

[0057] 2. Improved segmentation accuracy: This invention leverages inter-module variance information, which is often overlooked by most semi-supervised methods. This information, similar to uncertainty, can, to a certain extent, highlight challenging regions in the image, facilitating focus and information mining. Leveraging this information, the present invention designs a two-dimensional mean teacher (D-MT) framework. This framework, based on the mean teacher model, incorporates variance information to improve performance. This variance information is obtained by extracting the output differences between the final output of the teacher model and the intermediate and final outputs of the student model. This variance information is first constrained using a loss function. This variance information is then used to optimize the consistency loss between the student and teacher models. This guides the framework to focus on challenging regions with high variance, thereby capturing difficult-to-identify features and improving overall framework performance. This improved overall framework performance translates to improved model performance, which in turn improves the model's segmentation accuracy for left atrial images.

[0058] 3. Improved Efficiency: Most semi-supervised frameworks for various medical image segmentation tasks use 3D models as their base model. This is because compared to 2D models, 3D models have a simpler image processing pipeline, can take into account information from adjacent slices, and offer more stable and accurate segmentation performance. However, this high performance comes at a computational cost. Compared to 2D models, 3D models typically have more parameters, require more memory, and take longer to train, which are major limitations for their use. Furthermore, the limited amount of training data is a limitation for 3D models. 2D models use slices of 3D volume images, while 3D models use complete 3D volume images. While datasets typically have a limited number of complete 3D volume images, each 3D volume image can yield a large number of 2D slices. Therefore, in datasets with fewer 3D volume images, 2D models, leveraging the advantage of training data, can achieve comparable performance to 3D models while consuming less memory. The method framework (D-MT) of the present invention is based on a 2D model, and its performance is similar to that of more advanced methods based on 3D models, thereby improving the training speed of the overall framework in left atrial images, and being able to obtain a model with relatively high left atrial image segmentation accuracy in a shorter time, thereby improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 The figure is a schematic diagram of the framework of the existing 3D meanteacher model.

[0060] Figure 2 for Figure 1 Schematic diagram of the calculation process of the medium uncertainty map and the importance of high uncertainty areas.

[0061] Figure 3 A schematic diagram of an existing process of obtaining difference information by calculating the difference between similar decoders.

[0062] Figure 4 This is a schematic diagram of the framework of the most basic existing 2D mean teacher model.

[0063] Figure 5 Schematic diagram of the framework of the 2D mean teacher model of the present invention.

[0064] Figure 6 for Figure 5 Schematic diagram of the framework of the basic 2D Unet used in.

[0065] Figure 7 Flowchart of the present invention. DETAILED DESCRIPTION

[0066] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0067] It should be noted that when a component is referred to as being "mounted on" another component, it may be directly on the other component or there may be a central component. When a component is considered to be "set on" another component, it may be directly set on the other component or there may be a central component. When a component is considered to be "fixed to" another component, it may be directly fixed to the other component or there may be a central component.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0069] The method framework (D-MT) of the present invention is obtained by introducing difference information on the basis of the mean teacher framework. The overall performance of the model is improved by the introduced difference information. The mean teacher framework is a relatively popular semi-supervised learning framework, and many excellent semi-supervised frameworks are modifications of it. Specifically, the mean teacher model includes a student model and a teacher model. The teacher model is the weighted average of the student model and can produce more accurate targets. The student model learns from the teacher model by penalizing predictions that are inconsistent with the teacher model. The two optimize each other, so that the overall model achieves relatively high performance. Figure 4 Shown is the most basic mean teacher model.

[0070] Because the method framework (D-MT) of the present invention is an improvement on the basic mean teacher framework, like the basic mean teacher model, the framework of the present invention still encourages the student model and the teacher model to maintain consistent predictions under the same input with different disturbances, so that the student model and the teacher model can learn from each other and optimize each other, so as to capture more information from unlabeled data. The present invention improves the basic meanteacher model mainly by introducing difference information. The difference information introduced by the method framework (D-MT) of the present invention is obtained by extracting the output difference between the final output of the teacher model and the intermediate and final outputs of the student model. This difference information can, to a certain extent, more completely represent the uncertainty information between the models. For the extracted difference information, the present invention first uses a loss function to supervise and continuously reduce it, thereby achieving prediction consistency between the student model and the teacher model. In order to further achieve consistency between models, the present invention further utilizes the extracted difference information and improves the consistency loss between the student model and the teacher model, so that it has a higher loss value in areas with large difference information, that is, large uncertainty, and conversely has a lower loss value in areas with small difference, that is, small uncertainty. This enables the model to pay more attention to areas with high uncertainty, that is, more challenging areas, so that the method framework of the present invention (D-MT) has a certain processing capability for challenging areas in the image, thereby solving problems such as blurred target boundaries and small target areas to a certain extent. Figure 5 It is the specific method framework of the present invention.

[0071] The 2D mean teacher model using difference information of the present invention is a 2D model for segmenting left atrial images. The 2D mean teacher model includes the following steps during segmentation:

[0072] Step 1: Preprocess the obtained left atrial dataset, which includes multiple left atrial images. In step 1, the 3D data in the left atrial dataset must first be sliced, and then the obtained slices are uniformly converted to the size required by the 2D mean teacher model.

[0073] Step 2: Divide the processed left atrial data set into a training set and a test set. The data in the training set is used for model training of the 2D mean teacher model. After multiple trainings, a 2D mean teacher model optimized on the training set is obtained. In step 2, the division ratio of the training set and the test set is generally 80% for the training set and 20% for the test set. Then, the slices in the training set are divided into labeled slices and unlabeled slices, and the number of unlabeled slices is guaranteed to be greater than that of labeled slices.

[0074] In step three, the test set is used to verify the model obtained from the training set to further evaluate the performance of the model. In step three, the 2D mean teacher model includes a student model and a teacher model. All labeled slices in the training set enter the student model for processing, while unlabeled slices enter the teacher model for processing. Both the student model and the teacher model are a basic 2DUnet network, but an additional 1×1 convolution layer and a sigmoid layer are added to the last upsampling layer of the Unet network decoder, and an additional upsampling layer, a 1×1 convolution layer and a sigmoid layer are added to the penultimate upsampling layer, thereby obtaining an intermediate prediction map of the same size as the final prediction map of the 2D Unet network.

[0075] In this embodiment, in step 1, since the method framework of the present invention is a 2D model, if the data set is 3D, it is necessary to first perform slicing processing, and then uniformly convert the slices into the size required by the method framework of the present invention. In order to improve the training efficiency, some processing can be done on the images in the data set, such as removing the completely black images without segmented targets. If the content in the image is relatively dark, the pixel value of the image can be appropriately increased as a whole, and the image can be brightened to facilitate feature extraction.

[0076] In step 2, the training set and test set are generally divided into 80% for the training set and 20% for the test set. Since the method framework of the present invention is a semi-supervised method, the training set needs to be further divided into labeled data and unlabeled data. The ratio of these two types of data can generally be 20% labeled data and 80% unlabeled data, or 10% labeled data and 90% unlabeled data, or other ratios. However, it is generally necessary to ensure that the amount of unlabeled data is greater than the amount of labeled data.

[0077] A good model obtained on the training set refers to a model saved when the loss function is very small during training. Generally, the accuracy of the model saved at this time on the training set is very high, but this does not mean that the model's performance is very high. Further verification on the test set is necessary, as overfitting may occur. Overfitting means that the model performs very well on the training set but very poorly on the test set. Obviously, an overfitted model is not a good model.

[0078] In step 3: After the overall framework training is completed, since the method framework of the present invention is improved based on the 2D mean teacher method, generally two models will be trained, one is the student model and the other is the teacher model. To select which model to test and verify, you can choose the student model or the teacher model. Of course, you can also test several sets of data first and select the model with better performance between the student model and the teacher model for test and verification.

[0079] To further demonstrate the specific implementation process of the method framework of the present invention, we will use a public left atrial dataset as an example, namely the dataset of the 2018 Left Atrial Segmentation Challenge, to show how the method framework of the present invention (D-MT) processes this dataset.

[0080] The process of the method framework of the present invention can be divided into three stages, namely data preprocessing stage, training stage and testing stage.

[0081] 1 Data preprocessing stage

[0082] First, let's briefly introduce the basic situation of the dataset: This dataset provides 100 3D gadolinium-enhanced magnetic resonance imaging scans (GE MRIs) and LA segmentation masks for training and validation. The isotropic resolution of these scans is 0.625×0.625×0.625. In short, the dataset contains 3D left atrial images of 100 patients, of which 80 patients are used as training set data and 20 patients are used as test set data.

[0083] Since the method framework of the present invention is based on a 2D model, for the images of these patients, the present invention first converts them all into 2D slices along the Z axis. Then, in order to allow the framework to process these images better, the present invention performs a brightening operation on the images with darker targets in these 2D slices. The brightening operation on an image is achieved by increasing the pixel value of the entire image. At the same time, the present invention removes 2D slices without segmented targets, that is, slices that do not contain the left atrium, which is conducive to the training of the framework. Then, in order to allow these slices to enter the framework of the present invention for training, the present invention converts the size of these 2D slices into 128×128, and the corresponding labels are also converted into 128×128.

[0084] 2 Training Phase

[0085] Since the method framework of the present invention adopts a semi-supervised method, the data set needs to be further divided before training begins. The training set part of the data set, that is, the part containing 80 patients, is further divided into a labeled part and an unlabeled part. The division can be based on 20% labeled and 80% unlabeled, or 10% labeled and 90% unlabeled. Here, the present invention proceeds according to 20% labeled and 80% unlabeled, that is, the labeled data contains all 2D slices of 16 patients, and the unlabeled data contains all slices of 64 patients.

[0086] Each training iteration of the framework requires 12 2D slices as input, including 6 labeled slices and 6 unlabeled slices. The labeled slices are randomly selected from all slices of 16 labeled patients, and the unlabeled slices are randomly selected from all slices of 64 unlabeled patients. The training process of the framework consists of multiple epochs, each of which consists of many iterations. Verification is performed at the end of each epoch.

[0087] The 12 slices are slightly added with some interference information, and then enter the framework of the present invention. After entering the framework of the present invention, they will enter the two models in the framework of the present invention, one is the student model and the other is the teacher model. All 12 slices will enter the student model for processing, and only 6 unlabeled slices will enter the teacher model for processing. Both the student model and the teacher model are a basic 2DUnet network, except that the present invention adds an additional 1×1 convolution layer and sigmoid layer to the last upsampling layer of the Unet network decoder, and adds an additional upsampling layer, 1×1 convolution layer and sigmoid layer to the penultimate upsampling layer, so as to obtain an intermediate prediction map of the same size as the final prediction map of the network, which is convenient for subsequent operations. Basic 2D Unet such as Figure 6 shown.

[0088] The overall method framework of the present invention is based on the mean teacher model, so it is the same as the basic meanteacher model. Although the student model and teacher model of the present invention are two models, they have the same network structure. The difference between them is that the weights in the network are different during the training process, and the model optimization methods are different. Moreover, the student model and the teacher model learn from each other during the training process. Authoritative research has shown that if the model is integrated and predicted in different training processes, the prediction quality of the model for the target can be improved. Therefore, it can be used to improve the performance of the teacher model and improve the prediction results of the teacher model. Therefore, in the framework of the present invention, the student model and the teacher model have the same network architecture, and the weights of the teacher model are different. Use the weights of the student model The exponential moving average (EMA) of t is updated. The teacher model weight at step t will be updated as ,in It is used to control the update rate of EMA, that is, the proportion of the student model weight at step t and the teacher model weight at step t-1 in the weight of the teacher model at step t.

[0089] When the 12 slices enter the student model, the corresponding prediction results will be obtained at both the end and the middle of the student model. Then, for the 6 labeled slices, the present invention will calculate a supervised loss. This supervised loss is calculated based on the prediction results generated at the end of the student model after the 6 labeled slices pass through the student model and the labels corresponding to the 6 slices, as shown in the following formula:

[0090]

[0091] in Denotes the dice loss function, which is used to evaluate the quality of network predictions on labeled data input. Here, represents the segmentation network of the student model, is the network weight of the student model.

[0092] The present invention offers a different approach for the six unlabeled slices that enter the student model and the teacher model. The six unlabeled slices that enter the student model are identical to the six unlabeled slices that enter the teacher model, except that slightly different interference is added to the image. For these six slices, the present invention first calculates the difference between the student and teacher model predictions for these same six slices. To calculate this difference, the present invention obtains the following information: the student model's final prediction for these six slices, the student model's intermediate predictions for these six slices (including the predictions generated by the student model's last upsampling layer and the predictions generated by the student model's second-to-last upsampling layer), and the teacher model's final prediction for these six slices. The difference information D is obtained by calculating the difference between the teacher model's final prediction for these six slices and the student model's final and intermediate predictions for these six slices. This difference is calculated by calculating the pixel differences between the images. Once the difference information D is obtained, supervision can be performed. The difference loss of the present invention is calculated using an image with all zero values and the extracted difference information (D) This difference loss It can be used to constrain the prediction consistency between the student model and the teacher model, and encourage the student model and the teacher model to learn from each other.

[0093] The following is the difference information D and difference loss The specific calculation formula is:

[0094] ;

[0095] ;

[0096] in, 、 and represents the segmentation network of the student model, The output of is the final output of the student model segmentation network. The output of is the output of the last upsampling layer of the student model segmentation network. The output of is the output of the penultimate upsampling layer of the student model segmentation network, represents the segmentation network of the teacher model, is the mean square error loss, is the network weight of the student model, is the network weight of the teacher model, is the network perturbation of the student model, is the network perturbation of the teacher model, a, b, c are all constant coefficients, and Black is an image of all zero values of the same size as D.

[0097] After the above process, the present invention obtains the difference information D and difference loss calculated from these 6 unlabeled slices .

[0098] Then, the difference information (D) extracted by the present invention can represent the uncertainty of the overall framework of the present invention. For areas with large difference information, that is, areas with large uncertainty, the present invention believes that they should be paid special attention. Therefore, based on the difference information (D) extracted by the present invention, the present invention designs a new consistency loss function for the student model and the teacher model. This loss function can make the overall framework pay more attention to areas with high uncertainty during training, so that the overall framework can capture as many useful features as possible from areas with high uncertainty, thereby improving the segmentation performance of the overall framework. The consistency loss of the present invention is specifically expressed as:

[0099]

[0100] in, It is an indicator function that plays a role in judgment. H is the threshold coefficient of the selected area. and are all constant coefficients.

[0101] After the above process, the present invention obtains the consistency loss obtained from these 6 unlabeled slices. .

[0102] Afterwards, comprehensive D and Get the unsupervised loss , as shown below:

[0103]

[0104] Where p is the threshold coefficient.

[0105] Then combine it with the supervised loss obtained by passing 12 slices into the student model , we can get the final loss function of the present invention , as shown below:

[0106]

[0107] Where m is the threshold coefficient.

[0108] In this way, the present invention obtains the final loss of the framework of the present invention after some calculation through these 12 2D slices entering the framework , where all 12 slices enter the student model and 6 unlabeled slices enter the teacher model. The weight of the student model can be optimized through this final loss value, and the teacher model is optimized through the EMA of the student model weight. In this way, one iteration of the framework of the present invention is completed. Then the iteration is performed continuously, and a training epoch is completed after a certain number of iterations. After completing a training epoch, some labeled slices will be sent to the student model and the teacher model to verify the performance of the model. The specific verification process is to calculate the dice value between the model prediction and the corresponding label. The larger the dice value, the better the model prediction result and the better the model performance. This part of the labeled pictures has not entered the framework for training. When dividing the labeled data and unlabeled data entering the framework, several patients with unlabeled data can be reserved for verification from the 64 patients. After verification, one epoch is completed, and then the training is repeated for multiple epochs until the loss function The training can be stopped when the value no longer decreases and the performance of the model reaches the optimal level. Figure 5 It is the specific method framework of the present invention.

[0109] 3 Testing Phase

[0110] This stage is relatively simple. We directly feed the slice data in the test set into the trained model to obtain the prediction results. We then compare the slice prediction results with their corresponding labels and calculate some indicators that can represent the performance of the trained model. These indicators are as follows:

[0111] Dice similarity coefficient: ;

[0112] Jaccard similarity coefficient: ;

[0113] Surface average distance: ,

[0114] ;

[0115] Hausdorff distance_95: ;

[0116] Where TP, TN, FP, and FN refer to true positives, true negatives, false positives, and false negatives, respectively. X and Y represent the boundaries of the prediction and label, respectively. d(X,Y) represents the maximum distance between a point on X and the nearest point on Y. x represents the coordinates of a point on the boundary of the prediction, and y represents the coordinates of a point on the boundary of the label.

[0117] It should be noted that these indicators are all calculated based on 3D images. Therefore, after all the slices of a patient are predicted by the model, these 2D slices need to be combined into 3D images again, and the corresponding labels need to be combined into 3D images before these 3D indicators can be calculated.

[0118] The test model can be a trained student model or a trained teacher model. The main purpose is to select the better model by comparing the performance of the student model and the teacher model obtained in the final verification process of each epoch. Figure 7 Flowchart of the present invention.

[0119] The present invention has the following beneficial effects:

[0120] 1. Reduces the need for labeled data and eases the annotation burden on physicians: For medical images, obtaining labeled images is generally difficult. Accurate annotation requires a significant amount of time and expertise from medical personnel, significantly increasing the cost of obtaining labeled data and the burden on medical personnel. This problem persists for left atrial images. However, the method framework of this invention is a semi-supervised approach that can leverage unlabeled data to improve model performance. This reduces the need for labeled data and, to a certain extent, alleviates the annotation burden on physicians.

[0121] 2. Improved segmentation accuracy: This invention leverages inter-module variance information, which is often overlooked by most semi-supervised methods. This information, similar to uncertainty, can, to a certain extent, highlight challenging regions in the image, facilitating focus and information mining. Leveraging this information, the present invention designs a two-dimensional mean teacher (D-MT) framework. This framework, based on the mean teacher model, incorporates variance information to improve performance. This variance information is obtained by extracting the output differences between the final output of the teacher model and the intermediate and final outputs of the student model. This variance information is first constrained using a loss function. This variance information is then used to optimize the consistency loss between the student and teacher models. This guides the framework to focus on challenging regions with high variance, thereby capturing difficult-to-identify features and improving overall framework performance. This improved overall framework performance translates to improved model performance, which in turn improves the model's segmentation accuracy for left atrial images.

[0122] 3. Improved Efficiency: Most semi-supervised frameworks for various medical image segmentation tasks use 3D models as their base model. This is because compared to 2D models, 3D models have a simpler image processing pipeline, can take into account information from adjacent slices, and offer more stable and accurate segmentation performance. However, this high performance comes at a computational cost. Compared to 2D models, 3D models typically have more parameters, require more memory, and take longer to train, which are major limitations for their use. Furthermore, the limited amount of training data is a limitation for 3D models. 2D models use slices of 3D volume images, while 3D models use complete 3D volume images. While datasets typically have a limited number of complete 3D volume images, each 3D volume image can yield a large number of 2D slices. Therefore, in datasets with fewer 3D volume images, 2D models, leveraging the advantage of training data, can achieve comparable performance to 3D models while consuming less memory. The method framework (D-MT) of the present invention is based on a 2D model, and its performance is similar to that of more advanced methods based on 3D models, thereby improving the training speed of the overall framework in left atrial images, and being able to obtain a model with relatively high left atrial image segmentation accuracy in a shorter time, thereby improving efficiency.

[0123] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A 2D mean teacher model using difference information, which is a 2D model for left atrial image segmentation. The 2D mean teacher model includes the following steps during segmentation: Step 1: performing data preprocessing on the obtained left atrial data set, where the left atrial data set includes a plurality of left atrial images; Step 2: Divide the processed left atrial data set into a training set and a test set. The data in the training set is used for model training of the 2D mean teacher model. After multiple trainings, a 2D meanteacher model optimized on the training set is obtained. Step 3: Use the test set to verify the model obtained from the training set and further evaluate the performance of the model; Its characteristics are: In step 1, the 3D data in the left atrial dataset needs to be sliced first, and then the obtained slices are uniformly converted into the size required by the 2D mean teacher model; In step 2, the training set and test set are generally divided into 80% for the training set and 20% for the test set. Then, the slices in the training set are divided into labeled slices and unlabeled slices, and the number of unlabeled slices is guaranteed to be greater than that of labeled slices. In step 3, the 2D mean teacher model includes a student model and a teacher model. All labeled slices in the training set are processed in the student model, while unlabeled slices are processed in the teacher model and the student model. Both the student model and the teacher model are based on a basic 2D Unet network, but an additional 1×1 convolution layer and a sigmoid layer are added to the last upsampling layer of the Unet network decoder, and an additional upsampling layer, a 1×1 convolution layer, and a sigmoid layer are added to the second-to-last upsampling layer, thereby obtaining an intermediate prediction image of the same size as the final prediction image of the 2D Unet network. Among them, in the 2D mean teacher model, the student model and the teacher model have the same network architecture, and the weight of the teacher model Use the weights of the student model The exponential moving average EMA is updated, and the teacher model weight at step t will be updated to ,in Used to control the update rate of EMA, that is, the proportion of the student model weight at step t and the teacher model weight at step t-1 in the weight of the teacher model at step t; When the slice enters the student model, both the final and the middle of the student model will get the corresponding prediction results. Then, for the labeled slice, a supervised loss will be calculated. , this is a supervised loss It is calculated based on the prediction result generated by the student model after the labeled slice passes through the student model and the label corresponding to the labeled slice, as shown in the following formula: in, represents the dice loss function, which is used to evaluate the network prediction quality on labeled slice input; i Indicates the sequence number of the labeled slice, with a value of 1, 2, ..., L, where L is a positive integer representing the number of labeled slices; represents the segmentation network of the student model; is the network weight of the student model; For the i-th labeled slice; The label of the i-th labeled slice.

2. The 2D mean teacher model using difference information according to claim 1, characterized in that In step 1, all 3D data are converted into 2D slices along the Z axis, and then the images with darker targets in these 2D slices are brightened. At the same time, 2D slices without segmented targets, that is, 2D slices that do not contain the left atrium, are removed.

3. The 2D mean teacher model using difference information according to claim 2, characterized in that Brightening an image is achieved by amplifying the pixel values of the image as a whole.

4. The 2D mean teacher model using difference information according to claim 1, characterized in that The size required by the 2D meanteacher model is: the size of the 2D slice is transformed to 128×128, and the corresponding label is also transformed to 128×128.

5. The 2D mean teacher model using difference information according to claim 1, characterized in that Different interference is added to the unlabeled slices entering the student model and the teacher model respectively. The interference information added to the unlabeled slices entering the student model and the unlabeled slices entering the teacher model is different: First, we need to calculate the difference between the predictions of the student model and the teacher model for the same unlabeled slice; To calculate this difference, the following information is needed: the final prediction result of the student model for this unlabeled slice, the intermediate prediction result of the student model for this unlabeled slice, and the final prediction result of the teacher model for this unlabeled slice; the intermediate prediction result includes the prediction result generated by the last upsampling layer of the student model and the prediction result generated by the second to last upsampling layer of the student model; The difference information D is obtained by calculating the difference between the final prediction result of the teacher model for this unlabeled slice and the final prediction result and the intermediate prediction result of the student model for this unlabeled slice. The specific calculation method of the difference is to calculate the pixel difference between the images. After obtaining the difference information D, it can be supervised: use an image with all zero values and the extracted difference information D to calculate the difference loss ; This difference loss It is used to constrain the prediction consistency between the student model and the teacher model, and encourage the student model and the teacher model to learn from each other.

6. The 2D mean teacher model using difference information according to claim 5, characterized in that Difference information D and difference loss The specific calculation formula is: ， ； Where U represents the number of unlabeled slices, 、 and represents the segmentation network of the student model, The output of is the final output of the student model segmentation network. The output of is the output of the last upsampling layer of the student model segmentation network. The output of is the output of the penultimate upsampling layer of the student model segmentation network, represents the segmentation network of the teacher model, is the mean square error loss, is the network weight of the student model, is the network weight of the teacher model, is the network perturbation of the student model, is the network perturbation of the teacher model, a, b, c are all constant coefficients, and Black is an image of all zero values of the same size as D.

7. The 2D mean teacher model using difference information according to claim 6, characterized in that According to the extracted difference information D, a new consistency loss function is designed for the student model and the teacher model. ,for: in, It is an indicator function that plays a role in judgment. H is the threshold coefficient of the selected area. and All are constant coefficients; Afterwards, comprehensive D and Get the unsupervised loss , as shown below: Where p is the threshold coefficient; Then combine the supervised loss obtained by the slices entering the student model , get the final loss function of the 2D mean teacher model , as shown below: Where m is the threshold coefficient.

8. The 2D mean teacher model using difference information according to claim 1, characterized in that In step 3, we directly feed the slices in the test set into the trained model to obtain prediction results. We then compare the predicted results of the slices with their corresponding labels and calculate some metrics that represent the performance of the trained model. These metrics are as follows: Dice similarity coefficient: ; Jaccard similarity coefficient: ; Surface average distance: , ； Hausdorff distance_95: ; Where TP, TN, FP, and FN refer to true positives, true negatives, false positives, and false negatives, respectively; X and Y represent the boundaries of the prediction and label, respectively; d(X,Y) represents the maximum distance from a point on X to the nearest point on Y; x represents the coordinates of a point on the boundary of the prediction result, and y represents the coordinates of a point on the boundary of the label.

Images