Thyroid nodule semi-supervised positioning system based on double-flow CutMix

Abstract

The invention discloses a thyroid nodule semi-supervised positioning system based on double-flow CutMix, which fully utilizes the information of unlabeled images, considers the consistency of data preprocessing and the difference of distribution between labeled images and non-labeled images, and completes the mixing of labeled images and the mixing of unlabeled images through a double-flow CutMix method. Common semantic information between the two can be learned; according to an existing method, only copying marked data is considered, a part of foreground is simply copied from the marked data to unmarked data, a consistency learning strategy between the marked data and the unmarked data is ignored, and large distribution difference is possibly generated between the marked data and the unmarked data due to different data processing modes. The generalization of the model is further influenced; according to the method, through double-flow replication, the consistency strategy of the marked data and the non-marked data is better balanced, the distribution difference between the marked data and the non-marked data is aligned, and the non-marked data can be better utilized.

Description

Technical Field

[0001] The present invention relates to the technical field of ultrasonic images, and in particular to a thyroid nodule semi-supervised positioning system based on dual-stream CutMix. Background Art

[0002] Affected by the pressure and pace of life, the incidence of thyroid diseases has been on the rise in recent years. As one of the most common clinical signs, the prevalence of thyroid nodules in the population is around 4% to 7%. For doctors, accurately locating the area where thyroid nodules appear is conducive to more accurate judgment of the benign or malignant nature of the nodules, which becomes particularly important for the formulation of subsequent treatment plans for thyroid nodules.

[0003] At present, the main method for locating thyroid nodules is for doctors to use ultrasound images and pathology reports and combine their own experience to outline the nodule contour. With the rapid development of artificial intelligence and medical vision technology, ultrasound image thyroid nodule localization technology based on artificial intelligence has achieved relevant research results and has been clinically applied in some hospitals, which has improved the localization accuracy of thyroid nodules to a certain extent. The improvement of model positioning accuracy is closely related to the quantity and quality of data sets. In order to further improve the performance of thyroid nodule localization, the most efficient way is to increase training data. However, although a large number of thyroid ultrasound images can be collected, only a small number of cases have pathology reports, which leads to inaccurate annotation of doctors without pathology data. In addition, it is impractical to annotate all ultrasound images. In order to utilize limited high-quality annotated data and a large amount of unlabeled data, a semi-supervised thyroid nodule localization method was proposed, and its effectiveness in improving the accuracy of nodule localization was proved.

[0004] The core of semi-supervised learning lies in how to process unlabeled data. Existing related technologies try to use different transformations to process unlabeled data and constrain the feature consistency of the processed images; however, these technologies do not process labeled data in the same way. The inconsistent processing of labeled data and unlabeled data may lead to knowledge loss of the model and cannot reduce the gap in the empirical distribution between the two. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a thyroid nodule semi-supervised localization system based on dual-stream CutMix, including a data collection module, a data set division and annotation module, a teacher model pre-training module, a student model training module, a model evaluation module and a model reasoning verification module;

[0006] A data collection module, used for collecting ultrasound images from different sources;

[0007] The data set division and annotation module is used to divide the ultrasound images with corresponding pathology reports into labeled data sets, and the rest into unlabeled data sets; outline the range of thyroid nodules on the ultrasound images in the labeled data set according to the content of the pathology report, and then divide them into training sets, validation sets, and test sets in proportion; and divide all the ultrasound images in the unlabeled data set into the training set;

[0008] The teacher model pre-training module is used to pre-train the teacher model using a labeled training set;

[0009] The student model training module is used to define the parameter optimizer and perform image data processing, mask label prediction and processing, loss function calculation, and back-propagation update in sequence; the student model is trained, the teacher model freezes the gradient, and only updates the weights through moving average;

[0010] Among them, the image data processing is specifically as follows: randomly masking part of the labeled image, the pixel values ​​of this part of the area are filled with the pixel values ​​corresponding to the unlabeled image, and randomly masking part of the unlabeled image, the image values ​​of this part of the area are filled with the image values ​​corresponding to the labeled image; the mask label prediction and processing is specifically as follows: processing the unlabeled image through the teacher model to obtain the predicted mask of the unlabeled image, and taking the output of the current batch of teacher model predictions with confidence greater than the confidence threshold as the actual mask; the loss function calculation is specifically as follows: calculating the overall loss of the labeled mixed image and the unlabeled mixed image respectively; the back propagation update is specifically as follows: updating the parameters of the student model through the loss function, and updating the parameters of the teacher model through the moving average;

[0011] The model evaluation module is used to evaluate the performance of the student model using the validation set after each iteration;

[0012] The model inference verification module is used to load the student model with the highest performance in the historical training process and perform inference verification on the test data set to reflect the actual inference performance of the student model.

[0013] The technical solution further defined in the present invention is:

[0014] Furthermore, in the data collection module, ultrasound images of the nodule area without blood flow obstruction are collected from different hospitals and different ultrasound equipment, and the resolution of the ultrasound image is required to be greater than 600*1000 pixels.

[0015] As mentioned above, in the semi-supervised localization system for thyroid nodules based on dual-stream CutMix, in the data set division and annotation module, the doctor manually outlines the range of thyroid nodules on the ultrasound image in the labeled data set according to the content of the pathology report, and then divides the training set, validation set, and test set in a ratio of 7:1:2.

[0016] As described above, in the semi-supervised localization system for thyroid nodules based on two-stream CutMix, in the teacher model pre-training module, the training process is a supervised learning process, and the cross entropy loss and Dice loss are used to constrain the teacher model.

[0017] As described above, in a semi-supervised thyroid nodule localization system based on two-stream CutMix, when the student model training module processes image data, the labeled images are randomly blocked. The pixel value of this part of the area is determined by the unlabeled image The corresponding pixel values ​​are filled, and the processing process is shown as follows:

[0018]

[0019] Among them, M is a placeholder matrix, which indicates the original pixel information of the position that needs to be retained in the annotated image; ⊙ indicates the dot product operation; Indicates an annotated mixed image;

[0020] Randomly occlude unlabeled images The pixel value of this part of the area is determined by the labeled image The corresponding pixel values ​​are filled, and the processing process is shown as follows:

[0021]

[0022] in, Indicates that there are annotated images; represents an unlabeled image; M is a placeholder matrix, which represents the original pixel information that needs to be retained in the unlabeled image; ⊙ represents the dot product operation, Represents an unlabeled mixed image.

[0023] As described above, in the semi-supervised localization system for thyroid nodules based on two-stream CutMix, the process of mask label prediction and processing in the student model training module is as follows:

[0024]

[0025] Among them, Unet represents the defined segmentation network, Indicates the weight of the network as the teacher network during the epoch training, thres indicates the confidence threshold, and Represents the predicted mask for an unlabeled image.

[0026] As described above, a two-stream CutMix-based semi-supervised localization system for thyroid nodules was developed, and the confidence threshold thres was set to 0.5.

[0027] As described above, in a semi-supervised thyroid nodule localization system based on two-stream CutMix, when the student model training module calculates the loss function, the calculation formula for the overall loss of the labeled mixed image is as follows:

[0028]

[0029] Among them, L seg represents the segmentation loss function, which is a combination of Dice loss and cross entropy loss function; represents the weight of the student model at the epoch training; the overall loss consists of two parts, namely the mask loss of the labeled area and the mask loss of the unlabeled area; β(epoch) represents the hyperparameter that changes with the number of training times;

[0030] The overall loss for unlabeled mixed images is calculated as follows:

[0031]

[0032] Finally, there will be the overall loss of the labeled mixed image and the overall loss of unlabeled mixed images Add together to get the final loss function Λ all .

[0033] As described above, in a semi-supervised thyroid nodule localization system based on two-stream CutMix, when the student model training module performs back-propagation update, the parameters of the student network are updated through the loss function:

[0034]

[0035] Among them, SGD represents the optimizer used when updating parameters, lr represents the learning rate, represents the cumulative gradient;

[0036] Update the parameters of the teacher network by moving average:

[0037]

[0038] in, represents the weight of the student network during the current training, represents the weight of the teacher network during the last training, and w represents the weighting factor.

[0039] As described above, in the model evaluation module of the semi-supervised localization system for thyroid nodules based on dual-stream CutMix, the evaluation indicator used is the DICE coefficient; in the model reasoning verification module, the DICE coefficient of the test set is used to reflect the actual reasoning performance of the student model.

[0040] The beneficial effects of the present invention are:

[0041] In the present invention, the information of unlabeled images can be fully utilized, the demand for the number of labeled data is reduced, the labeling pressure of doctors is alleviated, and the segmentation accuracy during model training is guaranteed; compared with other semi-supervised methods, the main advantage of this method is that it takes into account the consistency of data preprocessing and the difference in distribution between labeled images and unlabeled images. Through the two-stream cutmix method, the mixing of labeled images and the mixing of unlabeled images are completed. Such a symmetrical processing method enables the common semantic information between the two to be learned; in addition, the method of the prior art only considers copying labeled data, simply copying part of the foreground from the labeled data to the unlabeled data. Such processing ignores the consistency learning strategy between labeled data and unlabeled data. The different data processing methods may lead to large distribution differences between labeled data and unlabeled data, which in turn affects the generalization of the model. The method of the present invention better balances the consistency strategy of labeled data and unlabeled data through two-stream replication, aligns the distribution difference between the two, and enables the model to better utilize these unlabeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the overall network architecture of the present invention. DETAILED DESCRIPTION

[0043] This embodiment provides a thyroid nodule semi-supervised localization system based on dual-stream CutMix. Figure 1 As shown, it includes a data collection module, a data set division and annotation module, a teacher model pre-training module, a student model training module, a model evaluation module and a model reasoning verification module.

[0044] The data collection module is used to collect ultrasound images from different sources. High-quality thyroid ultrasound images are collected from hospitals, which should meet the following standards: (1) The ultrasound image has a high resolution and contrast, that is, the resolution is greater than 600*1000 pixels. At this contrast, the thyroid contour can be seen in both dark and bright areas, and the thyroid contour is clearly visible. At the same time, there should be no obvious obstruction in the nodule area. (2) The ultrasound images should be collected from different hospitals and different ultrasound equipment. (3) The ultrasound image should have a sufficiently rich grayscale range, and the grayscale image pixel value histogram should show a smooth transition from black to white, rather than being overly concentrated in certain areas, so as to adapt to different scenarios.

[0045] The data set division and labeling module is used to divide the data set into labeled data sets and unlabeled data sets according to whether the image has a pathology report and whether the pathology report can correspond one-to-one with the image.

[0046] The ultrasound images with corresponding pathology reports are divided into labeled data sets, and the rest are divided into unlabeled data sets. For the labeled data sets, senior doctors outline the thyroid ultrasound images based on the pathology reports and outline the range of nodules. Then, the training set, validation set, and test set are divided according to the ratio of 7:1:2. The unlabeled data sets do not need to be labeled and are all divided into training sets.

[0047] The teacher model pre-training module is used to pre-train the teacher model using a labeled training set; the training process is a conventional supervised learning process, and the cross entropy loss and Dice loss are used to constrain the teacher model.

[0048] The student model training module is used to define the parameter optimizer and perform image data processing, mask label prediction and processing, loss function calculation and back-propagation update in sequence; the student model is trained, the teacher model freezes the gradient and only updates the weights through moving average.

[0049] The image data processing is as follows: Randomly block the labeled image The pixel value of this part of the area is determined by the unlabeled image The corresponding pixel values ​​are filled, and the processing process is shown as follows:

[0050]

[0051] Among them, M is a placeholder matrix, which indicates the original pixel information of the position that needs to be retained in the annotated image; ⊙ indicates the dot product operation; Indicates an annotated mixed image.

[0052] Similarly, randomly occlude unlabeled images The pixel value of this part of the area is determined by the labeled image The corresponding pixel values ​​are filled, and the processing process is shown as follows:

[0053]

[0054] in, Indicates that there are annotated images; represents an unlabeled image; M is a placeholder matrix, which represents the original pixel information that needs to be retained in the unlabeled image; ⊙ represents the dot product operation, Represents an unlabeled mixed image.

[0055] The specific steps of mask label prediction and processing are as follows: the unlabeled image is processed by the teacher model to obtain the predicted mask of the unlabeled image; according to experience, the output of the teacher model prediction of the current batch whose confidence is greater than the confidence threshold is taken as the actual mask. The confidence threshold is expressed as thres and is pre-set to 0.5. The process of mask label prediction and processing is shown in the following formula:

[0056]

[0057] Among them, Unet represents the defined segmentation network, Indicates that the network is the weight of the teacher network during the epoch training. and Represents the predicted mask for an unlabeled image.

[0058] The loss function is calculated as follows: the overall loss of the labeled mixed image and the unlabeled mixed image is calculated separately. The calculation formula for the overall loss of the labeled mixed image is as follows:

[0059]

[0060] Among them, L seg represents the segmentation loss function, which is a combination of Dice loss and cross entropy loss function; It represents the weight of the student model during the epoch training. The overall loss consists of two parts: the mask loss of the labeled area and the mask loss of the unlabeled area. β(epoch) represents the hyperparameter that changes with the number of training times, which is used to balance the loss of the two parts.

[0061] The overall loss for unlabeled mixed images is calculated as follows:

[0062]

[0063] The definitions of the relevant parameters in the above formula are the same as the loss formula for the labeled mixed image; finally, the overall loss of the labeled mixed image is and the overall loss of unlabeled mixed images Add together to get the final loss function Λ all .

[0064] In comparison, the mask of the labeled area is more accurate than the mask of the unlabeled area, and the mask of the unlabeled area contains more common semantic information. Therefore, this embodiment introduces a parameter β(epoch) that changes with the number of training times to balance the influence between the two.

[0065] Back propagation update is specifically: update the parameters of the student model through the loss function:

[0066]

[0067] Among them, SGD represents the optimizer used when updating parameters, lr represents the learning rate, represents the cumulative gradient.

[0068] Update the parameters of the teacher model by moving average:

[0069]

[0070] in, represents the weight of the student network during the current training, represents the weight of the teacher network during the last training, and w represents the weighting factor, which is pre-defined as 0.1.

[0071] The model evaluation module is used to evaluate the performance of the student model using the validation set after each iteration. The evaluation indicator used is the DICE coefficient. The DICE coefficient is used to calculate the similarity between the predicted results and the true labels. It is defined as the ratio of twice the intersection area of ​​the two sets (predicted results and true labels) to the total area of ​​the two sets. It is particularly suitable for binary image segmentation tasks. The higher the DICE coefficient, the higher the overlap between the predicted results and the true labels. The model with the highest DICE coefficient of historical results is saved during the training process.

[0072] The model reasoning verification module is used to load the student model with the highest DICE coefficient in the historical training process, perform reasoning verification on the test data set, and use the DICE coefficient of the test set to reflect the actual reasoning performance of the student model.

[0073] In order to alleviate the matching problem between labeled data and unlabeled data, this embodiment designs a universal process to perform similar processing on labeled data and unlabeled data, which requires: 1. Unlabeled data can learn comprehensive semantic distribution information from unlabeled data; 2. The empirical distribution of labeled data and unlabeled data can be aligned.

[0074] The two-stream CutMix method is an improvement on the CutMix method. The CutMix method was originally only used for information enhancement between unlabeled data. In order to take into account both labeled data and unlabeled data, this embodiment adopts a bidirectional CutMix method between labeled data and unlabeled data, that is, simultaneously performing the following: 1. Blocking part of the labeled data area and filling it with unlabeled data; 2. Blocking part of the unlabeled data area and filling it with labeled data; such a symmetrical processing flow can better encourage the model to learn common semantic information and align distribution differences.

[0075] This embodiment adopts the classic Mean-Teacher framework, and the segmentation network adopted is Unet. The method proposed in this embodiment only modifies the data processing and loss calculation, and has no binding relationship with the selection of the segmentation network architecture and backbone. Therefore, it can be plug-and-play and applicable to any segmentation network architecture and backbone. At the same time, the method of this embodiment is aimed at semi-supervised segmentation tasks in the medical field, and can be reused in various fields such as thyroid, skin diseases, and breast according to actual usage.

[0076] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solution formed by equivalent replacement or equivalent transformation falls within the protection scope required by the present invention.

Claims

1. A semi-supervised localization system for thyroid nodules based on dual-stream CutMix, characterized by: It includes data collection module, data set division and annotation module, teacher model pre-training module, student model training module, model evaluation module and model reasoning verification module; A data collection module, used for collecting ultrasound images from different sources; The data set division and annotation module is used to divide the ultrasound images with corresponding pathology reports into labeled data sets, and the rest into unlabeled data sets; outline the range of thyroid nodules on the ultrasound images in the labeled data set according to the content of the pathology report, and then divide them into training sets, validation sets, and test sets in proportion; and divide all the ultrasound images in the unlabeled data set into the training set; The teacher model pre-training module is used to pre-train the teacher model using a labeled training set; The student model training module is used to define the parameter optimizer and perform image data processing, mask label prediction and processing, loss function calculation, and back-propagation update in sequence; the student model is trained, the teacher model freezes the gradient, and only updates the weights through moving average; Among them, the image data processing is specifically as follows: randomly masking part of the labeled image, the pixel values ​​of this part of the area are filled with the pixel values ​​corresponding to the unlabeled image, and randomly masking part of the unlabeled image, the image values ​​of this part of the area are filled with the image values ​​corresponding to the labeled image; the mask label prediction and processing is specifically as follows: processing the unlabeled image through the teacher model to obtain the predicted mask of the unlabeled image, and taking the output of the current batch of teacher model predictions with confidence greater than the confidence threshold as the actual mask; the loss function calculation is specifically as follows: calculating the overall loss of the labeled mixed image and the unlabeled mixed image respectively; the back propagation update is specifically as follows: updating the parameters of the student model through the loss function, and updating the parameters of the teacher model through the moving average; The model evaluation module is used to evaluate the performance of the student model using the validation set after each iteration; The model inference verification module is used to load the student model with the highest performance in the historical training process and perform inference verification on the test data set to reflect the actual inference performance of the student model.

2. The thyroid nodule semi-supervised localization system based on dual-stream CutMix according to claim 1, characterized in that: In the data collection module, ultrasound images of nodule areas without blood flow obstruction are collected from different hospitals and different ultrasound equipment, and the resolution of the ultrasound images is required to be greater than 600*1000 pixels.

3. The thyroid nodule semi-supervised localization system based on dual-stream CutMix according to claim 1, characterized in that: In the dataset division and annotation module, doctors manually outline the range of thyroid nodules on the ultrasound images in the labeled dataset according to the content of the pathology report, and then divide the dataset into training set, validation set and test set in a ratio of 7:1:

2.

4. The thyroid nodule semi-supervised localization system based on dual-stream CutMix according to claim 1, characterized in that: In the teacher model pre-training module, the training process is a supervised learning process, and the teacher model is constrained by using cross entropy loss and Dice loss.

5. The thyroid nodule semi-supervised localization system based on dual-stream CutMix according to claim 1, characterized in that: When the student model training module processes image data, it randomly blocks the labeled images. The pixel value of this part of the area is determined by the unlabeled image The corresponding pixel values ​​are filled, and the processing process is shown as follows: Among them, M is a placeholder matrix, which indicates the original pixel information of the position that needs to be retained in the annotated image; ⊙ indicates the dot product operation; Indicates an annotated mixed image; Randomly occlude unlabeled images The pixel value of this part of the area is determined by the labeled image The corresponding pixel values ​​are filled, and the processing process is shown as follows: in, Indicates that there are annotated images; represents an unlabeled image; M is a placeholder matrix, which represents the original pixel information that needs to be retained in the unlabeled image; ⊙ represents the dot product operation, Represents an unlabeled mixed image.

6. The thyroid nodule semi-supervised localization system based on dual-stream CutMix according to claim 1, characterized in that: In the student model training module, the process of mask label prediction and processing is shown in the following formula: Among them, Unet represents the defined segmentation network, Indicates the weight of the network as the teacher network during the epoch training, thres indicates the confidence threshold, and Represents the predicted mask for an unlabeled image.

7. The thyroid nodule semi-supervised localization system based on dual-stream CutMix according to claim 6, characterized in that: The confidence threshold thres is set to 0.

5.

8. The dual-stream CutMix-based semi-supervised thyroid nodule localization system according to claim 1, characterized in that: When the student model training module calculates the loss function, the calculation formula for the overall loss of the labeled mixed image is as follows: Among them, L seg represents the segmentation loss function, which is a combination of Dice loss and cross entropy loss function; represents the weight of the student model at the epoch training; the overall loss consists of two parts, namely the mask loss of the labeled area and the mask loss of the unlabeled area; β(epoch) represents the hyperparameter that changes with the number of training times; The overall loss for unlabeled mixed images is calculated as follows: Finally, there will be the overall loss of the labeled mixed image and the overall loss of unlabeled mixed images Add together to get the final loss function Λ all .

9. The thyroid nodule semi-supervised localization system based on dual-stream CutMix according to claim 1, characterized in that: When the student model training module performs back propagation update, the parameters of the student network are updated through the loss function: Among them, SGD represents the optimizer used when updating parameters, lr represents the learning rate, represents the cumulative gradient; Update the parameters of the teacher network by moving average: in, represents the weight of the student network during the current training, represents the weight of the teacher network during the last training, and w represents the weighting factor.

10. The thyroid nodule semi-supervised localization system based on dual-stream CutMix according to claim 1, characterized in that: In the model evaluation module, the evaluation indicator used is the DICE coefficient; in the model reasoning verification module, the DICE coefficient of the test set is used to reflect the actual reasoning performance of the student model.

Images