End-to-end 3D breast ultrasound image tumor segmentation method based on Guide-MultiScale-Net

Abstract

The present invention discloses an end-to-end three-dimensional breast ultrasound image tumor segmentation method based on Guide-MultiScale-Net, including data preprocessing, model construction, model training and final segmentation results of the model. The architecture of the medical deep learning segmentation network is followed, that is, the idea of ​​encoder-decoder. The decoder adopts ResNet18 with better feature extraction. In order to restore detail information in the medical segmentation network, high-order semantic information is often combined with low-order semantic information, but there is more noise in the low-order semantic information, and direct fusion will be easily affected by noise. Therefore, high-order semantic information is used to guide low-order semantic information, suppress non-lesion areas, activate lesion areas, and reduce the impact of non-lesions. Since tumor lesions vary in size, multi-scale fusion based on a gating mechanism is used to capture multi-scale information and improve segmentation performance. Deep supervision technology is used to accelerate gradient propagation, and the information learned in the middle layer is made more discriminative, thereby improving the segmentation results of the model.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology and is a new method for tumor segmentation in three-dimensional breast ultrasound images. Background Art

[0002] Breast cancer still ranks first among malignant tumors in women in my country. Early detection and diagnosis are the key to breast cancer prevention and treatment. There are many ways to diagnose breast cancer. Ultrasound imaging has become the most effective way to diagnose breast lesions due to its advantages of no radiation, low cost, and fast imaging. Among them, three-dimensional ultrasound images have more information than two-dimensional ultrasound images, so they can better help doctors obtain information about tumors. However, the judgment and understanding of traditional breast ultrasound images are all done manually by doctors, which will cause doctor fatigue and inaccurate judgments, resulting in an increase in the misdiagnosis rate. Therefore, in order to reduce the burden on doctors and improve diagnostic results, it is particularly important to study reliable three-dimensional breast ultrasound tumor segmentation.

[0003] At present, the segmentation of 3D breast ultrasound tumor lesions is mainly divided into traditional machine learning methods and deep learning methods. Traditional machine learning methods mainly segment lesion areas based on texture, shape, and grayscale information. This method requires human participation in decision-making, has weak generalization and intelligence, and the segmentation results are not as good as deep learning methods. Existing deep learning methods mainly use end-to-end training methods to avoid human participation in decision-making, but the segmentation results are still not good enough. Small lesions are easily missed, larger lesions are not fully segmented, and the segmentation at the boundaries is discontinuous.

[0004] In recent years, deep learning has made great progress in image segmentation, object detection, posture estimation, and object tracking. This is due to the super learning and generalization capabilities of convolution. This makes it possible to design complex and high-precision end-to-end segmentation models.

[0005] The difference between the innovation of the present invention and the prior art

[0006] The existing three-dimensional breast ultrasound tumor segmentation based on deep learning mainly adopts a U-shaped network structure, but it is easy to miss smaller tumor lesions, incomplete segmentation of larger tumor lesions, and inaccurate tumor segmentation. The present invention proposes an end-to-end deep learning segmentation network. In view of the inaccurate tumor segmentation, a guidance module is proposed to reduce the impact of noise and improve the accuracy of tumor segmentation. In view of the inaccurate segmentation of different sizes and shapes, this paper proposes a fusion module based on gated multi-scale. Compared with the previous multi-scale fusion, this module can automatically select useful scales for fusion according to the current input. In order to improve the distinguishing ability of the middle layer, the present invention uses deep supervision technology to further improve the segmentation performance. The present invention conducts comparative experiments on the data set of three-dimensional breast ultrasound images provided by the School of Medicine of Lanzhou University, and the segmentation Dice value (an evaluation index of the segmentation algorithm) reaches 78.1. It is better than the results of the relevant currently disclosed experimental methods, and the mis-segmentation of lesions and missed segmentation areas are also significantly reduced. Summary of the invention

[0007] In order to solve the problem that the existing breast ultrasound image segmentation technology has low accuracy and complex data processing steps when processing multi-size lesions segmentation.

[0008] The technical solution adopted by the present invention is an effective end-to-end three-dimensional breast ultrasound tumor image segmentation method, which includes the following steps: data preprocessing, model construction, model training and model segmentation results.

[0009] Step 1, data preprocessing: Since there are relatively few 3D breast ultrasound images, if the model is directly used for learning, it is easy to cause overfitting, so data enhancement is needed to avoid model overfitting. 3D data is different from 2D data, and there are fewer data enhancement methods available, so this method mainly uses operations such as rotation, translation, and scaling.

[0010] Step 2: Model construction, using the encoder-decoder architecture.

[0011] Step 2.1, use ResNet18 with stronger feature extraction capability as the encoder of the present invention.

[0012] In step 2.2, a guidance module is proposed. In the feature fusion process, high-order semantic features are used to guide low-order features to suppress noise in low-order features and avoid inaccurate segmentation caused by noise.

[0013] Step 2.3, a multi-scale fusion module is proposed. Since the sizes of tumors in breast ultrasound images vary, it is difficult for traditional UNet networks to capture multi-scale features. Therefore, the present invention proposes a multi-scale module to capture multi-scale information in the model.

[0014] Step 2.4 introduces deep supervision technology to add constraints to the middle layer of the model to prevent the model from overfitting. It also enables the middle layer of the model to learn features with distinguishing capabilities and accelerate model convergence.

[0015] Step 3: Determine the loss function. The loss function is defined as:

[0016] The loss function corresponding to the top-level output of the model is defined as:

[0017]

[0018] This loss is composed of two kinds of losses, the former is weightCrossLoss, and the latter is DiceLoss, where α∈[0,1] represents the balance factor, which controls the ratio of the two losses, and is taken as 0.5 in the present invention. β∈[0,1] represents the weight for controlling false negatives and false positives, and is taken as 0.5 in the present invention. A value greater than 0.5 tends to penalize false positives, and a value less than 0.5 tends to penalize false negatives. N represents the number of pixels, t∈{0,1} represents the true segmentation result, p∈[0,1] represents the result of the model prediction processing, K represents the type of segmentation, and the default value is 1, and S represents the scaling factor, and the default value is 1e-5.

[0019] The loss function corresponding to the output of layer i∈{1, 2, 3, 4} in the decoder is defined as:

[0020]

[0021] Where N represents the total number of pixels, β∈[0,1] represents the control of the false negative and false positive error ratios, and is taken as 0.5 in the present invention. If it is greater than 0.5, the penalty for false positives will be greater, otherwise the penalty for false negatives will be greater. The present invention takes 0.5, t∈{0,1} represents the true label, and p∈[0,1] represents the probability value predicted by the model.

[0022] The total loss is defined as:

[0023]

[0024] The total loss function of the model consists of two steps. The first part is composed of the loss of the top layer, and is composed of the sum of the loss functions corresponding to the outputs of the four intermediate layer models.

[0025] Step 4: Network training: Use the back propagation mechanism of the neural network to optimize the model parameters using the loss function defined in the previous step until the model converges.

[0026] Step 5, model prediction: Use the trained segmentation model to segment the test data and obtain the segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a real annotated image corresponding to the breast ultrasound image involved in the present invention.

[0028] Figure 2 The network structure based on Guide-MultiScale-Net involved in the present invention.

[0029] Figure 3 It is the submodule structure involved in the network structure of step 2 involved in the present invention.

[0030] Figure 4 This is the segmentation execution process of step 5 involved in the present invention. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] The data set used in this paper is 171 three-dimensional breast ultrasound images provided by Lanzhou University School of Medicine. The average volume of each patient's slice is 318x552x714, and the thickness of the slice is 1x0.2x0.07. Due to the small amount of data, this paper performs data expansion, including flipping, random scaling, random center cropping, random translation and other methods. The final model uses a five-fold cross validation to evaluate the performance of the model.

[0033] Step 1: Model construction. The model adopts the encoder-decoder idea and introduces the residual idea into the encoder to improve the feature extraction ability of the model and the convergence speed of the model.

[0034] In step 1.1, a guidance module is proposed. In the feature fusion process, high-order semantic features are used to guide low-order features to suppress noise in low-order features and avoid inaccurate segmentation caused by noise.

[0035] Step 1.2, a multi-scale fusion module is proposed. Since the sizes of tumors in breast ultrasound images vary, it is difficult for traditional UNet networks to capture multi-scale features. Therefore, the present invention proposes a multi-scale module to capture multi-scale information in the model.

[0036] Step 1.3 introduces deep supervision technology to add constraints to the middle layer of the model to prevent the model from overfitting. It also enables the middle layer of the model to learn features with distinguishing capabilities and accelerate model convergence.

[0037] Step 1.4, determine the loss function, which is defined as:

[0038] The loss function corresponding to the top-level output of the model is defined as:

[0039]

[0040] This loss is composed of two kinds of losses, the former is weightCrossLoss, and the latter is DiceLoss, where α∈[0,1] represents the balance factor, which controls the ratio of the two losses, and is taken as 0.5 in the present invention. β∈[0,1] represents the weight for controlling false negatives and false positives, and is taken as 0.5 in the present invention. A value greater than 0.5 tends to penalize false positives, and a value less than 0.5 tends to penalize false negatives. N represents the number of pixels, t∈{0,1} represents the true segmentation result, p∈[0,1] represents the result of the model prediction processing, K represents the type of segmentation, and the default value is 1, and S represents the scaling factor, and the default value is 1e-5.

[0041] The loss function corresponding to the output of layer i∈{1, 2, 3, 4} in the decoder is defined as:

[0042]

[0043] Where N represents the total number of pixels, β∈[0,1] represents the control of the false negative and false positive error ratios, and is taken as 0.5 in the present invention. If it is greater than 0.5, the penalty for false positives will be greater, otherwise the penalty for false negatives will be greater. The present invention takes 0.5, t∈{0,1} represents the true label, and p∈[0,1] represents the probability value predicted by the model.

[0044] The total loss is defined as:

[0045]

[0046] The total loss function of the model consists of two steps. The first part is composed of the loss of the top layer, and is composed of the sum of the loss functions corresponding to the outputs of the four intermediate layer models.

[0047] Step 2, network training: Use the total loss of the previous step as the error and use the neural network back propagation optimization algorithm to train the model until it converges through a continuous input-output-calculation error-back propagation error optimization process.

[0048] The hyperparameters used in model segmentation are shown in Table X below:

[0049] parameter meaning value Batch_size The size of each batch fed into the network 8 epochs Number of rounds of model training 100 Learning_rate The learning rate used by the model 1e-3 Weight_decay Weight decay coefficient 5e-5 Optimizer Selected optimizer Adam Schedular Learning rate adjustment strategy ReduceLROnPlateau α Control the ratio between the two losses 0.5 β Controlling the false negative and false positive ratios 0.5

[0050] The experimental results are as follows:

[0051]

[0052] The method of the present invention constructs a segmentation model based on the Guide-MultiScale-Net end-to-end neural network, and trains the model through a custom loss function, so that the model has the ability to distinguish between the lesion and the parts similar to the lesion surrounding the breast ultrasound segmentation. During the test process, the method of the invention has a high accuracy in processing the segmentation of multi-sized lesion areas and the image segmentation of the lesion surrounding and the lesion. Due to this feature, the method can be widely used in medical clinical and other scenarios.

Claims

1. An end-to-end 3D breast ultrasound image tumor segmentation method based on Guide-MultiScale-Net, characterized in that: The following steps are involved: Step 1, obtaining a data set of breast ultrasound images and performing data preprocessing; Step 2: Use the pytorch deep learning framework to build the model; Step 3: Define the loss function; Step 4: Train the model until convergence; Step 5: Perform breast tumor lesion segmentation on the trained model to obtain the final segmentation result; The loss function corresponding to the top-level output of the model is defined as: In step 3, the loss function consists of two types, the former is weightCrossLoss, and the latter is DiceLoss, where α∈[0,1] represents the balance factor, which controls the ratio of the two losses and is taken as 0.5; β∈[0,1] represents the weight of controlling false negatives and false positives, which is taken as 0.

5. If it is greater than 0.5, it tends to penalize false positives, and if it is less than 0.5, it tends to penalize false negatives; N represents the number of pixels, t∈{0,1} represents the true segmentation result, p∈[0,1] represents the result of the model prediction processing, where K represents the type of segmentation, the default value is 1, and S represents the scaling factor, the default value is 1e-5; The loss function corresponding to the output of layer i∈{1,2,3,4} in the decoder is defined as: Where N represents the total number of pixels, β∈[0,1] represents the control of the false negative and false positive error ratio, which is 0.

5. If it is greater than 0.5, it tends to penalize false positives, otherwise it tends to penalize false negatives, which is 0.

5. β∈{0,1} represents the true label, and p∈[0,1] represents the probability value predicted by the model. The total loss is defined as: The total loss function of the model consists of two steps. The first part is composed of the loss of the top layer, and is composed of the sum of the loss functions corresponding to the outputs of the four intermediate layers.

2. According to the Guide-MultiScale-Net end-to-end 3D breast ultrasound image tumor segmentation method based on claim 1, the data preprocessing methods used in step 1 include random center cropping, random translation, random scaling, and normalization operations.

3. The Guide-MultiScale-Net end-to-end 3D breast ultrasound image tumor segmentation method according to claim 1, characterized in that Step 2: Model construction follows the encoder-decoder method with the following changes: 1) Introducing the residual idea into the encoder to improve the model feature extraction capability and the convergence speed of the model; 2) Propose a guidance module, which uses high-order semantic features to guide low-order features during feature fusion to suppress noise in low-order features and avoid inaccurate segmentation caused by noise; 3) A multi-scale fusion module is proposed. Since the sizes of tumors in breast ultrasound images vary, it is difficult for the UNet network to capture multi-scale features. Therefore, a multi-scale module is proposed to capture multi-scale information in the model. 4) Introducing deep supervision technology, adding constraints to the middle layer of the model can prevent the model from overfitting, and also enable the middle layer of the model to learn features with distinguishing capabilities, thereby accelerating model convergence.

Images