A method for tumor segmentation in breast ultrasound images

Abstract

The present invention discloses a method for segmenting tumors in breast ultrasound images, which includes preprocessing of breast ultrasound image data, building a deep neural network model, defining a loss function, model training, and result generation. In data preprocessing, the method of first mirror filling and then cropping is used, which neither changes the morphology of the breast tumor nor can obtain a breast ultrasound image that meets the size requirements. In the step of building a deep neural network model, the design pattern of the UNet model is followed as a whole. The present invention uses ResNet18 as the encoder of the entire network, so it has a stronger feature extraction capability, thereby being able to obtain higher accuracy. In the model decoder part, deep supervision technology is used to supervise the learning of each layer, and the channel-by-channel weighting module of SENet is added; in addition to eliminating the problems of incorrect segmentation and discontinuous segmentation boundaries, the present invention can also accurately capture tumor boundaries.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology and is an effective new method for tumor segmentation in breast ultrasound images. Background Art

[0002] Deep learning has become a hot topic in artificial intelligence. A rapidly developing machine learning technique, deep learning uses deep artificial neural networks as its model and leverages powerful hardware, massive amounts of data, and optimized algorithms to achieve high-performance learning. Deep convolutional neural networks (CNNs) have achieved remarkable results in many computer vision tasks, such as object classification, object detection, and object segmentation, significantly surpassing traditional machine learning and artificial intelligence technologies in many areas.

[0003] Breast cancer is one of the most common causes of death among women. While X-rays such as mammography can detect tumors in the early stages, they lack the ability to accurately identify dense breasts and carry a certain amount of radioactivity, which can be harmful to pregnant women and other groups. In contrast, breast ultrasound technology offers advantages such as ease of access, lack of radiation, and low equipment and treatment costs, making it an effective and versatile diagnostic tool. However, ultrasound breast images often exhibit high levels of noise, uneven grayscale, and complex and variable tumor shapes, significantly increasing the difficulty of segmentation and classification.

[0004] In recent years, some scholars have proposed different breast ultrasound image segmentation methods, such as active contour models and graph-based algorithms. These non-deep learning methods often require human participation in the segmentation process and are somewhat subjective. Another example is models such as UNet, Attention-UNet, and UNet++. These deep learning methods use end-to-end training to avoid human participation, but often have shortcomings such as mis-segmentation, under-segmentation, and discontinuous segmentation boundaries. Summary of the Invention

[0005] In response to the above-mentioned problems existing in the prior art, the present invention proposes a new and effective method for tumor segmentation in breast ultrasound images. The technical solution of the present invention is: first, the neural network model in the present invention follows the design pattern of the UNet model as a whole. The difference is that, unlike the original encoder of UNet, ResNet18 is used as the encoder of the entire network, so it has a stronger feature extraction capability, thereby being able to obtain higher accuracy. In addition, deep supervision technology is used in the decoder part of the model to supervise the learning of each layer, thereby accelerating the convergence of the model. Moreover, the channel-by-channel weighting module of SENet is added. By introducing such an attention mechanism, it can automatically balance the different effects of high-order coding features on the segmentation results, so that the model can more accurately capture the tumor boundaries in ultrasound images, bringing higher segmentation accuracy.

[0006] The technical solution adopted in the present invention is:

[0007] An effective breast ultrasound image tumor segmentation method includes the following steps: breast ultrasound image data preprocessing, deep neural network model construction, model training and result generation.

[0008] Step 1: Breast ultrasound image data preprocessing: For datasets with relatively small sample sizes, data augmentation, including flipping and rotation, is required. When resizing ultrasound images in breast ultrasound datasets with a small number of images and varying image sizes, especially those with large aspect ratios, directly resizing them through interpolation is detrimental to neural network learning. This method over-stretches the tumor morphology in the image, a distortion not necessarily present in real-world scenarios. In other words, this over-stretched tumor morphology does not conform to the distribution of real-world breast ultrasound data, widening the gap between the training and test sets. This increases model variance and ultimately leads to a significant delta error on the test set relative to the training set, resulting in high test set loss. Given the physical properties of human breast tissue, using data augmentation methods such as non-rigid deformations should be reasonable and feasible. However, improper parameter settings can also lead to these issues. The data augmentation method first determines whether the height of the original image A is below a set height threshold (448 in this case). If not, no processing is performed. If so, the original image A is mirrored upward and spliced to the upper part of the original image A. The original image A is mirrored downward and spliced to the lower part of the original image A to obtain image B that meets the height requirement. Then, it is determined whether the width of image B is below a set width threshold (512 in this case). If so, image B is mirrored left and right and spliced to the left and right sides to obtain image C that meets the height and width requirements. Finally, image C is center-cropped to obtain an image that meets the size requirements (448×448 in this case). Breast tumors are generally located in the center of the image, so using this resizing method will not change the tumor morphology.

[0009] Step 2: Build a neural network model: After processing the breast ultrasound images from Step 1, they are fed into the neural network designed in this step. Following the UNet architecture, the neural network is built using a step-by-step enhancement approach.

[0010] In step 2.1, we introduce ResNet18 as the encoder in the UNet architecture. ResNet18 consists of 18 convolutional and fully connected layers, with the final fully connected layer omitted. ResNet's identity mapping structure enables the training of extremely deep neural networks, addressing issues such as reduced accuracy and vanishing gradients, thereby improving the model's feature extraction capabilities. The initialization parameters for ResNet18 are those pre-trained on ImageNet.

[0011] In step 2.2, based on step 2.1, a deep supervision method is used to add auxiliary branches to the four layers of the decoder. Each newly added auxiliary branch consists of three convolutional layers. The newly added auxiliary branches can play a role in judging the quality of the hidden layer feature maps during training, thereby guiding the convergence of the model.

[0012] In step 2.3, based on step 2.2, the SENet attention module is added to the highest semantic layer of the model encoder. The attention module can improve the model's sensitivity to channel features. By weighting the channels, it emphasizes valid information related to the tumor and suppresses invalid information unrelated to the tumor, making the model more accurate in processing tumor edges.

[0013] Step 3: Determine the loss function. The breast ultrasound image dataset has a class imbalance problem, that is, in a breast ultrasound image, the number of tumor pixels is very small, and the vast majority are background pixels. The cross entropy loss function can solve the output imbalance problem, that is, the problem of false positives and false negatives, but it cannot effectively deal with the class imbalance problem. The Dice loss function can solve the class imbalance problem, but it cannot effectively deal with the output imbalance problem. Therefore, based on the characteristics of the breast ultrasound image dataset, the cross entropy loss function and the Dice loss function are combined, and the loss function is defined as follows:

[0014] Final output of the model The loss function is defined as:

[0015]

[0016] The loss function of deep supervision at layer i is defined as:

[0017]

[0018] The total loss is defined as:

[0019]

[0020] in, Represents the final segmentation prediction output of the model, y represents the true segmentation label, i represents the deep supervision branch index, Represents the output of the i-th deep supervision branch, binary_cross_entropy represents the binary cross entropy loss, and dice_loss represents the Dice loss.

[0021] Step 4: Network training: After preprocessing, the breast ultrasound image is input into the neural network designed in Step 2. This produces a predicted output from the neural network. This output is compared with the true label corresponding to the breast ultrasound image using the loss function designed in Step 3 to calculate the loss. The neural network backpropagation optimization algorithm is then used to train the model until convergence through a continuous optimization process of input-output-calculation error-backpropagation error.

[0022] Step 5, result generation: Use the trained segmentation model to segment the test data and obtain the segmentation results.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] (1) When scaling general natural images, linear interpolation is reasonable. However, for breast ultrasound images, especially those with a large aspect ratio, simple linear interpolation is unreasonable and can easily cause tumor distortion, which does not facilitate model training and generalization. The present invention uses a method of mirroring and filling followed by cropping, which can maintain the tumor morphology while obtaining an ultrasound image that meets the size requirements, which is beneficial to model training and improves the model's generalization ability.

[0025] (2) Breast ultrasound images are noisy, have blurred textures, and have uneven grayscale distribution, all of which pose challenges to breast ultrasound image tumor segmentation. The neural network designed in the present invention can effectively extract ultrasound image features while avoiding the problem of gradient vanishing. The introduction of the attention mechanism can suppress information irrelevant to the tumor and emphasize information related to the tumor, thereby improving the accuracy of segmentation. The use of deep supervision technology accelerates the entire model training process. In view of the characteristics of the breast ultrasound image dataset, the cross entropy loss function is combined with the Dice loss function, which can effectively solve the problem of class imbalance and the problem of output imbalance. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the method of the present invention;

[0027] Figure 2 This is a diagram of the size adjustment method of mirroring and filling first and then cropping;

[0028] Figure 3 It is a schematic diagram of the deep neural network structure of the present invention;

[0029] Figure 4This is the segmentation result after only introducing ResNet18 and deep supervision technology;

[0030] Figure 5 is the final segmentation result of the present invention;

[0031] Figure 6 This is a comparison diagram of the final segmentation results of the method of the present invention and other methods. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0033] In this example, the goal is to train a deep neural network model to effectively segment tumors in breast ultrasound images. The dataset used consists of 163 images from different women, each with one or more tumors, and most of the images are of different sizes.

[0034] See also Figure 1 As shown, the specific steps of the method in this embodiment are as follows:

[0035] Step 1, breast ultrasound image data preprocessing: The data enhancement methods selected by the present invention are mirror filling, center cropping, horizontal and vertical flipping and rotation. The schematic diagram of the size adjustment method of first mirror filling and then cropping is as follows Figure 2 After cropping, data augmentation is performed using random horizontal flipping, vertical flipping, rotation, and transposition, and pixel values are normalized to [0, 1].

[0036] Step 2: Build a neural network model: After the breast ultrasound images are preprocessed in Step 1, they are fed into the neural network model designed in this step. Following the UNet architecture, the neural network is built using a stepwise enhancement approach.

[0037] In step 2.1, introduce ResNet-18 and discard the final fully connected layer of ResNet-18, using it as the encoder in the entire network model to improve the model's feature extraction capabilities. Initialize the parameters of ResNet-18 using the pre-trained parameters of ResNet-18 on ImageNet.

[0038] In step 2.2, based on step 2.1, we use a deep supervision method to add auxiliary branches to the four layers in the decoder to guide the convergence of the model.

[0039] In step 2.3, based on step 2.2, the SENet attention module is added to the highest semantic layer of the model encoder to make the model process the tumor edge more accurately.

[0040] Step 3: Determine the loss function. Based on the characteristics of the breast ultrasound image dataset, the cross entropy loss function is combined with the Dice loss function. The loss function is defined as follows:

[0041] Final output of the model The loss function is defined as:

[0042]

[0043] The loss function of deep supervision at layer i is defined as:

[0044]

[0045] The total loss is defined as:

[0046]

[0047] in, Represents the final segmentation prediction output of the model, y represents the true segmentation label, i represents the deep supervision branch index, Represents the output of the i-th deep supervision branch, binary_cross_entropy represents the binary cross entropy loss function, and dice_loss represents the Dice loss function.

[0048] Figure 3 This is a complete neural network model structure diagram in the present invention.

[0049] Step 4, Network Training: After the breast ultrasound image data is preprocessed in Step 1, it is input into the neural network designed in Step 2. The neural network outputs a segmentation prediction. This output is compared with the true label corresponding to the breast ultrasound image using the loss function designed in Step 3 to calculate the loss. Then, using the neural network backpropagation optimization algorithm, the model is trained until convergence through a continuous input-output-calculation error-backpropagation error optimization process. This embodiment runs on a server with an NVIDIA GeForce 1080Ti (12G) GPU, with a batch size of 8. The Adam optimizer is used to optimize the network parameters. The two evaluation metrics used are Dice and Intersection over Union (IoU), as shown in the following formulas:

[0050]

[0051] Step 5, result generation: Use the trained segmentation model to segment the test data and obtain the segmentation results.

[0052] Figure 4The figure shows the test results of the neural network model after training without adding the attention mechanism (step 2.3). As can be seen from the figure, the neural network model at this time does not have the problem of incorrect segmentation and discontinuous segmentation boundaries. However, as can be seen from the enlarged part, the model at this time does not accurately capture the tumor boundary.

[0053] Figure 5 The figure shows the test results of the final neural network model of the present invention after training. It can be seen from the figure that the neural network model at this time has no mis-segmentation and no problem of discontinuous segmentation boundaries and can accurately capture the tumor boundary.

[0054] Figure 6 This is a comparison chart of the true segmentation result, the segmentation result of the present invention, and the segmentation results of other methods.

[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements made within the spirit and principles of the present invention, and the use of the loss function shown in the statement in other networks, etc., should be included in the scope of protection of the present invention.

Claims

1. A method for segmenting tumors in breast ultrasound images, characterized in that: The following steps are involved: Step 1: Obtain a breast ultrasound image dataset and perform data preprocessing; Step 2: Use the Pytorch deep learning framework to build a neural network model; Step 3: Define the loss function; Step 4: Use the preprocessed breast ultrasound images and the defined loss function to train the neural network model; Step 5: Use the neural network model trained in step 4 to perform tumor segmentation on the test data to obtain the segmentation results; In step 1, the acquired breast ultrasound image is first mirror-filled. Specifically, it is first determined whether the height of the original image A is lower than the set height threshold of 448. If not, no processing is performed. If so, the original image A is mirror-flipped upward and spliced to the upper part of the original image A, and the original image A is mirror-flipped downward and spliced to the lower part of the original image A to obtain an image B that meets the height requirement. Then, it is determined whether the width of image B is lower than the set width threshold of 512. If so, image B is mirror-flipped left and right and spliced to the left and right sides to obtain an image C that meets the height and width requirements. Finally, image C is center-cropped to obtain an image that meets the size requirement of 448×448. After cropping, horizontal flip, vertical flip, rotation, and transposition data augmentation methods are randomly used, and the pixel values are normalized to between [0,1]. In step 2, we build a neural network using a step-by-step enhancement approach based on the UNet architecture. This involves the following steps: Step 21: Introduce ResNet18 and discard the last fully connected layer of ResNet18, using it as the encoder of the entire network model to improve the feature extraction capability of the model; use the pre-trained parameters of ResNet18 on ImageNet to initialize the parameters of ResNet18; Step 22: Use deep supervision to add auxiliary branches to each layer of the decoder. Each newly added auxiliary branch consists of three convolutional layers. The newly added auxiliary branches can play a role in judging the quality of the hidden layer feature map during training. Step 23: Add the SENet attention module to the highest semantic layer of the model encoder. The attention module can improve the model's sensitivity to channel features. By weighting the channels, it emphasizes valid information and suppresses invalid information, making the model more accurate in processing tumor edges. In step 3, the neural network model loss function combines the cross entropy loss function with the Dice loss function and is defined as follows: Final output of the model The loss function is defined as: The loss function of deep supervision at layer i is defined as: The total loss is defined as: in, Represents the final segmentation prediction output of the model, y represents the true segmentation label, i represents the deep supervision branch index, Represents the output of the i-th deep supervision branch, binary_cross_entropy represents the binary cross entropy loss, and dice_loss represents the Dice loss.

Images