Breast ultrasound image segmentation method based on deep learning

Abstract

The invention provides a breast ultrasound image segmentation method based on deep learning, which comprises the following steps of: performing data enhancement on an image in a data set by using a breast cancer ultrasound data set which is widely used in previous research through methods of coordinate transformation, gray mapping and the like so as to extract features such as color, texture, granularity and the like; a medical expert carries out manual annotation on a tumor part in a breast ultrasound image data set, and due to the fact that breast image annotation needs to consume a large amount of labor cost and time cost, in order to more effectively utilize existing data, a training data set is enhanced; according to the breast ultrasound image segmentation method based on deep learning, in an actual diagnosis site, more lesion areas can be quickly and accurately identified, and meanwhile, the phenomena of false detection and missing detection are fewer; the training data is greatly increased after data enhancement, the semantic information of the image is richer, the number of effective features learned by the model during training is increased, and meanwhile, the oscillation condition generated in network training is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image analysis, and particularly relates to a breast ultrasound image segmentation method based on deep learning. Background Art

[0002] Breast cancer is one of the most common and serious types of cancer in women and is a global health problem. The World Health Organization reports that breast cancer is the most common disease in the world, with approximately 626,700 women dying from cancer-related diseases each year. More than 2 million new cases were diagnosed in 2018. Among them, there are some potential risk factors for breast cancer, such as age, genes, obesity, cigarettes, drugs, and contraceptive measures. Due to the lack of a full understanding of the causes of breast cancer, it is impossible to effectively prevent this disease. Research shows that due to the large population base, the number of breast cancer patients is increasing, and a large number of ultrasonic images are generated every day. Due to the diverse appearance and blurred boundaries of breast lesions, radiologists may misdiagnose breast cancer, and in some cases, it may even lead to the ineffective detection of breast tumors. However, the number of radiologists available to analyze these medical images is limited, which requires more specialized detection centers and medical experts.

[0003] Breast cancer is usually detected by mammography, ultrasound, and MRI imaging modalities. Through these techniques, breast cancer can be detected at an early stage to a large extent. However, mammography is often used because it is very effective in early tumor detection. However, this technique also has inherent limitations. Due to the close surrounding tissues and similar attenuation coefficients to breast tumors, the misdiagnosis rate is high, increasing the risk of radiation exposure for patients. Ultrasonic imaging examination methods, as a safer alternative to mammography, are used for the preliminary detection of breast cancer. The effectiveness of diagnosing breast cancer by observing ultrasonic images depends on the experience and skills of radiologists. Doctors interpret the speckle noise, complexity, and existence in ultrasonic images, and there are differences among observers. With the development of computer vision technology, the ability of artificial intelligence in some image segmentation tasks has approached that of humans. The present invention proposes a breast ultrasound image segmentation method based on deep learning, which can automatically segment the input breast ultrasound images, facilitating the rapid diagnosis of doctors. The lightweight algorithm proposed by the present invention can be deployed on portable edge computing devices, enabling the system to operate independently of bulky deep learning workstations and be deployed on doctors' desks, greatly improving the convenience of device use and thus expanding the application scenarios of the device. Therefore, the present invention proposes a breast ultrasound image segmentation method based on deep learning to solve the problems existing in the prior art. Summary of the Invention

[0004] In view of the deficiencies of the prior art, the present invention proposes a breast ultrasound image segmentation method based on deep learning. In the actual diagnosis site, this breast ultrasound image segmentation method based on deep learning can identify more lesion areas while generating fewer false positives and missed detections; after data augmentation, the training data increases significantly, the semantic information of the images is richer, the number of effective features learned by the model during training increases, and at the same time, the oscillation situation generated during network training is reduced, effectively improving the effect of the model.

[0005] The technical solution of the present invention is realized as follows: A breast ultrasound image segmentation method based on deep learning includes the following steps;

[0006] Step 1: Use the breast cancer ultrasound dataset widely used in previous studies, and perform data augmentation on the images in the dataset by methods such as coordinate transformation and gray mapping to extract features such as color, texture, and granularity;

[0007] Step 2: Medical experts manually annotate the tumor sites in the breast ultrasound image dataset. Since breast image annotation consumes a large amount of labor cost and time cost, in order to make more effective use of the existing data, we augmented the training dataset. The augmentation methods include: random rotation, image scaling, mirror transformation along the vertical or horizontal direction, etc. Then, the tumor area of each image is annotated with a polygon box, and the breast ultrasound images are saved in json format. Finally, the json file is converted into a mask image, using a selected image, graphic or object to occlude all or part of the image to be processed to control the area or process of image processing, where the selected image or object is called a mask;

[0008] Step 3: Establish a breast lesion area segmentation model based on deep learning, MobileNetV3-UPerNet combining the backbone network and the task network, and use MobileNetV3 as the backbone network to extract the features of the breast ultrasound image lesion area;

[0009] Step 4: Use the dataset annotated in Step 2 to train the breast lesion area segmentation model based on deep learning in Step 3; divide the dataset established in Step 2 into a training set, a validation set, and a test set according to the ratio of 8:1:1. During the training process, use the training set and the validation set to train the breast ultrasound image segmentation model, use the test set to obtain the accuracy of the segmentation model, and select the optimal breast ultrasound image segmentation model according to the accuracy;

[0010] Step 5: Segment the lesion area of the breast ultrasound image using a segmentation model; at the diagnosis site, on-site staff collect breast ultrasound images, input the images into the trained breast ultrasound image segmentation model to segment and identify the tumor area, obtain the lesion area and calculate the proportion of the lesion area in the entire breast image, and display the final analysis result.

[0011] A further improvement lies in: in the said Step 1, obtain a breast cancer ultrasound data set, wherein, the breast cancer ultrasound data set includes a labeled first breast ultrasound image set and an unlabeled second breast ultrasound image set.

[0012] A further improvement lies in: in the said Step 1, obtain the first breast ultrasound image set carried by the ultrasound data set, and perform processing such as randomly rotating, scaling the images, and mirror transformation along the vertical or horizontal direction on each breast ultrasound image in the first breast ultrasound image set.

[0013] A further improvement lies in: in the said Step 2, input the breast ultrasound image to be segmented into the neural network model, and label the breast ultrasound image through the neural network model to obtain an ultrasound image with lesion markings, specifically including inputting the breast ultrasound image to be segmented into the neural network model to obtain a mask image.

[0014] A further improvement lies in: in the said Step 3, the backbone network and the task network include an encoder and a decoder, and the encoder includes a number of Dense basic units.

[0015] A further improvement lies in: in the network in the said Step 3, a computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors.

[0016] A further improvement lies in: in the said Step 4, input the breast ultrasound image to be segmented into the neural network model, label the breast ultrasound image through the neural network model to obtain an ultrasound image with lesion markings, and input the second breast ultrasound image set into the initial neural network to obtain an ultrasound segmentation model corresponding to each breast ultrasound image in the second breast ultrasound image set.

[0017] A further improvement lies in: the inputting the first breast ultrasound image set and the second breast ultrasound image set into the initial neural network, and updating the network parameters through the backpropagation algorithm to obtain the neural network, specifically including inputting the first breast ultrasound image set and the second breast ultrasound image set into the initial neural network.

[0018] A further improvement lies in: in the fifth step, in the foreground prior image and background prior image feature aggregation guidance module, the specific process of extracting foreground features through the background prior image feature guidance network is as follows: A1. The feature aggregation guidance module first receives the foreground feature map and background feature map from the corresponding convolutional units of the foreground feature and background feature extraction network branches. Each received feature map is strengthened through a convolutional operation with a 1x1 convolutional kernel, and the channel connection of the foreground feature map and background feature map is performed.

[0019] A further improvement lies in: in the fifth step, after completing 2x2 upsampling of the output map of the previous module, a 3x3 convolution with a dilation rate of 2 is used, and a pixel-level summation operation is performed with the strengthened foreground feature map and background feature map.

[0020] Compared with the prior art, the present invention has the following advantages: in the actual diagnosis site, the invention can identify more lesion areas while having fewer false positives and missed detections; after data augmentation, the training data increases significantly, the semantic information of the images is richer, the number of effective features learned by the model during training increases, and at the same time, the oscillation generated during network training is reduced, effectively improving the effect of the model. In addition, the improvement of the network model makes the network more lightweight and the segmentation speed faster, meeting the requirements of real-time segmentation. Compared with traditional machine learning algorithms, the method of the present invention does not require a series of complex image preprocessing and manual feature extraction processes, can reduce human labor, effectively avoid the limitations of manual feature extraction, and at the same time has higher accuracy compared with traditional machine learning methods, providing new development ideas for medical image processing technology and providing intelligent means for medical experts to provide auxiliary diagnosis references, with great practical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1 It is a flowchart of the method of the present invention;

[0023] Figure 2 It is a structural diagram of the MobileNetV3-UPerNet segmentation model algorithm design module of the present invention;

[0024] Figure 3 It is a structural diagram of the improved DDR module of MobileNetV3 of the present invention;

[0025] Figure 4 This is the structural diagram of the UPerNet segmentation network of the present invention. Specific Embodiments

[0026] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0027] In the description of the present invention, it should be noted that unless otherwise clearly defined and limited, the terms "installation", "connection", and "connection" should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0028] Embodiment 1

[0029] See Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 The embodiments of the present invention disclose a breast ultrasound image segmentation method based on deep learning, including the following steps;

[0030] Step 1: Use the breast cancer ultrasound dataset widely used in previous studies, and perform data enhancement on the images in the dataset through methods such as coordinate transformation and gray mapping to extract features such as color, texture, and granularity;

[0031] Step 2: Medical experts manually annotate the tumor areas in the breast ultrasound image dataset. Since breast image annotation consumes a large amount of labor costs and time costs, in order to make more effective use of the existing data, we enhanced the training dataset. The enhancement methods include: random rotation, image scaling, mirror transformation along the vertical or horizontal direction, etc. Then, the tumor areas of each image are annotated with a polygon box, and the breast ultrasound images are saved in json format. Finally, the json file is converted into a mask image. Use a selected image, graphic, or object to block all or part of the image to be processed to control the area or process of image processing, where the selected image or object is called a mask;

[0032] Step 3: Establish a breast lesion area segmentation model based on deep learning, MobileNetV3-UPerNet which combines a backbone network and a task network, and use MobileNetV3 as the backbone network to extract the features of the lesion area in breast ultrasound images;

[0033] Step 4: Use the dataset labeled in Step 2 to train the breast lesion area segmentation model based on deep learning in Step 3; divide the dataset established in Step 2 into a training set, a validation set and a test set according to the ratio of 8:1:1. During the training process, use the training set and the validation set to train the breast ultrasound image segmentation model, use the test set to obtain the accuracy of the segmentation model, and select the optimal breast ultrasound image segmentation model according to the accuracy;

[0034] Step 5: Use the segmentation model to segment the lesion area in breast ultrasound images; at the diagnosis site, the on-site staff collect breast ultrasound images, input the images into the trained breast ultrasound image segmentation model to segment and identify the tumor area, obtain the lesion area and calculate the proportion of the lesion area in the whole breast image, and display the final analysis result;

[0035] In Step 1, obtain a breast cancer ultrasound dataset. The breast cancer ultrasound dataset includes a labeled first breast ultrasound image set and an unlabeled second breast ultrasound image set. Obtain the network parameters of the initial neural network according to the labeled breast ultrasound image set, and use the EM algorithm to train the network parameters to obtain the neural network.

[0036] In Step 1, obtain the first breast ultrasound image set carried by the ultrasound dataset, standardize each breast ultrasound image in the first breast ultrasound image set, and train the initial neural network through the standardized first breast ultrasound image set to obtain the network parameters of the initial neural network.

[0037] In Step 2, input the breast ultrasound image to be segmented into the neural network model, and label the breast ultrasound image through the neural network model to obtain an ultrasound image with lesion marks. Specifically, input the breast ultrasound image to be segmented into the neural network model to obtain a mask image, and add the mask image to the segmented breast ultrasound image to obtain a labeled ultrasound image.

[0038] In Step 3, the backbone network and the task network include an encoder and a decoder. The encoder includes several Dense basic units, the decoder includes upsampling and several Dense basic units, and the corresponding layers of the encoder and the decoder are connected.

[0039] In Step 3, a computer-readable storage medium in the network stores one or more programs, and the one or more programs can be executed by one or more processors.

[0040] In step four, the breast ultrasound image to be segmented is input into the neural network model. The neural network model adds labels to the breast ultrasound image to obtain an ultrasound image with lesion markings. The second breast ultrasound image set is input into the initial neural network to obtain the ultrasound segmentation model corresponding to each breast ultrasound image in the second breast ultrasound image set. The ultrasound segmentation map is used as the label for its corresponding breast ultrasound image to obtain the third breast ultrasound image set. The first breast ultrasound image set and the third breast ultrasound image set are input into the initial neural network, and the network parameters are updated through the backpropagation algorithm to obtain the neural network.

[0041] Inputting the first breast ultrasound image set and the third breast ultrasound image set into the initial neural network and updating the network parameters through the backpropagation algorithm to obtain the neural network specifically includes: inputting the first breast ultrasound image set and the third breast ultrasound image set into the initial neural network, updating the network parameters through the backpropagation algorithm, and determining whether the number of updates of the network parameters reaches the preset number. When the number of updates reaches the preset number, the neural network is determined according to the network parameters. When the number of updates does not reach the preset number, the operation of inputting the second breast ultrasound image set into the initial neural network is repeated.

[0042] In step five, in the foreground prior image and background prior image feature aggregation guidance module, the specific process of extracting the foreground features through the background prior image feature guidance network is as follows: The feature aggregation guidance module first receives the foreground feature map and the background feature map from the corresponding convolutional units of the foreground feature and background feature extraction network branches. Each received feature map is strengthened through a convolutional operation with a 1x1 convolutional kernel, and the channel connection of the foreground feature map and the background feature map is performed.

[0043] In step five, after performing 2x2 upsampling on the output map of the previous module, a 3x3 convolution with a dilation rate of 2 is used, and a pixel-level summation operation is performed with the strengthened foreground feature map and background feature map. The obtained foreground feature map and background feature map are respectively input into different branches to continue to complete the strengthening of the foreground feature and background feature. After the obtained background feature map undergoes three 1x1 and one 3x3 convolution operations, one of the data is output as the background feature and used as the background feature input sample for the next feature aggregation guidance module.

[0044] Embodiment 2

[0045] The image range obtained by dividing the present invention is more accurate, the edge is clearer, and the texture information is richer. Compared with the traditional method of using a single U-net to segment images, the image area obtained by the present invention is more coherent, and less image data is lost. The present invention connects two U-net networks through a feature aggregation guidance module, and has a stronger ability to extract image features than a single U-net network. After obtaining an image containing prior information of breast tumors through preprocessing, the U-Net network framework is used to extract features of the lesion area for accurate segmentation. This method uses the guidance of foreground and background prior information to improve the segmentation accuracy of breast ultrasound lesions.

[0046] When using this deep learning-based breast ultrasound image segmentation method, the breast ultrasound image to be segmented input into the neural network model is a standardized breast ultrasound image, that is, the ultrasound image to be segmented is standardized before being input into the neural network model, that is, the image to be segmented is scaled to a preset image size, and gray level de-mean is performed to achieve centering and gray level normalization. Among them, the preset image size is the image size after the initial training samples of the neural network model. In addition, when the neural network model recognizes the breast ultrasound image to be segmented, the breast ultrasound image to be segmented first passes through the encoder and then through the decoder to obtain and output a mask image with lesion segmentation marks. The mask image is added to the original breast ultrasound image to obtain the final breast ultrasound image with marked lesions. In order to improve the fineness of segmentation, after passing through the decoder, the image obtained by the decoder is refined by a conditional random field to obtain a mask image with lesion segmentation marks. Of course, in practical applications, the conditional random field constraint is replaced by a Markov random field constraint. Based on the above-mentioned semi-supervised learning-based breast ultrasound image lesion segmentation method, the present invention provides a computer-readable storage medium. The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors. A terminal device is provided, which includes at least one processor, a display screen, and a memory, and may further include a communication interface. Among them, the processor, the display screen, the memory, and the communication interface can complete communication with each other through a bus. The display screen is set to display a preset user guidance interface in the initial setting mode. The communication interface can transmit information. The processor can call logical instructions in the memory. The storage can include a storage program area and a storage data area. Among them, the storage program area can store an operating system and application programs required for at least one function, and the storage data area can store data created according to the use of the terminal device, etc.

[0047] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A breast ultrasound image segmentation method based on deep learning, characterized in that, It includes the following steps; Step 1: Use the breast cancer ultrasound dataset widely used in previous studies, and perform data augmentation on the images in the dataset through methods such as coordinate transformation and grayscale mapping to extract features such as color, texture, and granularity; Step 2: Medical experts manually annotate the tumor areas in the breast ultrasound image dataset. Since breast image annotation consumes a large amount of labor and time costs, in order to make more effective use of the existing data, we enhanced the training dataset. The enhancement methods include: random rotation, image scaling, mirror transformation along the vertical or horizontal direction, etc. Then, the tumor areas of each image are annotated with polygon boxes, and the breast ultrasound images are saved in json format. Finally, the json file is converted into a mask image, and a selected image, graphic, or object is used to occlude all or part of the image to be processed to control the area or process of image processing, where the selected image or object is called a mask; Step 3: Establish a deep learning-based breast lesion area segmentation model, MobileNetV3-UPerNet that combines a backbone network and a task network, and use MobileNetV3 as the backbone network to extract the features of the breast ultrasound image lesion area; Step 4: Use the dataset labeled in Step 2 to train the deep learning-based breast lesion area segmentation model in Step 3; Divide the dataset established in Step 2 into a training set, a validation set, and a test set according to the ratio of 8:1:

1. During the training process, use the training set and the validation set to train the breast ultrasound image segmentation model, use the test set to obtain the accuracy of the segmentation model, and select the optimal breast ultrasound image segmentation model according to the accuracy; Step 5: Use the segmentation model to segment the breast ultrasound image lesion area; At the diagnosis site, on-site staff collect breast ultrasound images, input the images into the trained breast ultrasound image segmentation model to segment and identify the tumor area, obtain the lesion area and calculate the proportion of the lesion area in the entire breast image, and display the final analysis result.

2. The method for segmenting breast ultrasound images based on deep learning according to claim 1, wherein: In Step 1, a breast cancer ultrasound dataset is obtained, where the breast cancer ultrasound dataset includes a labeled first breast ultrasound image set and an unlabeled second breast ultrasound image set.

3. A method for segmenting breast ultrasound images based on deep learning according to claim 1, characterized in that: In Step 1, the first breast ultrasound image set carried by the ultrasound dataset is obtained, and each breast ultrasound image in the first breast ultrasound image set is randomly rotated, scaled, and mirror-transformed along the vertical or horizontal direction.

4. A method for segmenting breast ultrasound images based on deep learning according to claim 1, characterized in that: In Step 2, input the breast ultrasound image to be segmented into the neural network model, and label the breast ultrasound image through the neural network model to obtain an ultrasound image with lesion markings. Specifically, input the breast ultrasound image to be segmented into the neural network model to obtain a mask image.

5. A method for breast ultrasound image segmentation based on deep learning according to claim 1, characterized in that: In Step 3, the backbone network and the task network include an encoder and a decoder, and the encoder includes several Dense basic units.

6. A method for breast ultrasound image segmentation based on deep learning according to claim 1, characterized in that: In Step 3, one or more programs are stored in a network computer-readable storage medium, and the one or more programs can be executed by one or more processors.

7. A method for segmenting breast ultrasound images based on deep learning according to claim 1, characterized in that: In the fourth step, the breast ultrasound image to be segmented is input into the neural network model. The neural network model adds labels to the breast ultrasound image to obtain an ultrasound image with lesion markings. The second breast ultrasound image set is input into the initial neural network to obtain an ultrasound segmentation model corresponding to each breast ultrasound image in the second breast ultrasound image set.

8. A method for segmenting breast ultrasound images based on deep learning according to claim 1, characterized in that: Inputting the first breast ultrasound image set and the second breast ultrasound image set into the initial neural network and updating the network parameters through the backpropagation algorithm to obtain the neural network specifically includes inputting the first breast ultrasound image set and the second breast ultrasound image set into the initial neural network.

9. A method for segmenting breast ultrasound images based on deep learning according to claim 1, characterized in that: In the fifth step, in the foreground prior image and background prior image feature aggregation guidance module, the specific process of extracting foreground features through the background prior image feature guidance network is as follows. The feature aggregation guidance module first receives the foreground feature map and the background feature map from the corresponding convolutional units of the foreground feature and background feature extraction network branches. Each received feature map is enhanced through a convolutional operation with a 1x1 convolutional kernel, and the channel connection of the foreground feature map and the background feature map is performed.

10. A method for breast ultrasound image segmentation based on deep learning according to claim 1, characterized in that: In the fifth step, after performing 2x2 upsampling on the output map of the previous module, a 3x3 convolution with a dilation rate of 2 is used, and a pixel-level summation operation is performed with the enhanced foreground feature map and background feature map.

Images