A multi-task breast cancer ultrasound detection method based on transfer learning

Abstract

The present invention provides a multi-task breast cancer ultrasound detection method based on transfer learning. The breast cancer tumor localization and classification tasks are integrated into a unified framework. In order to improve the technical support for auxiliary diagnosis, collaborative learning is used to migrate the hardness information in the elastic imaging ultrasound (EUS) modality to the B-mode ultrasound (BUS) channel network; and on this basis, a dual-channel ResNet-GAP is proposed, one for the BUS modality and the other for the EUS modality. In each channel, a convolution kernel is used to generate a class activation map (CAM), and the consistency of the CAM between the two channels is used to repeatedly iterate the training. Based on the trained network model, the benign and malignant nature of breast cancer tumors is predicted. The method of the present invention realizes end-to-end positioning detection of breast cancer tumors, maximizes the use of elastic ultrasound information, and improves the doctor's work efficiency and diagnostic accuracy.

Description

Technical Field

[0001] The present invention relates to the field of ultrasound image diagnosis auxiliary technology and enhancement technology, and in particular to a multi-task breast cancer ultrasound auxiliary diagnosis method based on transfer learning. Background Art

[0002] Since the 1990s, the incidence of breast cancer has continued to rise and has become one of the most common cancers among women worldwide. B-mode ultrasound (BUS) imaging has become one of the important means of clinical examination and diagnosis due to its advantages such as low cost, non-invasiveness and safety. However, due to the low-quality imaging caused by noise and artifacts, the reading of BUS is highly dependent on experienced and well-trained radiologists. Even the most experienced radiologists still have a high misdiagnosis rate in breast cancer diagnosis based on BUS images. In order to overcome these problems, the development of advanced breast ultrasound computer-aided diagnosis methods can make ultrasound diagnosis more intelligent and accurate.

[0003] In recent years, convolutional neural networks (CNNs) have been successfully applied to breast cancer computer-aided diagnosis (CAD) models. Compared with traditional CAD methods that rely heavily on manually extracted features, CNNs automatically learn discriminative feature representations in a data-driven manner. So far, many CNN-based methods have emerged to classify breast cancer in BUS images.

[0004] CNN-based breast cancer CAD methods mainly include three steps: image preprocessing, localization, and classification. Image preprocessing removes redundant areas in the acquired ultrasound images, the localization step identifies the tumor location, and the classification step classifies the tumor into benign and malignant. In most literature, breast cancer CAD methods treat localization and classification as two independent steps, making the classification performance of breast cancer greatly affected by the above-mentioned localization results. In addition, these methods either perform classification tasks or localization tasks, and few methods perform both at the same time.

[0005] BUS imaging is one of the important bases for breast cancer diagnosis. BUS images contain important features of breast lesions, including shape, direction, and edge features, but due to the overlap and diversity of benign and malignant tumors in BUS images, there are limitations in the classification of benign and malignant tumors using a single BUS. According to previous studies, the hardness of malignant tumor tissue is generally higher than that of benign tumor tissue, which provides a potential basis for distinguishing benign and malignant breast tumors. Elastic ultrasound (EUS) imaging provides information about the vascular and tissue stiffness of the tumor. Compared with BUS, it has higher specificity and is more helpful in distinguishing benign and malignant tumors. However, due to the limitations of imaging equipment, EUS is not popular in clinical examinations. Therefore, migrating stiffness information from EUS to BUS is of great significance for developing BUS-based CAD models. Summary of the invention

[0006] In order to overcome the above-mentioned technical defects, the purpose of the present invention is to provide a multi-task breast cancer ultrasound detection method based on transfer learning, which uses computer technology and deep learning methods to automatically locate and detect lesions in type B breast ultrasound images. The present invention is not a direct disease treatment and diagnosis method, but through indirect technical assistance, it is conducive to technology-assisted diagnosis. By providing medical images, it improves the effective guarantee and optimization of diagnostic accuracy by auxiliary technical means, thereby improving the efficiency and quality of diagnostic work.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A multi-task breast cancer ultrasound detection method based on transfer learning, comprising the following steps:

[0009] 1) Collecting B-type breast ultrasound image samples and preprocessing the ultrasound image samples;

[0010] 2) Train a regression network based on the labeled data of type B breast ultrasound image samples to automatically crop the images;

[0011] 3) Mapping the cropped B-type breast ultrasound image coordinates to the elastic ultrasound modality and cropping them to obtain ROI images of the two modalities;

[0012] 4) The cropped B-type breast ultrasound image and elastic ultrasound image are sent to the dual-channel network to calculate the loss function, and the best network model is obtained through repeated iterative training for output;

[0013] 5) Based on the trained network model, the type of breast tumor is predicted and a tumor location heat map is obtained.

[0014] The step 1) preprocessing includes image sample annotation and image sample quality improvement.

[0015] In the step 2), the B-type breast ultrasound image label is a manually marked rectangular frame, and the rectangular frame contains the breast tumor lesion.

[0016] The step 3) of extracting the tumor region and preprocessing process includes:

[0017] (1) Using a regression network to locate the tumor position in the B-type breast ultrasound image and obtain the tumor position coordinates;

[0018] (2) The tumor position coordinates are transferred to the elastic ultrasound image and cropped to obtain the ROI images of B-type breast ultrasound and elastic ultrasound.

[0019] The feature extraction network in step 4) is a dual-channel ResNet18 network; the cropped B-type breast ultrasound and elastic ultrasound ROI images are sent to the dual-channel ResNet18 network and the hardness information in the elastic ultrasound is migrated to the B-type breast ultrasound channel using a consistency learning strategy.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The method of the present invention utilizes a consistency learning strategy to transfer the stiffness information in the elastic ultrasound channel to the B-type ultrasound channel. Compared with the traditional B-type ultrasound image diagnosis model, it contains more information and combines computer technology and deep learning methods to achieve end-to-end positioning detection of breast cancer tumors, which is beneficial to technology-assisted diagnosis, maximizes the use of elastic ultrasound information, and improves the work efficiency of doctors. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] Figure 1 It is a schematic diagram of the process method of the present invention;

[0024] Figure 2 This is the result localization map of breast tumors according to the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The high noise of BUS images often reduces the performance of subsequent breast cancer localization and classification. To solve this problem, the present invention uses a CNN regression network to automatically identify the area in the BUS image that contains the entire tumor. At the same time, the roughly identified regional coordinates are mapped to the corresponding EUS image. The design of this operation is inspired by observing the doctor's diagnosis process, and through enhanced technical means, the accuracy of diagnosis is indirectly improved. We noticed that doctors often find a rough location for the tumor target during the diagnosis process, and then make a final decision by carefully comparing the image details. The output of the regression network is the center coordinates (x, y) of the detected area, as well as the width w and height h of the area.

[0027] The CNN regression network is trained using a set of original BUS images that are manually labeled by radiologists. To facilitate the calculation of the loss function, the original image is normalized to its center (0,0) with a height and width range of -1 to 1. The loss function of the regression network is defined as: Among them, o1, o2, o3, and o4 are the coordinate values ​​predicted by the network, (x i ,y i ) is the normalized center coordinate, (w i ,h i ) are the width and height of the normalized box respectively. By minimizing the loss function, the parameters of the CNN regression network are optimized.

[0028] The present invention is a multi-task breast cancer ultrasound detection method based on transfer learning, comprising the following steps:

[0029] Collecting breast ultrasound image samples and preprocessing the ultrasound image samples; the preprocessing includes image sample annotation and image sample quality improvement;

[0030] The B-type breast ultrasound image label is a manually marked rectangular frame, and the rectangular frame contains the breast tumor lesion.

[0031] The image is automatically cropped based on the regression network trained based on the labeled data of B-type ultrasound image samples;

[0032] The cropped B-mode ultrasound image coordinates are mapped to the elastic ultrasound mode and cropped to obtain ROI images of the two modes;

[0033] (1) Using a regression network to locate the tumor position in the B-type breast ultrasound image and obtain the tumor position coordinates;

[0034] (2) The tumor position coordinates are transferred to the elastic ultrasound image and cropped to obtain the ROI images of B-mode ultrasound and elastic ultrasound;

[0035] The cropped B-mode ultrasound and elastic ultrasound ROI images are sent to the dual-channel ResNet18 network and the hardness information in the elastic ultrasound is transferred to the B-mode ultrasound channel using the consistency learning strategy;

[0036] Based on the trained network model, the type of breast tumor is predicted and a heat map of the tumor location is obtained.

[0037] The present invention uses the classic ResNet as the backbone network of the diagnostic network. Since the fully connected layer in ResNet is prone to overfitting, a convolutional layer is used instead of a fully connected layer to output 512 feature maps of size 7×7. Specifically, the present invention uses a single convolution kernel (size 1×1×512) to convolve the 512 feature maps to generate a feature map It is considered as a CAM image representing the probability of tumor. The pixel value of the spatial position (x, y) in the k-th feature map generated by the last convolutional layer is recorded as f k (x,y),k=1,…,512. The value of position (x,y) is: w k is the weight of the k-th feature map, indicating that the k-th feature map f k The importance of (x,y) in generating CAM. The size of each pixel value in CAM indicates the possibility that it belongs to a malignant tumor. GAP operation is performed on CAM to obtain the final output score of ResNet-GAP, and it is normalized using the sigmoid function σ(x) = 1 / (1+exp(-x)). The classification loss function designed by cross entropy is as follows: cls (p,y)=-[ylog(p)+(1-y)log(1-p)], where y∈{0,1} is the image-level label and p is the probability that the output belongs to the malignant class.

[0038] The EUS modality provides stiffness information of tumor tissue and is an important supplement to the BUS modality in auxiliary diagnosis of breast cancer. A pair of BUS and EUS images collected from the same area of ​​the human body should have the same judgment on the lesion information of the tumor. Although the two images are collected in different ways, they have the same semantic information. Therefore, it is necessary to develop a collaborative learning mechanism to improve the performance of the BUS diagnostic model by utilizing the stiffness information in the EUS modality.

[0039] To this end, the present invention designs a dual-channel ResNet-GAP for collaborative learning. The network has two channels, one is the BUS channel and the other is the EUS channel. The two channels are collaboratively optimized during the training process. If one of the channels cannot learn the feature representation of the input area, it can guide the training by using the modal information of the other channel. This operation is repeated continuously during the training process to achieve the effect of collaborative training. Since the EUS and BUS images are spatially aligned and the tumor appears in the same position in these two images, collaborative learning can achieve quite ideal results. We expect the two channels to generate similar CAMs so that the two channels have a similar understanding of the input image. Minimize the inconsistency loss function for: like Figure 1 As shown in Figure 1, a pair of cropped BUS and EUS images are fed into the dual-channel ResNet-GAP. The inconsistency loss function is introduced into the error back propagation process, allowing the network to automatically capture the consistency information between different modalities.

[0040] The present invention defines three different loss functions to calculate the back propagation error, which are the classification loss functions for each modality, namely, cls (p BUS ,y) and l cls (p EUS ,y), and the regularized consistency loss function Finally, the weighted sum of the three loss functions of the dual-channel ResNet-GAP is:

[0041] After the dual-channel ResNet-GAP network converges, the parameters in the BUS channel are migrated to ResNet-GAP for auxiliary technical processes in breast cancer diagnosis, such as Figure 2 As shown. In the test phase, the tested BUS image is input into the ResNet-GAP model to obtain the prediction result. The last convolutional layer of ResNet-GAP outputs the CAM, which is further adjusted to the same size as the test image through bilinear interpolation. Pixels in the CAM whose local maximum value is greater than the adjacent response value are regarded as suspected tumor pixels. Finally, the threshold is calculated adaptively through the OTSU operation, and pixels with pixel values ​​less than the threshold are regarded as outliers and excluded.

[0042] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-task breast cancer ultrasound detection method based on transfer learning, characterized in that: The following steps are involved: 1) Collecting B-type breast ultrasound image samples and preprocessing the ultrasound image samples; 2) Train a regression network based on the labeled data of type B breast ultrasound image samples to automatically crop the images; 3) Mapping the cropped B-type breast ultrasound image coordinates to the elastic ultrasound modality and cropping them to obtain ROI images of the two modalities; 4) The cropped B-type breast ultrasound image and elastic ultrasound image are sent to the dual-channel network to calculate the loss function, and the best network model is obtained through repeated iterative training for output; 5) Based on the trained network model, the type of breast tumor is predicted and the tumor location heat map is obtained to complete breast cancer ultrasound detection.

2. The multi-task breast cancer ultrasound detection method based on transfer learning according to claim 1 is characterized in that: The step 1) preprocessing includes image sample annotation and image sample quality improvement.

3. The multi-task breast cancer ultrasound detection method based on transfer learning according to claim 1 is characterized in that: In the step 2), the B-type breast ultrasound image label is a manually marked rectangular frame, and the rectangular frame contains the breast tumor lesion.

4. The multi-task breast cancer ultrasound detection method based on transfer learning according to claim 1 is characterized in that: The step 3) of extracting the tumor region and preprocessing process includes: (1) Using a regression network to locate the tumor position in the B-type breast ultrasound image and obtain the tumor position coordinates; (2) The tumor position coordinates are transferred to the elastic ultrasound image and cropped to obtain the ROI images of B-type breast ultrasound and elastic ultrasound.

5. The multi-task breast cancer ultrasound detection method based on transfer learning according to claim 1, characterized in that: The feature extraction network in step 4) is a dual-channel ResNet18 network; the cropped B-type breast ultrasound and elastic ultrasound ROI images are sent to the dual-channel ResNet18 network and the hardness information in the elastic ultrasound is migrated to the B-type breast ultrasound channel using a consistency learning strategy.

Images