Method and system for breast ultrasonic image diagnosis using weakly-supervised deep learning artificial intelligence

Abstract

The method for diagnosing a breast ultrasound using weakly supervised deep learning artificial intelligence includes an ultrasound image preprocessing step of deleting a patient's personal information from a breast ultrasound image and generating input data including only an image region required for learning; a deep learning step of receiving the input data and acquiring a feature map from the received input data using a convolutional neural network (CNN) and global average pooling (GAP) to perform re-learning; a differential diagnosis step of discriminating the input data as normal, benign, or malignant using global average pooling (GAP) and calculating a probability of malignancy (POM) representing the accuracy of the discrimination if the input data is discriminated as malignant; and a contribution region determination and visualization step of backpropagating the discrimination result through the convolutional neural network (CNN), calculating the contribution of each pixel that contributed to the discrimination result from gradients and features, and visualizing the contribution region that contributed to the discrimination together with the probability of malignancy (POM) based on the calculated contribution of each pixel, and the deep learning step is learned based on the verified performance of the contribution region and the probability of malignancy (POM). The present invention improves work efficiency when compared to fully supervised algorithms, since it is possible to omit the additional annotation work of displaying lesions, and improves diagnostic accuracy by providing visualization of regions of interest together with the probability of malignancy.

Description

[Technical Field]

[0001] The present invention relates to artificial intelligence technology for diagnosing breast cancer using breast ultrasound images, and in particular to a method and system for developing a weakly supervised deep learning algorithm to diagnose breast cancer based on ultrasound images without image annotation for regions of interest, and for locating and visualizing regions that contribute to the diagnosis. [Background technology]

[0002] Breast ultrasound is the primary examination device used to differentiate between benign and malignant breast masses, and is used for both diagnostic and screening purposes. Evidence is emerging that ultrasound can detect occult cancer, which is difficult to detect with mammography, especially in women with dense breasts, and this has garnered increasing attention. Breast cancer is the most common cancer in women, and early diagnosis is an important step in reducing its mortality rate. Due to its convenience, stability, and high accuracy, ultrasound-based breast cancer diagnosis is widely used to diagnose and observe breast lesions.

[0003] The principle of ultrasound diagnosis is as follows: First, sound waves emitted from the probe of an ultrasound examination device enter the mammary gland and are absorbed or reflected by various tissue structures. The reflected waves return to the probe and are processed as an image by the ultrasound examination device. In general, distinguishing between benign and malignant tumors and segmenting the region of interest from ultrasound images is more difficult than using natural images because: 1) ultrasound images have a low contrast ratio and a lot of speckle noise; 2) different tumors vary greatly in shape and appearance, especially between benign and malignant tumors; 3) there are ultrasound artifacts and normal tissues that resemble tumors; and 4) the tumor boundary is not clearly defined.

[0004] Korean Patent Application Publication No. 1020140109320 (Publication Date: September 15, 2014, Title: "Ultrasound Image Processing Method and Apparatus, and Breast Cancer Diagnosis Apparatus") discloses a breast cancer diagnosis apparatus including the steps of: performing over-segmentation on a received ultrasound image to obtain multi-layered segments; extracting features of each segment from the obtained multi-layered segments; performing target object inspection on the received ultrasound image to generate an overcomplete sliding window as an inspection window, and transferring the image features to the obtained multi-layered segments through the inspection window; integrating the extracted features of each segment with the features transferred to the segment through the inspection window to train a segment classifier; receiving the output of the segment classifier and providing the integrated voltage to a paired CRF model to obtain the target object of the segment.

[0005] However, even with this technology, there can be differences in diagnostic ability among experts when it comes to breast ultrasound, and in particular, there is a significant disparity in diagnostic ability between non-experts and experts, resulting in many false positive and false negative results, which can lead to unnecessary tissue tests and surgeries, not only causing great harm to the patient's body but also increasing medical costs, becoming a major social problem.

[0006] To overcome these challenges, various studies are being conducted to assist in the diagnosis of breast cancer based on ultrasound images, and efforts to adopt deep learning algorithms, which are the artificial intelligence technology that has recently attracted the most attention in the field of medical imaging, are gradually spreading.

[0007] Previous research into deep learning algorithms for diagnosing cancer using ultrasound images is based on fully supervised learning, in which tumor lesions are manually segmented as regions of interest (ROIs), and the segmented ROIs are used as input to distinguish benign from malignant lesions. However, this generally requires an annotation process in which an expert manually draws the lesion regions of interest (ROIs) within the image.

[0008] Furthermore, even when using an automatic ROI segmentation method instead of manual annotation, an expert must still verify the ROI. The image annotation process for such regions of interest has the following challenges:

[0009] First, the image annotation process for regions of interest requires a lot of time and labor, which limits the development of robust deep learning models (with high diagnostic performance and applicable to other data) based on huge amounts of training data (generally, thousands or tens of thousands of images are required for training).

[0010] Second, the manual image annotation process for regions of interest is performed based on subjective prior assessment of lesion characteristics (borderline of possible benign / malignant, pattern), which may introduce bias into deep learning algorithms that determine parameters based on data, negatively impacting model performance.

[0011] Third, models based on image annotations for regions of interest require similarly annotated images as input data for clinical application. Therefore, to obtain the prediction values ​​of deep learning models for images, users must specify the region of interest and perform ROI segmentation each time, which is time-consuming and difficult to use clinically.

[0012] Fourth, when differentially diagnosing lesions, information on the area surrounding the lesion (e.g., changes in surrounding tissues and echograms) rather than the lesion itself is often useful. However, conventional auxiliary diagnosis methods based on fully supervised learning only use the lesion for learning, which limits their ability to comprehensively learn image information.

[0013] Fifth, conventional diagnostic methods are limited to the task of differentially diagnosing breast masses, and therefore cannot utilize normal images as input data.

[0014] This is a major obstacle to clinical use, since the majority of breast ultrasound images are actually images of normal (lesion-free) breasts.

[0015] Therefore, there is a strong demand for an artificial intelligence learning method and system based on weakly-supervised learning that can develop deep learning algorithms without the need for image annotation processes for regions of interest (ROIs) for breast cancer diagnosis using ultrasound images. [Prior art documents] [Patent documents]

[0016] [Patent Document 1] Korean Patent Application Publication No. 1020140109320 (Publication Date: September 15, 2014, Title of Invention: "Ultrasound Image Processing Method and Apparatus, and Breast Cancer Diagnosis Apparatus") Summary of the Invention [Problem to be solved by the invention]

[0017] The objective of the present invention is to develop an efficient deep learning algorithm that utilizes an artificial intelligence learning method based on weakly supervised learning and does not require the process of image annotation for regions of interest (ROI), and to provide a system that can perform differential diagnosis to distinguish between normal, benign, and malignant breast images from breast ultrasound images and diagnose breast cancer by visualizing the areas that contribute to the diagnosis. [Means for solving the problem]

[0018] To achieve the above-mentioned objectives, one aspect of the present invention relates to a method for diagnosing breast cancer by developing a deep learning algorithm based on weakly supervised learning, performing differential diagnosis to distinguish between normal, benign, and malignant breast images based on breast ultrasound images that have not been annotated with image annotations for regions of interest (ROIs), and visualizing the areas that contributed to the diagnosis.

[0019] The breast ultrasound diagnosis method using weakly supervised deep learning (DL) artificial intelligence includes an ultrasound image preprocessing step of removing patient personal information from breast ultrasound images and generating input data including only image regions required for learning; a deep learning step of receiving the input data and acquiring a feature map from the received input data using a convolutional neural network (CNN) and global average pooling (GAP) to re-learn; a differential diagnosis step of classifying the input data as normal, benign, or malignant using GAP, and if the input data is determined to be malignant, calculating a probability of malignancy (POM) that indicates the accuracy of the classification; and a differential diagnosis step of backpropagating the classification result through the convolutional neural network (CNN), calculating the contribution of each pixel that contributed to the classification result from a gradient and feature, and then classifying the relevant contributing region that contributed to the classification based on the calculated contribution of each pixel. The method includes a contributing region determination and visualization step for visualizing a contributing region (a contributing region) together with the probability of malignancy (POM), and the deep learning step is characterized in that it is learned based on verified performance of the contributing region and the probability of malignancy (POM).

[0020] In particular, the ultrasound image pre-processing step is characterized by including: deleting information related to the patient's personal information from the DICOM (Digital Imaging and Communications in Medicine) header information using a dedicated de-identification module developed for ultrasound images; extracting an anonymized image by trimming the edges of the ultrasound image to remove remaining personal information and body marks and text tags that interfere with learning, or generating image data based on the ultrasound signal received from the ultrasound device; and resizing the anonymized image or image data to input data of the same size as the image.

[0021] The deep learning stage is characterized by including: applying a data augmentation process by adding at least one of geometric and optical variations to the anonymized image to reduce verification errors and create a reproducible model; configuring a deep learning algorithm architecture using at least one convolutional neural network (CNN) that performs image classification to learn whether the input data is normal / benign / malignant, and obtaining feature maps corresponding to each convolutional layer; and determining and re-learning parameters related to the obtained feature maps.

[0022] Furthermore, the differential diagnosis step is characterized by including: acquiring features of each feature map of the last convolutional layer using a global average pooling (GAP) layer added next to the convolutional layer in the architecture of the deep learning algorithm; calculating class scores by applying learnable weights to the acquired features; and calculating a probability of malignancy (POM) based on the calculated class scores.

[0023] The contributing region determination and visualization step includes: applying weights to the feature maps to obtain class activation maps (CAMs) for each class; scaling the obtained class activation maps (CAMs) by applying min-max normalization; determining the contributing regions by binarizing the scaled class activation maps (CAMs) by applying a threshold; and visualizing the class activation maps (CAMs) together with the probability of malignancy (POMs) as a heat map.

[0024] In particular, the geometric transformations include at least one of horizontal flipping, vertical flipping, rotation, scaling, and translation, and the optical transformations include at least one of deletion, speckle noise processing, sharpening, and Gaussian blurring.

[0025] To achieve the above-mentioned objectives, one aspect of the present invention relates to a system that develops a deep learning algorithm based on weakly supervised learning, performs differential diagnosis to distinguish between normal, benign, and malignant breast images based on breast ultrasound images that have not been annotated for regions of interest (ROIs), and can diagnose breast cancer by visualizing the areas that contribute to the diagnosis.

[0026] This breast ultrasound diagnostic system using weakly supervised deep learning (DL) artificial intelligence includes an ultrasound image preprocessing module that removes patient personal information from breast ultrasound images and generates input data containing only image regions required for learning; a deep learning module that receives the input data and acquires a feature map from the received input data using a convolutional neural network (CNN) and global average pooling (GAP) to perform re-learning; a differential diagnosis module that uses global average pooling (GAP) to classify the input data as normal, benign, or malignant, and if determined to be malignant, calculates a probability of malignancy (POM) that represents the accuracy of the classification; and a contribution region determination and visualization module that backpropagates the classification results through the convolutional neural network (CNN), calculates the contribution of each pixel that contributed to the classification result from gradients and features, and visualizes the contribution region that contributed to the classification together with the probability of malignancy (POM) based on the calculated contribution of each pixel, and the weakly supervised deep learning algorithm is characterized in that it is trained based on the verified performance of the contribution region and the probability of malignancy (POM).

[0027] In particular, the ultrasound image preprocessing module is configured to extract an anonymized image by using a dedicated de-identification module developed for ultrasound images to remove information related to the patient's personal information from the DICOM header information, trimming the edges of the ultrasound image to remove remaining personal information and body marks and text tags that interfere with learning, or to generate image data based on ultrasound signals received from an ultrasound device, and resize the anonymized image or image data to input data of the same size as the image.

[0028] In addition, the deep learning module is configured to apply a data augmentation process by adding at least one of geometric and optical variations to the anonymized image to reduce verification errors and create a reproducible model, configure a deep learning algorithm architecture using at least one convolutional neural network (CNN) that performs image classification to learn whether the input data is normal / benign / malignant, obtain feature maps corresponding to each convolutional layer, and determine and re-learn parameters related to the obtained feature maps.

[0029] Furthermore, the differential diagnosis module is configured to use a global average pooling (GAP) layer added after the convolutional layer in the architecture of the deep learning algorithm to acquire features of each feature map of the last convolutional layer, apply learnable weights to the acquired features to calculate class scores, and calculate a probability of malignancy (POM) based on the calculated class scores.

[0030] The contributing region determination and visualization module is configured to apply weights to the feature maps to obtain class activation maps (CAMs) for each class, scale the obtained class activation maps (CAMs) by applying min-max normalization, binarize the scaled class activation maps (CAMs) by applying a threshold value to determine the contributing regions, and visualize the class activation maps (CAMs) as heat maps together with the probability of malignancy (POM).

[0031] Furthermore, the geometric transformation includes at least one of horizontal flip, vertical flip, rotation, scaling, and translation, and the optical transformation includes at least one of deletion, speckle noise reduction, sharpening, and Gaussian blurring. [Effects of the Invention]

[0032] The weakly supervised deep learning method of the present invention is a form of deep learning that allows learning for image classification using only images that have only image-level labels (normal / benign / malignant) without image annotation for regions of interest. Such a weakly supervised deep learning algorithm has the following advantages:

[0033] First, compared to fully supervised learning, which requires image annotation of regions of interest, learning can be achieved using only image-level labels (normal / benign / malignant), saving time, effort, and cost. It also enables image data to be learned through an efficient process, making it possible to develop robust deep learning models (with high diagnostic performance and applicable to other data). In particular, since deep learning performance is determined by the data, an efficient data learning process using a weakly supervised deep learning algorithm can facilitate big data learning, improve the performance of deep learning models, reduce validation errors when applying models, and develop reproducible models.

[0034] Second, because the weakly supervised deep learning algorithm does not require manual image annotation of regions of interest, it minimizes bias that may arise in subjective pre-judgment of lesion characteristics (boundaries and patterns based on the likelihood of benign / malignant), making it possible to develop reproducible models that can operate stably even on external validation data.

[0035] Third, models based on image annotation for regions of interest require annotated images as input data for clinical application. This means that users must specify a region of interest and perform ROI segmentation each time to obtain the prediction value of the deep learning model for an image, which is time-consuming and reduces its clinical utility. However, the invented weakly supervised learning method allows ultrasound images to be easily input using the co-developed preprocessing method, making the workflow using the model more efficient.

[0036] Fourth, when differentially diagnosing lesions, information on the area surrounding the lesion, rather than the lesion itself, is often useful. However, conventional auxiliary diagnosis methods based on fully supervised learning only use the lesion for learning, which limits their ability to comprehensively learn image information. However, weakly supervised learning learns not only the lesion area but also the area surrounding the lesion, making it possible to comprehensively learn information useful for differential diagnosis, ultimately leading to further improvements in diagnostic performance.

[0037] Fifth, conventional diagnostic methods are limited to the task of differentially diagnosing breast masses and therefore cannot use normal (lesion-free) images as input data. However, a deep learning model based on weakly supervised learning can use the entire ultrasound image as input data, making it more suitable for use in ultrasound screening, where abnormalities must be identified from a large number of normals.

[0038] Finally, the developed deep learning algorithm can accurately and reproducibly indicate the probability of malignancy based on parameter values ​​learned from a large amount of data, thereby reducing the disparity in diagnoses between users, and can reduce errors in user detection by providing visualized contribution areas. Such diagnostic assistance functions can reduce unnecessary tissue examinations and surgeries, minimize harm to patients, and reduce societal medical costs. [Brief explanation of the drawings]

[0039] [Figure 1] FIG. 1 is a flow chart that outlines a method for normal / benign / malignant image classification and visualization (localization) of contributing regions according to one aspect of the present invention. [Figure 2a] FIG. 2a is a block diagram illustrating the operation of one embodiment of a method for localizing and visualizing malignant tumors according to one aspect of the present invention in comparison with the prior art. [Figure 2b] FIG. 2b is a block diagram illustrating a test data set used to verify the operation of the method shown in FIG. 2a. [Figure 2c]FIG. 2c is a graph illustrating the performance of the method shown in FIG. 2a. [Figure 2d] FIG. 2d is a diagram showing an example of the results visualized by the method shown in FIG. 2a. [Figure 3] FIG. 3 is a block diagram illustrating the configuration and operation of an embodiment of a system for locating and visualizing malignant tumors according to another aspect of the present invention. [Figure 4] FIG. 4 is a diagram illustrating a dilated convolution process that can be applied to the system for localizing and visualizing malignant tumors in FIG. [Figure 5a] FIG. 5a is a diagram illustrating the concept of a learnable box filter that can be applied to the system for localizing and visualizing malignant tumors in FIG. [Figure 5b] FIG. 5b is a diagram illustrating the concept of a learnable box filter that can be applied to the system for localizing and visualizing malignant tumors in FIG. [Figure 5c] FIG. 5c is a diagram illustrating the concept of a learnable box filter that can be applied to the system for localizing and visualizing malignant tumors in FIG. [Figure 5d] FIG. 5d is a diagram illustrating the concept of a learnable box filter that can be applied to the system for localizing and visualizing malignant tumors in FIG. [Figure 5e] FIG. 5e is a diagram illustrating the concept of a learnable box filter that can be applied to the system for localizing and visualizing malignant tumors in FIG. [Figure 6] FIG. 6 is a block diagram illustrating the configuration and operation of another embodiment of a system for locating and visualizing malignant tumors according to another aspect of the present invention. [Figure 7] FIG. 7 is a diagram illustrating a CAM algorithm that can be applied to the system for locating and visualizing malignant tumors in FIG. [Figure 8]FIG. 8 is a diagrammatic summary of the architecture of the system for localizing and visualizing malignant tumors in FIG. [Figure 9] FIG. 9 is a graph showing the performance of the system for localizing and visualizing the malignant tumor in FIG. [Figure 10] FIG. 10 is a diagram illustrating several examples of heat maps implemented by the system of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0040] For a full understanding of the present invention, its operating advantages, and the objects attained by its practice, reference should be made to the accompanying drawings which illustrate a preferred embodiment of the invention and the contents thereof.

[0041] In order to explain the present invention in detail, preferred embodiments of the present invention will be described below with reference to the accompanying drawings. However, the present invention can be embodied in various different forms and is not limited to the embodiments described below. In order to clearly explain the present invention, parts that are not relevant to the description will be omitted, and the same reference numerals in the drawings will refer to the same elements.

[0042] FIG. 1 is a flow chart that outlines a method for normal / benign / malignant image classification and visualization (localization) of contributing regions according to one aspect of the present invention.

[0043] In the method of the present invention for localizing and visualizing malignant tumors based on breast ultrasound images with only image-level labels using a convolutional neural network (CNN) to which a weakly supervised deep learning (DL) algorithm is applied, an ultrasound image is first received (S110). Because the received image may contain information that identifies the patient and information related to the ultrasound modality, such information is removed to leave only the image information (S130). A de-identification technique for automatically distinguishing image regions from an ultrasound image can be implemented using presets for each ultrasound image modality and image processing techniques (e.g., region growing + hole filling). However, it should be noted that this is provided for illustrative purposes only and is not intended to limit the present invention.

[0044] The input image may include both malignant and benign tumors. Next, the input data is received, and a feature map is acquired from the received input data using a convolutional neural network (CNN) and global average pooling (GAP) for re-learning (S150). Unlike conventional techniques, the present invention does not require a process of marking and annotating suspected lesions in the input image, but instead acquires and re-learns a feature map directly from the input image. This significantly reduces the time and effort required for manual annotation, improves learning effectiveness, and increases the reliability of the results.

[0045] Next, the input data is classified as normal, benign, or malignant using global average pooling (GAP), and if the data is classified as malignant, the probability of malignancy (POM) is calculated, which indicates the accuracy of the classification (S170). In other words, in the present invention, the probability of a result being classified as malignant is also provided, making it possible to easily verify the classification result.

[0046] Once the probability of malignancy (POM) is calculated, the discrimination results of the input data are backpropagated through a convolutional neural network (CNN), the contribution of each pixel that contributed to the discrimination result is calculated from the gradient and features, and the relevant region that contributed to the discrimination based on the calculated contribution of each pixel is visualized together with the probability of malignancy (POM) (S190). That is, in the method for locating and visualizing malignant tumors of the present invention, rather than displaying only the contributing region without the input image, the contributing region classified from the entire input image is directly displayed and visualized using different colors according to the probability of malignancy (POM). Therefore, since it is possible to diagnose by referring to the shape and characteristics of the surrounding area of ​​the lesion, the accuracy of diagnosis is improved, user convenience is enhanced, and the method can be effectively utilized in the process of breast cancer diagnosis.

[0047] Such a weakly supervised deep learning algorithm is trained by verifying the performance of the diagnostic results and repeating the weakly supervised deep learning algorithm based on the verified performance. In this way, the weakly supervised deep learning algorithm is trained using the diagnostic results, so it is possible to quickly and accurately improve diagnostic performance without manual annotation.

[0048] FIG. 2a is a block diagram illustrating the operation of one embodiment of a method for localizing and visualizing malignant tumors according to one aspect of the present invention in comparison with the prior art.

[0049] As shown in Figure 2a, unlike conventional techniques, the method for localizing and visualizing malignant tumors in the present invention does not require manual or automatic extraction (or cropping) and annotation of lesion areas from input images. Instead, the input image that has undergone the de-identification process is directly input to the classifier. In other words, rather than only the image of the lesion being input to the CNN, almost the entire input image including the lesion is input. In addition, a heat map is provided along with the diagnosis result for user convenience, helping to better identify the lesion area along with its probability.

[0050] FIG. 2b is a block diagram illustrating a test data set used to verify the operation of the method shown in FIG. 2a.

[0051] First, 1,400 ultrasound images were collected from two hospitals. A weakly supervised deep learning algorithm, implemented by three networks (VGG16, ResNet34, and GoogLeNet), was trained using 1,000 ultrasound images (500 benign and 500 malignant masses) with only per-image labels. Two image sets of 200 images (100 benign and 100 malignant masses) were used as internal and external validation sets, respectively.

[0052] ROI annotation is then performed manually and automatically using a deep segmentation network (U-Net) for comparison with a fully supervised algorithm. Diagnostic performance is calculated as the area under the receiver operating characteristic curve (AUC) and compared between the weakly supervised and fully supervised deep learning algorithms. Class activation maps are used to determine how accurately the weakly supervised deep learning algorithm localized breast masses.

[0053] Figure 2c shows the performance of the method shown in Figure 2a. Receiver operating characteristic (ROC) curves for different classifiers using a weakly supervised algorithm and a fully supervised deep learning (DL) algorithm with manual and automatic ROI annotation on the internal (A) and external (B) validation sets. (A) The area under the receiver operating characteristic (AUC) of the weakly supervised DL algorithm is not different from that of the fully supervised DL algorithm (all Ps > 0.05). (B) The AUC of the weakly supervised DL algorithm is either statistically insignificant (Ps > 0.05) or higher (P = 0.04, VGG16 with automated ROI annotation) compared to that of the fully supervised DL algorithm.

[0054] As shown in Figure 2c, our method for localizing and visualizing malignant tumors achieved performance comparable to or better than conventional fully supervised learning, even when untagged input images were fed into the CNN.

[0055] Figure 2d shows an example of the results visualized by the method shown in Figure 2a. Examples of true positives (A), false negatives (B), false positives (C), and true negatives (D) are displayed for each network (VGG16, ResNet34, and GoogLeNet). (A) The ultrasound image shows a 17-mm irregular, spiky invasive ductal carcinoma, which was predicted as malignant with a probability of malignancy (POM) of 1.00, 1.00, and 0.999 for VGG16, ResNet34, and GoogLeNet, respectively. (B) The ultrasound image shows an 11-mm oval, circumscribed, isoechoic mucinous carcinoma, which was predicted as benign with a POM of 0.007, 0.000, and 0.000, respectively. (C) Ultrasound image shows a 29-mm ovoid macrocalcification (unchanged over 46 months of follow-up) considered benign, predicted as malignant with POMs of 1.000, 0.994, and 1.000, respectively. (D) Ultrasound image shows a 6-mm ovoid circumscribed mass (unchanged over 55 months of follow-up) considered benign, predicted as benign with POMs of 0.434, 0.006, and 0.006, respectively.

[0056] As a result, for the internal validation set, the weakly supervised deep learning algorithm achieved superior diagnostic performance, with AUCs of 0.92–0.96, which were not statistically different (all Ps > 0.05) from those of the fully supervised deep learning algorithm with manual or automatic ROI annotation (AUC, 0.92–0.96). For the external validation set, the weakly supervised deep learning algorithm achieved AUCs of 0.86–0.90, which were either not statistically different (Ps > 0.05) from those of the fully supervised deep learning algorithm (AUC, 0.84–0.92) or higher (P = 0.04 for automatic ROI annotation, VGG16). On both the internal and external validation sets, the weakly supervised algorithms were able to localize 100% of malignant masses, except for ResNet34 (98%).

[0057] More specifically, we utilize a weakly supervised deep learning algorithm using three representative networks (VGG16, ResNet34, and GoogLeNet) to predict malignant from benign masses along with a probability score. We also test the discriminative localization performance of the weakly supervised deep learning algorithm, which uses class activation maps (CAMs) to highlight regions of the input image that contribute to the predicted outcome. Of the 1,400 images collected, 700 were images of cancer confirmed by biopsy or surgery, and 700 were images of benign masses confirmed by biopsy (n=163) or by subsequent imaging over at least two years (n=537). The dataset included a training set and a validation set. The training set included 500 benign masses and 500 malignant masses obtained from Source A (data collection period: January 2011 - August 2013). The validation sets were divided into an internal validation set and an external validation set, each containing 200 images (100 benign and 100 malignant masses in each set). The internal validation images were temporarily separated from Source A (data collection period: September 2013–July 2014) and were not used for algorithm training. The external validation images were continuously acquired from Source B (data collection period: May 2011–August 2015). All breast ultrasound images were extracted from an image archiving and communication system and saved in JPEG format. For the training and internal validation sets obtained from Source A, a single ultrasound machine was used to generate the images, while for the external validation set (obtained from Source B), various ultrasound machines were used.

[0058] FIG. 3 is a block diagram illustrating the configuration and operation of an embodiment of a system for locating and visualizing malignant tumors according to another aspect of the present invention.

[0059] The system for localizing and visualizing malignant tumors according to the present invention includes a de-identification module (310), a data enrichment module (330), a CNN (350), and a contributing region determination and visualization module (370).

[0060] The de-identification module 310, as part of an ultrasound image pre-processing module (not shown), generates an input image of diagnostic interest in a breast ultrasound image, which may include one or more malignant and benign tumors. To this end, the de-identification module 310 may be configured to crop edges of the breast ultrasound image, remove patient body marks and text tags to generate an anonymized image, and extract the anonymized image of the same size as the input image.

[0061] The data enhancement module (330) is also included in the ultrasound image preprocessing module (not shown), and can add one of geometric and optical modifications to the anonymized image to enhance diagnostic performance. Geometric modifications can include horizontal flip, vertical flip, rotation, scaling, translation, etc., while optical modifications can include removal, speckle noise reduction, sharpening, Gaussian blur, etc. However, it should be noted that this is provided for illustrative purposes only and is not intended to limit the present invention.

[0062] The output of the data enrichment module 330 is input to the CNN 350. In the present invention, the CNN 350 can be considered to perform the functions of a deep learning module (not shown) and a differential diagnosis module (not shown) depending on its function.

[0063] In this case, a deep learning module (not shown) receives input data and uses a convolutional neural network (CNN) and global average pooling (GAP) to obtain a feature map from the received input data and re-learn it. Also, a differential diagnosis module (not shown) uses global average pooling (GAP) to classify the input data as normal, benign, or malignant, and if it is determined to be malignant, calculates the probability of malignancy (POM), which represents the accuracy of the classification.

[0064] Next, the contribution area determination and visualization module (170) backpropagates the results of discrimination from the input data through the convolutional neural network (CNN), calculates the contribution of each pixel that contributed to the discrimination result from the gradient and features, and visualizes the contribution area that contributed to the discrimination based on the calculated contribution of each pixel, along with the probability of malignancy (POM).

[0065] More specifically, the de-identification module 310 extracts an anonymized image by using a dedicated de-identification module developed for ultrasound images to remove information related to the patient's personal information from the DICOM header information and trimming the edges of the ultrasound image to remove remaining personal information and body marks and text tags that interfere with learning. Alternatively, the de-identification module 310 can directly generate image data based on ultrasound signals received from an ultrasound device. In this case, it is preferable that the size of the anonymized image or image data be adjusted to match the size of the input data.

[0066] The deep learning module can also apply a data augmentation process by adding at least one of geometric and optical variations to the anonymized images to reduce validation errors and generate a reproducible model. Furthermore, the deep learning module configures a deep learning algorithm architecture using at least one convolutional neural network (CNN) for image classification, such as VGG16, ResNet34, or GoogLeNet, to learn whether the input data is normal, benign, or malignant, and obtains feature maps for each convolutional layer. The deep learning module determines parameters for the obtained feature maps and retrains them to improve learning accuracy.

[0067] The differential diagnosis module uses a global average pooling (GAP) layer added after the convolutional layer in the architecture of the deep learning algorithm to acquire features of each feature map of the last convolutional layer, and calculates a class score by applying learnable weights to the acquired features. The differential diagnosis module then calculates a probability of malignancy (POM) based on the calculated class scores.

[0068] After the probability of malignancy (POM) is calculated, the contributing region determination and visualization module applies weights to the feature maps to obtain class activation maps (CAMs) for each class, scales the obtained class activation maps (CAMs) by applying min-max normalization, and performs binarization by applying a threshold to the scaled class activation maps (CAMs) to determine the contributing region.The contributing region determination and visualization module then visualizes the determined class activation maps (CAMs) together with the probability of malignancy (POM) as a heat map.

[0069] The visualized results can be used to train a weakly supervised deep learning algorithm based on the validated performance of the region of interest (ROI) and probability of malignancy (POM).

[0070] Table 1 shows the results obtained using the enriched data.

[0071] [Table 1]

[0072] The overall operation of the present invention will be described in more detail below using a specific embodiment of the present invention.

[0073] Image annotation and preprocessing Images were anonymized by cropping the image edges to remove body marks and text tags. For the weakly supervised deep learning algorithm, no ROI annotation was performed, and no additional data curation was performed to test the performance of the proposed system. To compare with the fully supervised deep learning algorithm, ROI annotation was performed using two methods: manual drawing and automated deep learning-based segmentation. For manual drawing, a radiologist indicated the ROIs and used an in-house drawing tool to create a binary mask for each mass. For automated deep learning-based segmentation, a deep segmentation network, U-Net, was employed, which was developed for segmenting medical images. After ROI annotation, a rectangle with a fixed 30-pixel margin surrounding the corresponding mass was extracted. The image was resized to 224x224 pixels, and pixel intensities were normalized to 0–1 using the maximum intensity value.

[0074] Deep Learning Classification Model For the deep learning classifier, three representative convolutional neural networks, VGG16, ResNet34, and GoogLeNet, which have achieved state-of-the-art performance in various computer vision tasks, are adopted. However, it should be noted that this is provided for illustrative purposes only and is not intended to limit the present invention. The classification performance of such CNN models has been evaluated in multiple studies.

[0075] VGG16 is a feedforward neural network stacked with five convolutional blocks, each consisting of two 3x3 convolutional layers and a max-pooling layer to learn relevant image features for classification tasks. The ResNet34 architecture is a deep CNN with a residual connection function that can solve the vanishing gradient problem as a shortcut after the weighting layer. Finally, GoogLeNet consists of a stack of inception modules for learning visual patterns at different scales. A single inception module contains various convolutional branches with different filter sizes, from 1x1 to 5x5, to cost-effectively expand the receptive field. The architectural details of CNN models are publicly known.

[0076] To test the performance of discriminative localization by weakly supervised deep learning algorithms, classification models can be extended with a global average pooling (GAP) layer added to the last convolutional layer of each model. GAP then calculates the average feature map (f k ) are averaged as follows to calculate the feature score (F k )

[0077]

number

[0078] where i and j are k is a spatial index of

[0079] The number of feature maps is the number of classes (N c ) and the model performs linear classification using fully connected layers and a softmax function. The learnable weights are W={w k,c The fully connected layer with} calculates the score (S c ) can be calculated as follows:

number

[0080] The class scores are applied to a softmax function to obtain the predicted probabilities for all classes. c ) and probability of malignancy (POM) are calculated as follows:

number

number

number

[0081] M c The relative intensities of the are scaled using min-max normalization to visually distinguish and compare important and unimportant regions in estimating class scores. The scaled class activation maps (M' c) can be obtained as follows:

number

[0082] The regions that had a significant impact when estimating tumor characteristics were identified by scaling the scaled map with a threshold (M' c ≥ 0.3), the threshold can be empirically selected to consider the overlap of multiple binarized maps in the training set on which manual ROI annotations have been performed.

[0083] Performance metrics and statistical analysis For the diagnosis of malignant versus benign tumors, the area under the receiver operating characteristic curve (AUC) was used as the primary metric for comparing the performance of algorithms, and the DeLong test of significance was used to compare the AUC of two correlated receiver operating characteristic curves (ROCs). The exact McNemar test was used to test for differences in sensitivity and specificity. Discriminative localization was considered accurate if the segmented area overlapped with the manually annotated area. All statistical analyses were performed using the statistical software MedCalc, version 17.1 (Mariakerke, Belgium). A two-sided P value of <0.05 was considered statistically significant.

[0084] Performance Metrics For the internal validation test set, the weakly supervised deep learning models achieved high performance in distinguishing benign from malignant breast masses, with AUCs of 0.96 (95% CI: 0.92, 0.98), 0.92 (95% CI: 0.88, 0.96), and 0.94 (95% CI: 0.90, 0.97) for the VGG16, ResNet34, and GoogLeNet models, respectively (Table 2). The AUCs for the fully supervised deep learning models with manual annotations were 0.96 (95% CI: 0.93, 0.98), 0.94 (95% CI: 0.89, 0.97), and 0.96 (95% CI: 0.92, 0.98) for the VGG16, ResNet34, and GoogLeNet models, respectively. The AUCs of the fully supervised deep learning model with automatic annotation were 0.96 (95% CI: 0.92, 0.98), 0.92 (95% CI: 0.87, 0.95), and 0.95 (95% CI: 0.91, 0.98), respectively. Thus, the AUCs of the weakly supervised deep learning model did not differ from those of the fully supervised deep learning model with manual or automatic ROI annotation (all Ps > 0.05).

[0085] The sensitivity of the weakly supervised deep learning models was 87% (87 / 100), 82% (82 / 100), and 87% (87 / 100) for the VGG16, ResNet34, and GoogLeNet models, respectively, and the specificity was 91% (91 / 100), 91% (91 / 100), and 94% (94 / 100), respectively. The sensitivity and specificity did not differ between the weakly supervised and fully supervised deep learning models (all Ps > 0.05).

[0086] For the external validation test set, weakly supervised deep learning models achieved high diagnostic performance, with somewhat lower AUCs of 0.89 (95% CI: 0.84, 0.93), 0.86 (95% CI: 0.81, 0.91), and 0.90 (95% CI: 0.85, 0.94) for the VGG16, ResNet34, and GoogLeNet models, respectively. The AUCs for fully supervised deep learning models with manual annotations were 0.91 (95% CI: 0.86, 0.95), 0.89 (95% CI: 0.84, 0.93), and 0.92 (95% CI: 0.87, 0.95) for the VGG16, ResNet34, and GoogLeNet models, respectively. The AUCs of the fully supervised deep learning model with automatic annotation were 0.85 (95% CI: 0.79, 0.89), 0.84 (95% CI: 0.78, 0.88), and 0.87 (95% CI: 0.82, 0.92), respectively. The AUCs of the weakly supervised deep learning model were not statistically different from those of the fully supervised deep learning model with manual ROI annotation (all Ps > 0.05).

[0087] For the VGG16 network, the AUC was significantly higher for the weakly supervised deep learning model than for the fully supervised deep learning model with automatic ROI annotation (P = 0.04). For the ResNet34 and GoogLeNet networks, no significant differences were found between the weakly supervised and fully supervised deep learning models with automatic ROI annotation (all Ps > 0.05). For VGG16 and GoogLeNet, the sensitivity was not significantly different between the weakly supervised (91% [91 / 100] and 88% [88 / 100]) and fully supervised deep learning models (85% [85 / 100] and 87% [87 / 100]) (all Ps > 0.05).

[0088] For the ResNet34 model, sensitivity was lower in the weakly supervised deep learning model (78% [78 / 100]) than in the fully supervised model with manual annotation (89% [89 / 100], P < 0.001), but was not significantly different from the fully supervised deep learning model with automatic ROI annotation (81% [81 / 100], P = 0.66). The specificity of the weakly supervised deep learning models (72% [72 / 100], 80% [80 / 100], and 76% [76 / 100] for the VGG16, ResNet34, and GoogLeNet networks, respectively) was not significantly different from that of the fully supervised deep learning models with manual ROI annotation for the VGG16 (52% [52 / 100]) and GoogLeNet models (63% [63 / 100]) (all Ps > 0.05), but was lower than that of the ResNet34 model (90% [90 / 100], P < 0.001).

[0089] Specificity was statistically significant or borderline significant (P<0.001, P=0.07, P=0.04 for VGG16, ResNet34, and GoogLeNet models, respectively) and higher in the weakly supervised deep learning models than in the fully supervised deep learning models with automatic ROI annotation.

[0090] Performance metrics for discriminative localization On the internal validation set, the weakly supervised deep learning model using VGG16 and ResNet34 networks was able to localize 99% (99 / 100) of benign masses and 100% (100 / 100) of malignant masses. The GoogLeNet network was able to localize 100% (100 / 100) of both benign and malignant masses. On the external validation set, the weakly supervised deep learning model using VGG16, ResNet34, and GoogLeNet networks was able to localize 99%, 96%, and 97% (97 / 100) of benign masses and 100%, 98%, and 100% (100 / 100) of malignant masses, respectively.

[0091] As described above, compared with fully supervised deep learning algorithms based on conventional techniques, the weakly supervised deep learning algorithm of the present invention was found to have superior diagnostic performance (AUC: 0.86-0.96), and was not deficient when compared with fully supervised deep learning algorithms with manual annotation (AUC: 0.89-0.96) and fully supervised deep learning algorithms with automatic annotation (AUC: 0.84-0.96). Furthermore, the weakly supervised deep learning algorithm can accurately localize benign and malignant masses with a nearly perfect rate (96%-100%).

[0092] Such excellent classification and localization performance is achieved even on a relatively small dataset and an external validation set using different breast images and ultrasound equipment. Taking the above together, we demonstrate that weakly supervised deep learning algorithms can be used to support decision support and aid in the detection and diagnosis of breast cancer from ultrasound images through a highly efficient data-curation process that does not require manual drawing or automatic ROI segmentation.

[0093] Although deep learning algorithms have been successfully applied to mammography analysis, the development of deep learning algorithms for breast ultrasound images is still in its early stages. Nevertheless, it is promising that deep learning algorithms can perform well in diagnosing breast masses in ultrasound images.

[0094] While prior art CAD-based diagnosis still has limitations when applied to images acquired by other equipment, the method and system for localizing and visualizing malignant tumors according to the present invention is versatile because it accepts input images of the same size as input. That is, the present invention can be widely applied to image-based classification using chest X-rays, MRIs, or histopathology images, and region-based classification using breast ultrasound images.

[0095] FIG. 4 is a diagram illustrating a dilated convolution process that can be applied to the system for localizing and visualizing malignant tumors in FIG.

[0096] The receptive field refers to the region in the input space where a CNN model can operate to perform a given task, but to widen the receptive field, it is necessary to increase the size of the filter or stack more layers, which may result in an explosive increase in the learnable parameters and numerical operations in the neural network.

[0097] To solve this problem, we propose a dilation / convolution process in which a predefined gap is set between pixels before performing the convolution operation. As shown in Figure 4, the dilation / convolution process can expand the receptive field without increasing the amount of computation.

[0098] 5a to 5e are diagrams illustrating the concept of a trainable box filter that can be applied to the system for localizing and visualizing malignant tumors in FIG.

[0099] Referring to Figure 5a, the trainable box filter averages over all positions and treats the box coordinates as trainable parameters.

[0100] Referring to Figure 5b, a convolutional average filter adds up all the values ​​in the filter and then divides them by the filter size. In a similar way, a box filter keeps the filter values ​​fixed, but varies the filter size parameter θ = (x min ,x max ,y min ,y max ) is learned. Note that although an example of a 3x3 box mean filter is shown in Figure 5b, this is provided for illustrative purposes only and is not intended to limit the present invention.

[0101] Referring to Figure 5c, the box convolution layer supports channel-specific operations and trains B different box filters for C input channels, respectively. Therefore, the output consists of CB channels, and the number of learnable parameters is a fixed size of 4CB, which is independent of the filter size.

[0102] Thereby, the input map I can be expanded using zero padding so that the output feature map maintains a size of O. In the forward pass, O can be determined by a convolution operation with a box filter.

number

[0103] 5d is a diagram illustrating an integral image. As shown in FIG. 5d, an integral image is an algorithm for generating the sum of values ​​within a rectangular subset, such as a grid. It should be noted that FIG. 5d is provided for convenience of explanation only and is not intended to limit the present invention.

[0104] 5e is an example of a box block that can be applied to the present invention. The box block has a cross-channel 1x1 convolution layer because the box convolution layer operates independently for each channel. The box block can also be extended through constellation normalization and residual connection.

[0105] FIG. 6 is a block diagram illustrating the configuration and operation of another embodiment of a system for locating and visualizing malignant tumors according to another aspect of the present invention.

[0106] Figure 6 is the same as the block diagram of Figure 3, except that it includes a convolution block (CB) and a box convolution block (BB) as components of the CNN. Therefore, for the sake of brevity, a redundant description will be omitted.

[0107] FIG. 7 is a diagram illustrating a CAM algorithm that can be applied to the system for locating and visualizing malignant tumors in FIG.

[0108] A class activation map (CAM) is a map of specific classes that can be used to evaluate localization performance. It allows for the visualization of lesion-related regions in the input image.

[0109] Classification methods using deep learning algorithms can be categorized into region-based and image-based classification. In region-based classification, tumor ROI extraction precedes the classification process. In the present invention, image segmentation methods such as U-Net, fully convolutional networks, and Faster R-CNN are employed to extract contributing regions. Image-based classification receives the entire ultrasound image (including the entire breast tissue surrounding the tumor) as input and learns ultrasound features relevant to tumor classification. Our weakly supervised deep learning algorithm performs better than fully supervised deep learning algorithms when distinguishing malignant and benign breast masses from ultrasound images. The human-constructed algorithm employed in the Breast Imaging Reporting and Data System (BI-RADS) considers both the breast tissue of the mass and the surrounding breast tissue. Therefore, weakly supervised deep learning, which uses information from the entire image (not limited to the mass or its surroundings), can demonstrate advantages over fully supervised deep learning.

[0110] The proposed algorithm is able to learn important parts of the BI-RADS lexicon technical information outside the mass (e.g., posterior features, architectural distortion, and skin changes and edema) that are known to be useful for differential diagnosis.

[0111] Furthermore, the weakly supervised deep learning algorithm can also be applied to a deep learning-assisted detection framework for ultrasound breast lesions, because the proposed algorithm can learn both diseased and disease-free regions in breast tissue. Learning disease-free regions is particularly important for developing a detection framework because the ultrasound appearance of breast tissue (especially the glandular area) varies from person to person and varies greatly from part to part of the breast, especially depending on the amount and distribution of fibrous and glandular elements. Furthermore, knowing the normal appearance of anatomical structures from the skin to the chest wall is essential for detecting pathological structures.

[0112] In summary, the GAP used in the present invention has the following advantages:

[0113] First, the computational load and parameters can be greatly reduced, thus making the network faster and more robust against overfitting problems.

[0114] Second, the present invention enhances feature maps to preserve class-related spatial information, allowing them to be used to interpret the decisions of CNN models. This method for identifying diagnostically influential regions that distinguish malignant from benign cases using GAP with CAM is related to the concept of eXplainable AI (XAI). XAI, or responsible-AI, is an emerging paradigm for overcoming the inherent "black box problem" inherent in deep learning frameworks, which makes it impossible to understand how decisions are made. CAM provides intuition for analyzing the decision-making process embodied by AI. Furthermore, weakly supervised deep learning with CAM enables healthcare providers to develop deep learning-assisted detection frameworks for clinically important regions.

[0115] Third, an advantage of the present invention is that training with a large dataset is not required to implement the proposed weakly supervised deep learning algorithm.

[0116] Fourth, a weakly supervised deep learning algorithm without image ROI annotation for detecting and diagnosing malignant from benign breast masses accurately detects breast masses and performs comparably to a conventional fully supervised deep learning algorithm. The successful performance of diagnostic and discriminative localization for distinguishing malignant from benign breast masses and the high efficiency of data curation obtained by the weakly supervised deep learning algorithm suggest that the present invention is feasible and a valuable candidate for testing in future clinical studies.

[0117] FIG. 8 is a diagram that schematically summarizes the architecture of the system for localizing and visualizing malignant tumors in FIG. 3, and FIG. 9 is a graph showing the performance of the system for localizing and visualizing malignant tumors in FIG. 3.

[0118] As mentioned above, weakly supervised deep learning algorithms do not require image annotation of the lesion region of interest (ROI), while fully supervised deep learning algorithms require tumor segmentation (manual or automatic) and cropping of the ROI before entering the classifier. For weakly supervised deep learning algorithms, class activation maps (CAMs) are generated to visualize the areas detected by such algorithms using a global average pooling layer (GAP) added to the last convolutional layer.

[0119] Such performance can be determined by the following mathematical formula:

number

[0120] For information on Equation 8, see Chanho Kim, Won Hwa Kim, Hye Jung Kim, and Jaeil Kim, “Weakly-supervised US breast tumor characterization and localization with a box convolution network.”

[0121] FIG. 10 is a diagram illustrating several examples of heat maps implemented by the system of FIG.

[0122] Referring to FIG. 10, the method and system for localizing and visualizing malignant tumors according to the present invention provides performance comparable to or better than conventional fully supervised learning techniques and directly represents image information about the tissue surrounding the lesion, thereby improving the accuracy of screening.

[0123] While the present invention has been described with reference to the embodiments shown in the drawings, these are for illustrative purposes only, and those skilled in the art will recognize that numerous variations and equivalent embodiments can be conceived based thereon. For example, to broaden the use of weakly supervised deep learning algorithms in medical imaging, such algorithms can be applied to images obtained from other body parts or other imaging modalities.

[0124] The method according to the present invention can also be embodied as computer-readable code on a computer-readable storage medium. The computer-readable storage medium can include any type of recording device on which data readable by a computer system is stored. Examples of computer-readable storage media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., and can also be embodied in the form of carrier wave (e.g., transmission over the Internet). The computer-readable storage medium can also store computer-readable code that can be executed in a distributed manner by distributed computer systems connected via a network.

[0125] In terms used in this specification, the singular includes the plural unless the context clearly dictates otherwise, and the predicate or modifier terms "comprise," "include," and the like mean the presence of a stated feature, number, step, operational component, component, or combination thereof, but do not exclude the possibility of the presence or addition of one or more other features, numbers, steps, operational components, components, or combinations thereof. Furthermore, the terms "module," "device," "module," "block," and the like used in this specification mean a unit that processes at least one function or operation, and can be embodied in hardware, software, or a combination of hardware and software.

[0126] Therefore, the present embodiment and the drawings attached to this specification merely clearly show a part of the technical ideas contained in the present invention, and it is obvious that all examples of modifications and specific embodiments that can be easily conceived by a person skilled in the art within the scope of the technical ideas contained in the specification and drawings of the present invention are included in the scope of the rights of the present invention. [Industrial Applicability]

[0127] The weakly supervised deep learning algorithm developed in this invention provides good localization and It can be used for ultrasound diagnosis of breast cancer, which can distinguish between malignant and benign cancers. is. The inventions described in the original claims of this application are set forth below. [Appendix 1] A breast ultrasound diagnosis method using weakly supervised deep learning (DL) artificial intelligence. an ultrasound image preprocessing stage that removes patient personal information from breast ultrasound images and generates input data containing only the image regions required for learning; a deep learning stage in which the input data is received, and a feature map is obtained from the received input data using a convolutional neural network (CNN) and global average pooling (GAP) to perform re-learning; A differential diagnosis step of classifying the input data as normal, benign, or malignant using global average pooling (GAP), and calculating a probability of malignancy (POM) representing the accuracy of the classification if the data is determined to be malignant; and The discrimination result is backpropagated through the convolutional neural network (CNN), and the contribution of each pixel that contributed to the discrimination result is calculated from the gradient and feature amount. Based on the calculated contribution of each pixel, the relevant contribution area that contributed to the discrimination is calculated. a contributing region determination and visualization step of visualizing the contributing region together with the probability of malignancy (POM); A breast ultrasound diagnosis method using weakly supervised deep learning artificial intelligence, characterized in that the deep learning step is learned based on the verified performance of the contributing region and the probability of malignancy (POM). [Appendix 2] In Appendix 1, The ultrasound image pre-processing step includes: Using a dedicated de-identification module developed for ultrasound images to remove patient information from the DICOM (Digital Imaging and Communications in Medicine) header; Extracting anonymized images by trimming the edges of the ultrasound images to remove remaining personal information and distracting body marks and text tags, or generating image data based on ultrasound signals received from the ultrasound machine; and A breast ultrasound diagnosis method using weakly supervised deep learning artificial intelligence, characterized in that it includes resizing the anonymized image or image data to input data of the same size as the image. [Appendix 3] In Appendix 2, The deep learning stage includes: applying a data augmentation process to the anonymized images by adding at least one of geometric and optical variations to reduce validation errors and make the model reproducible; To learn whether the input data is normal / benign / malignant, a deep learning algorithm architecture is constructed using at least one convolutional neural network (CNN) that performs image classification, and a feature map corresponding to each convolutional layer is obtained; and A breast ultrasound diagnosis method using weakly supervised deep learning artificial intelligence, characterized in that it includes determining and re-learning parameters related to the acquired feature map. [Appendix 4] In Appendix 3, The differential diagnosis step comprises: Using a global average pooling (GAP) layer added after the convolutional layer in the architecture of the deep learning algorithm, to obtain the feature values ​​of each feature map of the last convolutional layer; A breast ultrasound diagnosis method using weakly supervised deep learning artificial intelligence, characterized by including calculating class scores by applying learnable weights to acquired features, and calculating a probability of malignancy (POM) based on the calculated class scores. [Appendix 5] In Appendix 4: The contributing area determination and visualization step comprises: applying weights to the feature maps to obtain class activation maps (CAMs) for each class; Scaling the obtained class activation maps (CAMs) by applying min-max normalization; performing threshold binarization on the scaled class activation map (CAM) to determine the contributing regions; and A breast ultrasound diagnosis method using weakly supervised deep learning artificial intelligence, comprising visualizing the class activation map (CAM) together with the probability of malignancy (POM) as a heat map. [Appendix 6] In Appendix 3, the geometric transformation comprises at least one of a horizontal flip, a vertical flip, a rotation, a scaling, and a translation; A breast ultrasound diagnostic method using weakly supervised deep learning artificial intelligence, characterized in that the optical modifications include at least one of deletion, speckle noise processing, sharpening, and Gaussian blurring. [Appendix 7] A breast ultrasound diagnostic system using weakly supervised deep learning (DL) artificial intelligence, An ultrasound image preprocessing module that removes patient personal information from breast ultrasound images and generates input data containing only the image regions required for learning; a deep learning module that receives the input data and re-learns by using a convolutional neural network (CNN) and global average pooling (GAP) to obtain a feature map from the received input data; A differential diagnosis module that uses global average pooling (GAP) to classify the input data as normal, benign, or malignant, and if the data is determined to be malignant, calculates a probability of malignancy (POM) that indicates the accuracy of the classification; and a contribution region determination and visualization module that backpropagates the discrimination result through the convolutional neural network (CNN), calculates the contribution of each pixel that contributed to the discrimination result from gradients and features, and visualizes the contribution region that contributed to the discrimination based on the calculated contribution of each pixel together with the probability of malignancy (POM); A breast ultrasound diagnostic system using weakly supervised deep learning artificial intelligence, characterized in that the weakly supervised deep learning algorithm is trained based on the verified performance of the contributing region and the probability of malignancy (POM). [Appendix 8] In Appendix 7: The ultrasound image pre-processing module includes: Using a dedicated de-identification module developed for ultrasound images, we remove any information related to the patient's personal information from the DICOM header. Anonymized images are extracted by trimming the edges of the ultrasound image and removing any remaining personal information and body marks and text tags that may interfere with learning, or image data is generated based on the ultrasound signal received from the ultrasound device. A breast ultrasound diagnostic system using weakly supervised deep learning artificial intelligence, characterized in that it is configured to resize the anonymized image or image data to input data of the same size as the image. [Appendix 9] In Appendix 8: The deep learning module applying a data augmentation process to the anonymized images by adding at least one of geometric and optical variations to reduce validation errors and to make the model reproducible; In order to learn whether the input data is normal / benign / malignant, a deep learning algorithm architecture is constructed using at least one convolutional neural network (CNN) that performs image classification, and a feature map corresponding to each convolutional layer is obtained; A breast ultrasound diagnostic system using weakly supervised deep learning artificial intelligence, characterized in that it is configured to determine and re-learn parameters related to the acquired feature map. [Appendix 10] In Appendix 9: The differential diagnosis module includes: A global average pooling (GAP) layer is added next to the convolution layer in the architecture of the deep learning algorithm to obtain the feature values ​​of each feature map of the last convolution layer; A breast ultrasound diagnostic system that uses weakly supervised deep learning artificial intelligence, characterized by being configured to calculate class scores by applying learnable weights to acquired features and to calculate the probability of malignancy (POM) based on the calculated class scores. [Appendix 11] In Appendix 7: The contributing area determination and visualization module: Applying weights to the feature maps to obtain class activation maps (CAMs) for each class; The obtained class activation maps (CAMs) are scaled by applying min-max normalization. performing threshold binarization on the scaled class activation map (CAM) to determine the contributing regions; A breast ultrasound diagnostic system using weakly supervised deep learning artificial intelligence, characterized in that it is configured to visualize the class activation map (CAM) together with the probability of malignancy (POM) as a heat map. [Appendix 12] In Appendix 9: the geometric transformation comprises at least one of a horizontal flip, a vertical flip, a rotation, a scaling, and a translation; A breast ultrasound diagnostic system using weakly supervised deep learning artificial intelligence, characterized in that the optical modifications include at least one of removal, speckle noise processing, sharpening, and Gaussian blurring. [Explanation of symbols]

[0128] 310: De-identification Module 330: Data Enrichment Module 350:CNN 370:Contributing region determination and visualization module

Claims

1. 1. A method implemented by a computing device for diagnostic ultrasound imaging, comprising: acquiring ultrasound image data; generating classification information related to the ultrasound image data using a deep learning model; and utilizing the deep learning model to generate diagnostic visualization information related to the ultrasound image data; Including, The deep learning model A model in which a lesion area and its surrounding area are learned based on classification information of the entire training ultrasound image data without information related to segmentation of the training ultrasound image data, Each pixel of the ultrasound image data includes visualized information indicating a degree of contribution to generating the classification information, and the visualized information is obtained by backpropagating the classification information through a convolutional neural network including one or more convolution layers. method.

2. In claim 1, The classification information is At least one of normal, benign, and malignant method.

3. In claim 1, The deep learning model preprocessing one or more training ultrasound image data sets; Labeling the preprocessed ultrasound image data with classification information to construct a training dataset; and Utilizing the training data set, training the deep learning model to output classification information for each of the ultrasound image data included in the training data set; This is a model trained including method.

4. In claim 3, The step of pre-processing the one or more training ultrasound image data includes: removing patient personal information from the one or more training ultrasound image data; trimming the edges of the ultrasound image data from which the personal information has been removed; and Resizing the edge-trimmed ultrasound image data; Including, method.

5. In claim 3, performing data augmentation on the preprocessed training ultrasound image data; further comprising The step of performing data augmentation on the pre-processed training ultrasound image data includes: performing at least one of geometric transformation and optical transformation on the pre-processed training ultrasound image data to generate at least one new image data; and including the at least one new image data item in the training data set; Including, method.

6. In claim 5, The geometric variation is including at least one of horizontal flip, vertical flip, rotation, scaling, and translation; method.

7. In claim 5, The optical mutation is At least one of the following techniques may be used: removal, speckle noise processing, sharpening, and Gaussian blurring. method.

8. In claim 1, The step of generating classification information related to the ultrasound image data by utilizing the deep learning model includes: generating one or more feature maps based on the ultrasound image data using the deep learning model; Utilizing the deep learning model to calculate a score for each class of the ultrasound image data based on the one or more feature maps; and generating the classification information based on the scores for each class; Including, method.

9. In claim 8, Utilizing the deep learning model to generate one or more feature maps based on the ultrasound image data includes: inputting the ultrasound image data into the convolutional neural network and generating the one or more feature maps therefrom; Including, method.

10. In claim 8, The step of calculating a score for each class of the ultrasound image data based on the one or more feature maps using the deep learning model includes: inputting the one or more feature maps into a global average pooling layer and calculating a score for each class of the ultrasound image data based on the one or more feature maps; Including, method.

11. In claim 8, The step of utilizing the deep learning model to generate diagnostic visualization information related to the ultrasound image data includes: calculating a probability of malignancy based on the classification information; Based on the one or more feature maps, a class activation map (Class generating an Activation Map; and generating the diagnostic visualization information based on the malignancy probability, the class activation map, and the ultrasound image data; Including, method.

12. In claim 11, The step of generating a class activation map for each class based on the one or more feature maps comprises: generating a class activation map for each of the classes based on the one or more feature maps and the weights of the deep learning model; Including, method.

13. In claim 12, The weights of the deep learning model are: The deep learning model includes weights learned in the process of generating the classification information. method.

14. 1. A method implemented by a computing device for diagnostic ultrasound imaging, comprising: acquiring ultrasound image data; generating classification information related to the ultrasound image data using a deep learning model; and utilizing the deep learning model to generate diagnostic visualization information related to the ultrasound image data; Including, The deep learning model A model in which a lesion area and its surrounding area are learned based on classification information of the entire training ultrasound image data without information related to segmentation of the training ultrasound image data, Each pixel of the ultrasound image data includes visualized information indicating a degree of contribution to generating the classification information, and the visualized information is obtained by backpropagating the classification information through a convolutional neural network including one or more convolution layers; The step of generating classification information related to the ultrasound image data by utilizing the deep learning model includes: generating one or more feature maps based on the ultrasound image data using the deep learning model; Utilizing the deep learning model to calculate a score for each class of the ultrasound image data based on the one or more feature maps; and generating the classification information based on the scores for each class; Utilizing the deep learning model to generate the diagnostic visualization information related to the ultrasound image data includes: calculating a probability of malignancy based on the classification information; Based on the one or more feature maps, a class activation map (Class generating an Activation Map; generating the diagnostic visualization information based on the malignancy probability, the class activation map, and the ultrasound image data; normalizing the class activation maps; binarizing the normalized class activation map using a preset threshold; determining a contributing region based on the binarized class activation map; and generating the diagnostic visualization information by taking the contributing regions into consideration; further comprising: method.

15. 1. A computer program stored on a computer-readable storage medium comprising operations for a computing device to diagnose an ultrasound image, the operations comprising: acquiring ultrasound image data; Utilizing a deep learning model to generate classification information related to the ultrasound image data; and utilizing the deep learning model to generate diagnostic visualization information related to the ultrasound image data; Including, The deep learning model A model in which a lesion area and its surrounding area are learned based on classification information of the training ultrasound image data without information related to segmentation of the training ultrasound image data, Each pixel of the ultrasound image data includes visualized information indicating a degree of contribution to generating the classification information, and the visualized information is obtained by backpropagating the classification information through a convolutional neural network including one or more convolution layers. A computer program stored on a computer-readable storage medium.

16. In claim 15, The operation of generating classification information related to the ultrasound image data by utilizing the deep learning model includes: utilizing the deep learning model to generate one or more feature maps based on the ultrasound image data; Utilizing the deep learning model to calculate a score for each class of the ultrasound image data based on the one or more feature maps; and generating the classification information based on the scores for each class; Including, A computer program stored on a computer-readable storage medium.

17. In claim 16, The operations of utilizing the deep learning model to generate diagnostic visualization information related to the ultrasound image data include: calculating a malignancy probability based on the classification information; generating a class activation map for each class based on the one or more feature maps; and generating the diagnostic visualization information based on the malignancy probability, the class activation map, and the ultrasound image data; Including, A computer program stored on a computer-readable storage medium.

18. In claim 17, The operation of generating a class activation map for each class based on the one or more feature maps is as follows: generating a class activation map for each of the classes based on the one or more feature maps and the weights of the deep learning model; Including, A computer program stored on a computer-readable storage medium.

19. In claim 18, The weights of the deep learning model are: The deep learning model includes weights learned in the process of generating the classification information. A computer program stored on a computer-readable storage medium.

20. 1. A computing device comprising: a processor including one or more cores; memory; Including, The processor: Acquire ultrasound image data; Utilizing a deep learning model to generate classification information related to the ultrasound image data; and Utilizing the deep learning model to generate diagnostic visualization information related to the ultrasound image data; The deep learning model A model in which a lesion area and its surrounding area are learned based on classification information of the training ultrasound image data without information related to segmentation of the training ultrasound image data, Each pixel of the ultrasound image data includes visualized information indicating a degree of contribution to generating the classification information, and the visualized information is obtained by backpropagating the classification information through a convolutional neural network including one or more convolution layers. Computing equipment.

21. In claim 20, Utilizing the deep learning model to generate classification information related to the ultrasound image data includes: utilizing the deep learning model to generate one or more feature maps based on the ultrasound image data; Utilizing the deep learning model to calculate a class score for the ultrasound image data based on the one or more feature maps; and generating the classification information based on the scores for each class; Including, Computing equipment.

22. 22. In claim 21, Utilizing the deep learning model to generate diagnostic visualization information related to the ultrasound image data includes: calculating a malignancy probability based on the classification information; generating a class activation map for each class based on the one or more feature maps; and generating the diagnostic visualization information based on the malignancy probability, the class activation map, and the ultrasound image data; Including, Computing equipment.

23. In claim 22, Generating a class activation map for each class based on the one or more feature maps includes: generating a class activation map for each of the classes based on the one or more feature maps and the weights of the deep learning model; Including, Computing equipment.

24. In claim 23, The weights of the deep learning model are: The deep learning model includes weights learned in the process of generating the classification information. Computing equipment.

Images