Object detection method and apparatus using deep learning architecture

Abstract

An object detection method using a deep learning architecture, comprising: a data collection step for collecting information on the location and characteristics of nodules from a dataset of lung CT scan images; a data preprocessing step for improving the quality of the lung CT scan images; a data augmentation step for increasing the number of data to improve the diversity of the lung CT scan images; an image segmentation step for dividing the lung CT scan images into various patch sizes to narrow the search space of the deep learning model; a training step for training the deep learning model using the preprocessed dataset of the lung CT scan images; and a detection and classification step for inputting arbitrary lung CT scan images into the trained deep learning model to detect lung nodules and classifying the detected images into three categories.

Description

Technology Field

[0001] The present invention relates to a deep learning architecture for detecting lung nodules from lung CT scan images. Background Technology

[0002] Recent advancements in biomedical imaging technology have presented tremendous opportunities for the medical and biomedical fields. However, collecting, measuring, and analyzing large volumes of health-related data, such as images, is a difficult and time-consuming task for medical professionals. In this regard, the application of artificial intelligence (including machine and deep learning systems) can aid in the early diagnosis of various infectious and cancerous diseases, such as lung cancer. Since lung cancer may not present distinct early symptoms, it is essential to develop computer-aided detection (CAD) systems that enable medical professionals to classify and detect lung nodules at an early stage.

[0003] Furthermore, identifying small lung nodules generally requires examining the entire lung and generating a large number of images for each patient. Examining these large images by removing unrelated tissues, such as blood vessels and bronchi, and identifying nodules across a large number of CT scans is a challenging task for medical professionals. The fatigue, distraction, and confusion experienced by medical staff due to the abnormal, large, and intimidating volume of images can easily lead to misdiagnosis. Therefore, there is a need to develop an effective automated computer system for lung nodule monitoring and screening. Prior art literature

[0004] (Patent Document 0001) KR 10-2020-0065751 A The problem to be solved

[0005] To solve the above problems, the present invention introduces a deep learning system for classifying lung nodules to analyze the problems of lung cancer diagnosis by classifying and detecting benign and malignant lung nodules in CT images. A new, state-of-the-art detection architecture including Faster-RCNN, YOLOv3, and SSD was used for training and detection purposes, and as a result, it can be confirmed that the False Positive Rate (FPR) decreased and accuracy improved. means of solving the problem

[0006] A lung nodule detection method according to an embodiment of the present invention for achieving the above objective may include, in an object detection method using a deep learning architecture, (A) a data collection step for collecting information on the location and features of nodules from a dataset of lung CT scan images; (B) a data preprocessing step for improving the quality of the lung CT scan images; (C) a data augmentation step for increasing the number of data to improve the diversity of the lung CT scan images; (D) an image segmentation step for dividing the lung CT scan images into various patch sizes to narrow the search space of the deep learning model; (E) a training step for training the deep learning model using the dataset of the preprocessed lung CT scan images; and (F) a detection and classification step for inputting an arbitrary lung CT scan image into the trained deep learning model to detect lung nodules and classifying the detected images into three categories.

[0007] In the data preprocessing step of the lung nodule detection method according to one embodiment of the present invention, the method may further include: a coordinate transformation step of loading the lung CT scan image and converting it into voxels; a data scaling step of readjusting the size of the lung CT scan image using the mean and standard deviation of the calculated data; and a normalization step of converting the pixel values ​​of the lung CT scan image into lung CT scan HU (Houndsfield Units) values.

[0008] In the data modulation step of the lung nodule detection method according to one embodiment of the present invention, noise enhancement, color enhancement, and spatial enhancement may be included.

[0009] In the learning step of the lung nodule detection method according to one embodiment of the present invention, a transfer learning approach using weights of the learned model is adopted, and the generalization performance of the deep learning model can be explored.

[0010] In the normalization step of the lung nodule detection method according to one embodiment of the present invention,

[0011]

[0012] Calculated based on,

[0013] Here, the CT scan HU (Houndsfield Units) value is an absorption coefficient for density analysis of computed tomography images, and can be characterized as a numerical representation of the different absorption rates of each component of the body for the contrast agent injected into the body.

[0014] In the coordinate transformation step of the lung nodule detection method according to one embodiment of the present invention,

[0015] Calculated based on,

[0016] Here, a voxel represents a 3D extension of a pixel (picture element), which is the smallest unit constituting a 2D image, and a single voxel can mean having a width and height equal to the pixel spacing.

[0017] In the transfer learning approach of the lung nodule detection method according to one embodiment of the present invention, deep learning architectures such as VGG-16, ResNet-50, and MobileNet can be utilized for classification tasks for learning.

[0018] In the learning step of the lung nodule detection method according to one embodiment of the present invention, once the classification work for learning is completed, Faster R-CNN, SSD, and YOLOv3 can be utilized as deep learning architectures for identifying the location of lung nodules in the lung CT scan image and identifying them into three stages: nodular, benign, and malignant.

[0019] In the lung nodule detection method according to one embodiment of the present invention, the Faster R-CNN may have a small network applied to subdivide the detected anchors, and the loss function determining the anchor lesion is

[0020]

[0021] Calculated based on,

[0022] Here, ε is a parameter that adjusts the weights between the classification loss and the regression loss, i is a single anchor, pi is the probability that the anchor obtained through classification is an object, ti is a vector of box adjustment values ​​obtained through bounding box regression, and pi* and ti* can represent ground truth labels to indicate the actual coordinates of the bounding box.

[0023] A lung nodule detection method according to one embodiment of the present invention may be characterized in that if the pi value is 1, the detected anchor may be associated with a positive class indicating that it is classified as a nodule, and conversely, if the pi value is 0, the detected anchor may be described with a negative class indicating that it is not classified as a nodule.

[0024] A lung nodule detection method according to one embodiment of the present invention, when using the SSD, the loss function is

[0025]

[0026] Calculated based on,

[0027] Here, x is the matching metric, c is the confidence level of various classes, l is the parameter of the predicted box, g is the parameter of the ground truth box, and N is the number of matched default boxes. is the stabilization weight for the loss function, is confidence loss, This can represent localization loss due to errors in the predicted bounding box location and size.

[0028] Among the lung nodule detection method according to one embodiment of the present invention, the above Is,

[0029]

[0030] Calculated based on, and the above Is,

[0031]

[0032] It is calculated based on,

[0033] Here, ε is an indicator of whether the i-th default box and the j-th ground truth box match for class p, c represents the confidence for different nodes, and It can be characterized by the fact that the value can be 1 and 0.

[0034] Among the lung nodule detection method according to one embodiment of the present invention, the above Is,

[0035]

[0036] Calculate based on,

[0037] Here, cx and cy are the center coordinates of the object of the class to be classified, w and h are the image width and height, indicates whether the i-th default box of class p matches the j-th ground truth box, and represents a smooth L1 loss function, and

[0038] Here, Regarding the values, when m is each cx, cy, w, h

[0039] It is calculated as a ratio of the image width, and

[0040] It is calculated as the ratio of the image height, and

[0041] The image width and image height , It can be characterized by applying a logarithmic scale as shown.

[0042] Loss function when using YOLOv3 in the lung nodule detection method according to one embodiment of the present invention Is

[0043]

[0044] Calculated based on,

[0045] Here, is a parameter that adjusts the weights between the classification loss, which helps distinguish background regions by determining the category to which the object in the prediction box belongs, and the box regression loss, which is applied only when the object is included in the prediction bounding box. is a parameter that varies depending on whether an object is detected in the bounding box above, assigning 1 or 0 depending on whether an object is detected, and if an object is detected, the actual data and predicted data It can be characterized by representing the position and size of the bounding box as a parameter.

[0046] Among the lung nodule detection method according to one embodiment of the present invention bounding box regression loss value silver

[0047]

[0048] Calculate based on,

[0049] Here, IOU (Interest of Union) represents the proportion of overlap among multiple objects classified as the same class, and is a parameter that adjusts the weights between the classification loss, which helps distinguish background regions by determining the category to which the object in the prediction box belongs, and the box regression loss, which is applied only when the object is included in the prediction bounding box. and is the actual and predicted data of the confidence value at the i-th grid cell, It can be characterized as a parameter representing a nodule detected in the i-th grid cell and the j-th bounding box.

[0050] A lung nodule detection device according to one embodiment of the present invention is an object detection device using a deep learning architecture, comprising: (A) a data collection unit that collects information on the location and features of nodules from a dataset of lung CT scan images; (B) a data preprocessing unit for improving the quality of the lung CT scan images; (C) a data augmentation unit that increases the number of data to improve the diversity of the lung CT scan images; (D) an image segmentation unit that divides the lung CT scan images into various patch sizes to narrow the search space of the deep learning model; (E) a learning unit that trains the deep learning model using the dataset of the preprocessed lung CT scan images; and (F) a detection and classification unit that inputs an arbitrary lung CT scan image into the trained deep learning model to detect lung nodules and classifies the detected images into three categories. Effects of the invention

[0051] Computer-aided detection (CAD) systems can be developed using systems such as deep learning to enable medical professionals to detect lung nodules early, and automated deep learning-based systems can be introduced to classify and detect lung nodules in CT scans. This can reduce the False Positive Rate (FPR) and improve the accuracy of the classification structure and the average detection accuracy. Brief explanation of the drawing

[0052] FIG. 1 is a flowchart of an object detection method using a deep learning architecture according to an embodiment of the present invention. Figure 2 shows the changes in areas of interest in biomedical research related to biomedical applications. FIG. 3 is an object detection device using a deep learning architecture according to an embodiment of the present invention. Figure 4a is a scan slice of the CT value histogram of HU for the sample CT. Figure 4b is a 2D axial slice for the sample CT. Figure 4c is a lung nodule image for the sample CT. Figure 5 shows the general architecture of the VGG-16. Figure 6 shows the general network architecture of Resnet50. Figure 7 shows the general architecture of MobileNet. Figure 8 shows the general network architecture of Faster-RCNN trained and tested for nodule detection. Figure 9 shows the general architecture of an SSD trained and tested for nodule detection. Figure 10 shows the general network architecture of YOLOv3 trained and tested for nodule detection. Figure 11 shows the loss of the detection architecture trained and tested on the LIDC-IDRI dataset according to the number of training iterations. Figure 12 shows the accuracy of the detection architecture trained and tested on the LIDC-IDRI dataset according to the number of training iterations. Figure 13 shows the classification and detection results of the deep learning architecture for benign lung nodules. Figure 14 shows the classification and detection results of the deep learning architecture for malignant lung nodules. Figure 15 shows the classification accuracy of the detection architecture trained and tested on the LIDC-IDRI dataset. Figure 16 shows the average TPR, accuracy, precision, and recall of a deep learning-based detector. Specific details for implementing the invention

[0053] The objects, specific advantages, and novel features of the present invention will become more apparent from the following detailed description and preferred embodiments in conjunction with the accompanying drawings. In this case, identical components in each drawing are denoted by the same reference numerals whenever possible. Additionally, detailed descriptions of already known functions and / or configurations are omitted.

[0054] The contents disclosed below focus on the parts necessary for understanding the operation according to various embodiments, and omit descriptions of elements that may obscure the gist of the explanation. Additionally, some components of the drawings may be exaggerated, omitted, or depicted schematically. The size of each component does not entirely reflect its actual size, and therefore, the contents described herein are not limited by the relative sizes or spacing of the components depicted in each drawing.

[0055] In describing the embodiments of the present invention, detailed descriptions of known technologies related to the present invention are omitted if it is determined that such detailed descriptions may unnecessarily obscure the essence of the invention. Furthermore, the terms described below are defined in consideration of their functions within the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe the embodiments of the present invention and should not be limiting in any way. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as "include" or "comprise" are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described.

[0056] Additionally, terms such as first, second, etc., may be used to describe various components, but said components are not limited by said terms, and said terms are used only for the purpose of distinguishing one component from another.

[0057] Hereinafter, with reference to the attached drawings, an object detection method and apparatus using a deep learning architecture according to an embodiment of the present invention will be described.

[0058] A flowchart of the object detection method using the deep learning architecture of the present invention is shown in FIG. 1.

[0059] The object detection method using the deep learning architecture of the present invention has the following key steps: (A) a data collection step (step S102) for collecting information on the location and features of nodules from a dataset of lung CT scan images; (B) a data preprocessing step (step S106) for improving the quality of the lung CT scan images; (C) a data augmentation step (step S108) for increasing the number of data to improve the diversity of the lung CT scan images; (D) an image segmentation step (step S110) for dividing the lung CT scan images into various patch sizes to narrow the search space of the deep learning model; (E) a training step (step S112) for training the deep learning model using the preprocessed dataset of the lung CT scan images; and (F) a detection and classification step (step S114) for inputting arbitrary lung CT scan images into the trained deep learning model to detect lung nodules and classifying the detected images into three categories.

[0060] FIG. 3 is a diagram showing the configuration of an object detection device using a deep learning architecture according to one embodiment of the present invention.

[0061] Referring to FIG. 3, an object detection device (300) using a deep learning architecture according to one embodiment of the present invention may include: (A) a data collection unit (310) for collecting information on the location and features of nodules from a dataset of lung CT scan images; (B) a data preprocessing unit (320) for improving the quality of the lung CT scan images; (C) a data augmentation unit (330) for increasing the number of data to improve the diversity of the lung CT scan images; (D) an image segmentation unit (340) for dividing the lung CT scan images into various patch sizes to narrow the search space of the deep learning model; (E) a learning unit (350) for training the deep learning model using the dataset of the preprocessed lung CT scan images; and (F) a detection and classification unit (360) for inputting an arbitrary lung CT scan image into the trained deep learning model to detect lung nodules and classifying the detected images into three categories.

[0063] < Data Collection Phase >

[0064] Referring to FIG. 1, in step S102, the data collection unit (310) can collect information about the location and characteristics of the nodules from a data set of lung CT scan images.

[0065] According to one embodiment of the present invention, a dataset collected from the LIDC-IDRI database can be applied. A publicly available LIDC-IDRI dataset consisting of a total of 1,018 lung cancer cases can be utilized.

[0066] Referring to Fig. 1, in step S104, information regarding CT scan images and the location and size of each nodule can be read from an XML file. For every case, an XML file containing a clinical chest CT scan and reports from four experienced thoracic radiologists may be included.

[0067] The above XML file may include various features regarding the nodules. In particular, the edge map may indicate the nodule location, that is, the pixel location of the nodule boundary. These results can be obtained using a method that utilizes the phase difference of the image.

[0068] The first stage, also commonly referred to as the blinded reading stage, allows another radiologist to individually review all CT scans. Lesions can be classified into one of three grades: non-nodular ≥ 3 mm, nodular ≥ 3 mm, and nodular < 3 mm.

[0069] In the second stage, known as the non-blinded reading stage, all radiologists can provide a final conclusion by individually examining their own marks and other annotated marks. The goal is to effectively identify all lung nodules in every CT scan without restricted consent.

[0070] The above nodules are annotated nodules graded from 1 to 5, each representing the extremes of benign and malignant. Grade 0 indicates an unusable analysis and can be ignored.

[0071] The above nodule can be used by examining lung nodules diagnosed by at least 3 experts and estimating the median of the annotation class for the diagnosed nodule, and if the median is less than 3, it can be used as benign, and if it is greater than 3, it can be used as malignant.

[0073] < Preprocessing Step >

[0074] Referring to FIG. 1, in step S106, the data preprocessing unit (320) can preprocess data to improve the quality of the lung CT scan image.

[0075] Since the above dataset contains various inappropriate images, a filtering step can be performed to remove irrelevant images that balance the dataset for nodular and non-nodular image samples.

[0076] Coordinate transformation step

[0077] According to one embodiment of the present invention, the position of the nodule can be converted into voxel coordinates.

[0078] The location of each of the above nodules is determined in the world coordinate system, but voxel coordinates can be used to locate and identify nodules in a CT scan.

[0079] Additionally, a function can be applied that takes world coordinates as input and converts all locations into the required voxel coordinates provided with the dataset as follows.

[0080]

[0081] Here, a voxel represents a 3D extension of a pixel (picture element), which is the smallest unit constituting a 2D image, and a single voxel can mean having a width and height equal to the pixel spacing.

[0083] Data scaling step

[0084] According to one embodiment of the present invention, the collected image can be rescaled using the mean and standard deviation.

[0086] Normalization step

[0087] According to one embodiment of the present invention, data extracted from a CT scan image can be obtained from an X-ray beam representing different body parts with multiple frequency bands to identify various tissues and organs.

[0088] The above CT scan image data increases with higher density or frequency of the substance and can be measured in HU units. The range of lung CT values ​​can be from -600 HU to -450 HU. Blood can be within the range of 13 HU to 32 HU, and body fat can be within the range of -20 HU to -10 HU.

[0089] In addition, cases containing air, bone, and water can also be considered raw CT images. For example, the CT value of human bone is approximately 1,000 HU, air is approximately -1,000 HU, and water is approximately 0 HU.

[0090] Meanwhile, since the range of CT scan HU values ​​is from 0 to 3000, a problem of gradient vanishing and exploding may occur for the deep learning architecture. This can occur when gradients are applied to update weights upon reaching the initial layer of the architecture of the present invention. Consequently, the weights of the initial layer are not changed, so it may not be learned efficiently.

[0091] According to one embodiment of the present invention, the pixel values ​​of data extracted from a CT scan image can be converted into HU (Hounsfield units) and normalized.

[0092] As shown in Equation 2, normalizing the input values ​​makes it easier to determine the number of iterations, thus resolving the difficulty and accelerating the learning process. First, you can input a minimum HUmin of 500 and a maximum HUmax of 1000.

[0093]

[0094] Here, the CT scan HU (Houndsfield Units) value is an absorption coefficient for density analysis of computed tomography images, which can quantify the different absorption rates of the contrast agent injected into the body for each component of the body.

[0095] Figure 4a shows a scan slice of the CT value histogram of HU for the sample CT.

[0096] As shown in Fig. 4a, a clear image can be obtained with a value of ≤ 1,000HU > 500HU, normalized to a range of 0-1, and unnecessary or irrelevant sample images can be filtered out using a threshold.

[0097] CT scans with clear lung nodules, as shown in Figures 4b and 4c, can be obtained.

[0098] Figure 4b shows a 2D axial slice of the sample CT.

[0099] Figure 4c shows an image of a lung nodule for the sample CT.

[0100] Afterward, the selected slice is preprocessed, and the developed system can read an XML file containing information about each nodule in the CT scan. As previously discussed, it may have many characteristics, such as a malignancy score ranging from 0 to 4.

[0101] The edge map represents the nodule location in pixels and can generally represent boundary pixel information of the nodule.

[0102] Based on mathematical formula 3, the centroid of the nodule can be calculated.

[0103]

[0104] Here, represents the centroid of the nodule and can be independently calculated as the sum of all positions i in x, y, and z according to the number of positions for each n.

[0106] < Data Modification Step >

[0107] Referring to FIG. 1, in step S108, the data augmentation unit (330) can increase the number of data to improve the diversity of the lung CT scan images.

[0108] According to one embodiment of the present invention, data growth after preprocessing can be performed to improve the heterogeneity of the data set and the system performance.

[0109] According to one embodiment of the present invention, a wide range of data enlargement, such as spatial data enlargement including mirroring, rotation, scaling, color enlargement (contrast, brightness, and gamma adjustment), and noise enlargement, can be performed.

[0111] < Image Segmentation Step >

[0112] Referring to FIG. 1, in step S110, the image segmentation unit (340) can divide the lung CT scan image into various patch sizes to narrow the search space of the deep learning model.

[0113] According to one embodiment of the present invention, since the search space is large for a deep learning architecture, an input CT scan can be cropped and 64 × 64 patches can be extracted from a depth of 512 × 512 ×. The cropped image patches can be divided into training and test samples.

[0115] < Learning Stage >

[0116] Referring to FIG. 1, in step S112, the learning unit (350) can train the deep learning model using the data set of the preprocessed lung CT scan images.

[0117] The above dataset can be randomly split into training and test sets in a ratio of 70% and 30%.

[0118] Referring to Fig. 1, three classification structures can be used to classify lung nodule images from non-nodule images (benign and malignant). These architectures can also be applied to image feature extraction.

[0119] VGG-16

[0120] Figure 5 shows the general architecture of VGG-16.

[0121] According to one embodiment of the present invention, VGG-16 can be used as a basic architecture for classification. It is a simple and commonly used convolutional neural network trained on the ImageNet dataset, and this architecture can be very deep with 16 layers with 138 million parameters.

[0122] According to one embodiment of the present invention, a fixed-size 224 × 224 image can be used during training, and the average value estimated for the training data set can be subtracted from every pixel.

[0123] The input image can be provided to a convolution layer block with a 3×3 filter.

[0124] Except for the 1 × 1 convolution filter, which is a configuration defined as a linear transformation of the input channel, more nonlinearity can be achieved using several parameters.

[0125] The input to the convolution layer is the stride, and the spatial padding can be made to 1 pixel for a 3×3 convolution layer.

[0126] It is possible to save the spatial rendering after the convolution layer.

[0127] You can have 5 max pooling layers that support convolution layers and aid in spatial pooling. Max pooling can be performed with a stride of 2 in a 2 × 2 pixel window. 3 fully connected layers can support a cluster of convolution layers.

[0128] Referring to Fig. 5, each layer can have a different depth.

[0129] The very last layer can be a softmax layer.

[0131] Resnet-50

[0132] Figure 6 shows the general network architecture of Resnet50.

[0133] According to one embodiment of the present invention, the second convolutional neural network architecture used for classification purposes may be Resnet-50.

[0134] There can be 50 deep layers with 26 million parameters.

[0135] The architecture can learn features from residuals and provide data through convolution layers using skip connections.

[0136] Resnet can build extensive neural networks by combining the input of the n-th layer with the (n+x)-th layer to accumulate additional layers. The network may have 48 convolutional layers, one max pooling layer, and one mean pooling layer. The kernel size of the convolutional layers is 7×7, and there may be 64 different kernels and max pooling with a stride size of 2. Subsequent convolutional blocks may have 3×3.64 kernels supported by 1×1,256 kernels.

[0137] In addition, the layers can be repeated a total of 3 times to ultimately provide 9 layers.

[0139] Mobile Net

[0140] Figure 7 shows the general architecture of MobileNet.

[0141] According to one embodiment of the present invention, there are two global hyperparameters, resolution and width multiplier, so that a trade-off between efficiency or latency may be possible depending on the conditions.

[0142] Basically, to generate smaller patterns, separable convolutions called depth-dependent layers can be applied instead of the existing layers applied to the previous network.

[0143] All depth-separable convolution layers can be composed of point-wise and depth-wise convolution layers. If these convolutions are evaluated as separate layers, there can be a total of 28 layers.

[0144] A general architecture can consist of 4.2 million parameters that can be modified by adjusting hyperparameters (e.g., width multiplier).

[0146] According to one embodiment of the present invention, Faster R-CNN, SSD, and YOLOv3 can be executed for nodule localization and identification of benign and malignant.

[0147] Faster-RCNN

[0148] Referring to Figure 2, Faster R-CNN can be executed for detection purposes.

[0149] Figure 8 shows the general network architecture of Faster-RCNN trained and tested for nodule detection.

[0150] According to one embodiment of the present invention, the first step may generate region anchors for nodular lesions using a Region Proposal Network (RPN), and the second step may perform classification (e.g., benign and malignant) by applying the detected anchor region information.

[0151] The output of the detector model may be a rectangular detection bounding box containing coordinate, height, and width information along with confidence scores and class label values.

[0152] Referring to Fig. 8, the entire method presented can, in principle, be composed of three steps.

[0153] Features of the input image are extracted through a convolution layer and can be further utilized to generate a feature map.

[0154] According to one embodiment of the present invention, a sliding window approach is used to generate a fixed lesion or area bounding box and can be further subdivided to specify nodules.

[0155] In addition, a small network is applied to subdivide the detected anchors, and a loss function for determining the most desired anchor lesion can be estimated as follows.

[0156] Based on mathematical equation 4, the loss function can be calculated.

[0157]

[0158] Here, ε is a parameter that adjusts the weights between the classification loss and the regression loss, i is a single anchor, pi is the probability that the anchor obtained through classification is an object, ti is a vector of box adjustment values ​​obtained through bounding box regression, and pi* and ti* can represent ground truth labels to indicate the actual coordinates of the bounding box.

[0159] The Region of Interest (ROI) can be obtained through a convolutional feature map using the trained RPN. Once the feature map is extracted using the ROI pool layer, a loss function including both classification and regression functions can be determined.

[0161] SSD

[0162] Figure 9 shows the general architecture of an SSD trained and tested for nodule detection.

[0163] According to one embodiment of the present invention, this may include a 3 × 3 depth-wise convolution layer supported by a point-wise convolution layer of 1 × 1 stride. The detector model can identify and locate specific nodules by extracting nodule features and then applying a convolution filter.

[0164] Currently, different feature maps can be applied to classification and localization regression.

[0165] Referring to Fig. 9, a basic set of bounding boxes can be defined for all blocks of a feature map. For all feature maps, four sizes of expected bounding boxes can be provided.

[0166] For all locations detected in the feature map, the number of k bounding boxes with various aspect ratios and sizes can be obtained.

[0167] The loss function can be determined based on mathematical equation 5.

[0168]

[0169] Here, x is the matching metric, c is the confidence level of various classes, l is the parameter of the predicted box, g is the parameter of the ground truth box, and N is the number of matched default boxes. is the stabilization weight for the loss function, is confidence loss, This can represent localization loss due to errors in the predicted bounding box location and size.

[0170] Based on mathematical formula 6 can calculate.

[0171]

[0172] Here, ε is an indicator of whether the i-th default box and the j-th ground truth box match for class p, c represents the confidence for different nodes, and It can be characterized by the fact that the value can be 1 and 0.

[0173] Based on mathematical formula 7 can calculate.

[0174]

[0175] Here, ε is an indicator of whether the i-th default box and the j-th ground truth box match for class p, c represents the confidence for different nodes, and The value can be 1 and 0.

[0176] Based on mathematical formula 8, can calculate.

[0177]

[0178] Calculate based on,

[0179] Here, cx and cy are the center coordinates of the object of the class to be classified, w and h are the image width and height, indicates whether the i-th default box of class p matches the j-th ground truth box, and represents a smooth L1 loss function, and

[0180] Here, Regarding the values, when m is each cx, cy, w, h

[0181] It is calculated as a ratio of the image width, and

[0182] It is calculated as the ratio of the image height, and

[0183] The image width and image height , It can be characterized by applying a logarithmic scale as shown.

[0185] YOLOv3

[0186] Figure 10 shows the general network architecture of YOLOv3 trained and tested for nodule detection.

[0187] Referring to Figure 10, it can be trained using a lung nodule dataset.

[0188] As an SSD, YOLOv3 can also be a one-stage architecture that predicts the same bounding box and nodule class probabilities for the entire input CT scan.

[0189] Convolutional layers are used to extract features from input images, and fully connected layers (FC) can be used to identify class probabilities and predictions.

[0190] The general architecture can be divided into regions defined by grid cells. For nodule detection, it relates to class probabilities and bounding box predictions. The center of a nodule detected in an image may or may not be inside the cell as predicted by these grid cells. Relevant regions or lesions can be determined for lung nodule detection. Confidence values ​​can be used to obtain the detected bounding box.

[0191] A loss function combining two features, including regression and classification, can be defined for the identified lung nodule bounding boxes.

[0192] Loss function based on mathematical equation 9 can be calculated.

[0193]

[0194] Calculated based on,

[0195] Here, is a parameter that adjusts the weights between the classification loss, which helps distinguish background regions by determining the category to which the object in the prediction box belongs, and the box regression loss, which is applied only when the object is included in the prediction bounding box. is a parameter that varies depending on whether an object is detected in the bounding box above, assigning 1 or 0 depending on whether an object is detected, and if an object is detected, the actual data and predicted data It can be characterized by representing the position and size of the bounding box as a parameter.

[0196] In mathematical formula 10 bounding box regression loss value can calculate.

[0197]

[0198] Here, IOU (Interest of Union) represents the proportion of overlap among multiple objects classified as the same class, and is a parameter that adjusts the weights between the classification loss, which helps distinguish background regions by determining the category to which the object in the prediction box belongs, and the box regression loss, which is applied only when the object is included in the prediction bounding box. and is the actual and predicted data of the confidence value at the i-th grid cell, It can be characterized as a parameter representing a nodule detected in the i-th grid cell and the j-th bounding box.

[0200] Perform test

[0201] According to one embodiment of the present invention, NVIDIA DIGITS (Deep Learning GPU Training System) is used, a network is defined, data augmentation is implemented, and training and testing can be performed using an Intel i5 system equipped with an NVIDIA GeForce GTX 1080 GPU.

[0202] The learning time for all architectures can be about 6 hours.

[0203] Figures 11 and 12 show the results of the learning and test observations of the architecture.

[0204] Training can be performed for 100 epochs.

[0205] In Figure 11, it can be seen that the loss decreases at the 10th epoch during training and testing, and the training and testing losses of Faster-RCNN are 0.6% and 0.5%, SSD is 0.45% and 0.4%, and YOLO is 0.45% and 0.3%, respectively.

[0206] In Figure 11, it can be seen that the training and test accuracy improved after the initial 20th epoch, and the training and test accuracy of Faster-RCNN is 0.86 and 0.84, respectively, SSD is 0.87 and 0.9, and YOLOv3 is 0.92 and 0.94, respectively.

[0208] < Detection and Classification Step >

[0209] Referring to FIG. 1, in step S114, the detection and classification unit (360) inputs an arbitrary lung CT scan image into a trained deep learning model to detect lung nodules and can classify the detected image into three categories.

[0210] Referring to Fig. 1, a deep learning architecture can be used to detect and locate lung nodules in CT scan images after classification.

[0211] In Figures 13 and 14, the classification and detection results of the deep learning-based automation system can be seen.

[0212] In Figure 13, the classification results of benign lung nodules can be seen. It can be seen that various CT scan images with different nodule sizes are accurately classified.

[0213] In Figure 13, the detection results can be observed. The original test image is displayed in the first column, and the results of Faster-RCNN are displayed in the second column. It can be seen that the model produces good results by detecting benign nodules of various sizes in CT images. Similarly, the results of the SSD and YOLOv3 detectors are displayed in the next two columns. It can be seen that they produce good results by detecting more precise bounding boxes for benign lung nodules of various sizes compared to Faster-RCNN.

[0214] In Fig. 14, the results of the malignant lung CT scan and the classification and detection of lung nodules can be seen.

[0215] As shown in Figure 13, the deep learning architecture provides excellent results in classifying lung CT scans from benign to malignant. However, although it is difficult because most nodule features are similar to those of malignant nodules, the deep learning architecture can automatically select stronger features and classify CT scans of malignant and benign nodules.

[0216] Nodules on a CT scan may also be in the yellow bounding box.

[0217] In addition, the results show that YOLOv3 and SSD have superior performance compared to Faster-RCNN.

[0219] Performance evaluation

[0220] It can be evaluated using various parameters such as True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN).

[0221] Precision, Recall, Accuracy, TPR (True Positive Rate), and FPR (False Positive Rate) can be calculated using these parameters.

[0222] Standard deviation and mean can be used to determine average precision, recall, and accuracy.

[0223] Figure 15 shows the classification accuracy of various deep learning-based architectures. It can be seen that the deep learning architecture provides good results for non-nodule, benign, and malignant CT scan images.

[0224] The classification accuracy of VGG-16 is 0.93, 0.92, and 0.93, and Resnet50 is 0.94, 0.94, and 0.95, while Mobilenet is 0.95, 0.95, and 0.95 for non-nodular, benign, and malignant images, respectively.

[0225] In Figure 16, the performance of the detector models can be verified using various parameters. It can be seen that all models provide good results for both benign and malignant lung nodules. The TPR, accuracy, precision, and recall for three different architectures are shown in the figure. The TPR, Accuracy, Precision, and Recall values ​​for Faster-RCNN were 0.91, 0.93, 0.89, and 0.70, and SSD achieved 0.92, 0.94, 0.9, and 0.75, respectively, while YOLOv3 produced 0.93, 0.94, 0.91, and 0.89.

[0227]

[0228] In Table 1, you can see the overall performance results of detection models with different classification and basic architectures.

[0229] It can be confirmed that the accuracy of Faster-RCNN(VGG-16) is 0.90, Faster-RCNN(Resnet50) is 0.91, Faster-RCNN(Mobilenet) is 0.91, SSD(VGG 16) is 0.92, SSD(Resnet50) is 0.925, and SSD(Mobilenet) is 0.94.

[0230] The accuracy may be more accurate compared to other models, namely YOLOv3(VGG-16) 0.93, YOLOv3(Resnet50) 0.935, and YOLOv3(Mobilenet) 0.94.

[0231] For various deep learning models, FPR can also decrease due to a filtering step between 0.2 and 0.06.

[0232] The TPR values ​​of Faster-RCNN can be 0.91, 0.92, and 0.92 in VGG 16, Resnet, and Mobilenet.

[0233] SSD and YOLOv3 models can achieve TPRs of 0.92, 0.93, 0.93, 0.93, 0.93, 0.93, and 0.93, 0.935, and 0.94, respectively, in VGG-16, Resnet, and Mobilenet.

[0235] In the present invention, a deep learning-based system can be utilized to perform classification and detection of benign and malignant tumors in CT scans to analyze lung cancer or the causes of lung cancer.

[0236] The developed system can investigate various basic architectures for lung nodule classification such as VGG-16, Resnet 101, and Mobilenetv3.

[0237] In addition, new deep learning detectors including Faster-RCNN, YOLOv3, and SSD can be used for detection purposes.

[0238] Training can be performed using the publicly available benchmark LIDC-IDRI dataset. To improve system performance, necessary preprocessing is performed, and a filtering step may be introduced to remove unnecessary samples / images from the dataset.

[0239] As a result, all deep learning models achieved excellent classification and detection accuracy. Test results showed that FPR decreased and accuracy improved, with an average accuracy of classification structures of 0.92–0.95 and an average detection accuracy of 0.93–0.94. Explanation of the symbols

[0241] 300: Object detection device using deep learning architecture 310: Lung CT image data acquisition unit 320: Data preprocessing unit 330: Data modulation unit 340: Image segmentation unit 350: Model training unit 360: Lung nodule detection and classification unit

Claims

Claim 1 (A) a data collection step in which information regarding the location and characteristics of nodules is collected from a dataset of lung CT scan images in a data collection unit; (B) a data preprocessing step in which the quality of the lung CT scan images is improved in a data preprocessing unit; (C) a data augmentation step in which the number of data is increased in a data modulation unit to improve the diversity of the lung CT scan images; (D) an image segmentation step in which the lung CT scan images are divided into various patch sizes in an image segmentation unit to narrow the search space of the deep learning model; (E) a learning step in which the deep learning model is trained using the dataset of the lung CT scan images preprocessed in a learning unit; and (F) a detection and classification step in which an arbitrary lung CT scan image is input into the trained deep learning model to detect lung nodules and classify the detected images into three categories; wherein, in the learning step, when the classification task for learning is completed, the location of lung nodules in the lung CT scan images is identified, and when an SSD is used as a deep learning architecture for identification into three stages of nodules, benign, and malignant, the loss function is It is calculated based on, where x is the matching metric, c is the confidence level of various classes, l is the parameter of the predicted box, g is the parameter of the ground truth box, and N is the number of matched default boxes. is the stabilization weight for the loss function, is confidence loss, An object detection method using a deep learning architecture, which represents localization loss due to errors in predicted bounding box location and size. Claim 2 The object detection method using a deep learning architecture according to claim 1, further comprising, in the data preprocessing step, a coordinate transformation step of loading the lung CT scan image and converting it into voxels; a data scaling step of readjusting the size of the lung CT scan image using the mean and standard deviation of the calculated data; and a normalization step of converting the pixel values ​​of the lung CT scan image into lung CT scan HU (Houndsfield Units) values. Claim 3 An object detection method using a deep learning architecture according to claim 1, wherein the data modulation step includes noise enhancement, color enhancement, and spatial enhancement. Claim 4 An object detection method using a deep learning architecture according to claim 1, characterized in that, in the learning step, a transfer learning approach using the weights of the learned model is adopted and the generalization performance of the deep learning model is explored. Claim 5 In paragraph 2, in the above normalization step, An object detection method using a deep learning architecture, calculated based on, wherein the CT scan HU (Houndsfield Units) value is an absorption coefficient for density analysis of a computed tomography image, characterized by quantifying the difference in absorption rates for each component of the body for a contrast agent injected into the body. Claim 6 delete Claim 7 An object detection method using a deep learning architecture according to claim 4, characterized in that the transfer learning approach utilizes deep learning architectures of VGG-16, ResNet-50, and MobileNet as classification tasks for learning. Claim 8 delete Claim 9 delete Claim 10 delete Claim 11 delete Claim 12 In paragraph 1, the above Is, Calculated based on, and the above Is, It is calculated based on, where, ε is an indicator of whether the i-th default box and the j-th ground truth box match for class p, c represents the confidence for different nodes, and An object detection method using a deep learning architecture characterized by the value being 1 and 0. Claim 13 In paragraph 1, the above Is, Calculated based on, where cx and cy are the center coordinates of the object of the class to be classified, w, h are the image width and height, indicates whether the i-th default box of class p matches the j-th ground truth box, and represents the smooth L1 loss function, where, Regarding the values, when m is each cx, cy, w, h It is calculated as a ratio of the image width, and It is calculated as the ratio of image height, and the image width and image height , An object detection method using a deep learning architecture characterized by applying a log scale as shown above. Claim 14 (A) A data collection step in which information regarding the location and characteristics of nodules is collected from a dataset of lung CT scan images in a data collection unit; (B) A data preprocessing step in which the quality of the lung CT scan images is improved in a data preprocessing unit; (C) A data augmentation step in which the number of data is increased in a data modulation unit to improve the diversity of the lung CT scan images; (D) An image segmentation step in which the lung CT scan images are divided into various patch sizes in an image segmentation unit to narrow the search space of the deep learning model; (E) A training step in which the deep learning model is trained using the dataset of the lung CT scan images preprocessed in a training unit; and (F) a detection and classification step in which an arbitrary lung CT scan image is input into a deep learning model trained in the detection and classification unit to detect lung nodules and classify the detected images into three categories; wherein in the training step, when the classification task for training is completed, the location of the lung nodules in the lung CT scan image is identified, and a loss function when using YOLOv3 as a deep learning architecture for identification into three stages of nodular, benign, and malignant Is It is calculated based on, and here, is a parameter that adjusts the weights between the classification loss, which helps distinguish background regions by determining the category to which the object in the prediction box belongs, and the box regression loss, which is applied only when the object is included in the prediction bounding box. is a parameter that varies depending on whether an object is detected in the bounding box above, assigning 1 or 0 depending on whether an object is detected, and if an object is detected, the actual data and predicted data An object detection method using a deep learning architecture characterized by representing the position and size of a bounding box as a parameter. Claim 15 In paragraph 1, bounding box regression loss value silver It is calculated based on, where IOU (Interest of Union) represents the proportion of overlap between multiple objects classified as the same class, and is a parameter that adjusts the weights between the classification loss, which helps distinguish background regions by determining the category to which the object in the prediction box belongs, and the box regression loss, which is applied only when the object is included in the prediction bounding box. and is the actual and predicted data of the confidence value at the i-th grid cell, An object detection method using a deep learning architecture, characterized in that is a parameter representing a nodule detected in the i-th grid cell and the j-th bounding box. Claim 16 (A) a data collection unit that collects information on the location and characteristics of nodules from a dataset of lung CT scan images; (B) a data preprocessing unit for improving the quality of the lung CT scan images; (C) a data augmentation unit that increases the number of data to improve the diversity of the lung CT scan images; (D) an image segmentation unit that divides the lung CT scan images into various patch sizes to narrow the search space of the deep learning model; (E) a learning unit that trains the deep learning model using the preprocessed dataset of the lung CT scan images; and (F) a detection and classification unit that inputs an arbitrary lung CT scan image into the trained deep learning model to detect lung nodules and classifies the detected images into three categories; wherein, when the classification task for learning is completed, the learning unit identifies the location of lung nodules in the lung CT scan images and, when using SSD as a deep learning architecture for identification into three stages of nodular, benign, and malignant, the loss function is It is calculated based on, where x is the matching metric, c is the confidence level of various classes, l is the parameter of the predicted box, g is the parameter of the ground truth box, and N is the number of matched default boxes. is the stabilization weight for the loss function, is confidence loss, An object detection device using a deep learning architecture, representing localization loss due to errors in predicted bounding box location and size.

Images