Neural network-based medical image processing apparatus and method

Abstract

A neural network-based medical image processing apparatus according to the present invention, which identifies a small intestine region from a medical image acquired by a capsule endoscope, comprises:

Description

Neural network-based medical image processing device and method

[0001] The present invention relates to a neural network-based medical image processing device and method, and more particularly, to a neural network-based medical image processing device and method for distinguishing organs from internal medical images acquired by a capsule endoscope.

[0002] Capsule endoscopy is commonly used to diagnose various small bowel diseases. Diagnosing small bowel diseases requires organ segmentation to identify the small bowel region from the medical images acquired through capsule endoscopy.

[0003] This technology for organ segmentation in medical images has already been disclosed in "Korean Patent Publication No. 10-2237198, Artificial Intelligence-Based Medical Image Interpretation Service System (April 1, 2021)". This registered invention performs organ segmentation in medical images containing multiple organs based on a classification model.

[0004] However, capsule endoscopy captures dozens of still images per second for internal medical imaging, operating within the body for 8 to 12 hours. During this time, over 50,000 still images are generated for gastrointestinal imaging. Therefore, clinicians must interpret numerous still images to diagnose small intestinal diseases.

[0005] However, interpreting numerous still images is a tedious task and can lead to errors in diagnostic results. Therefore, research and development are being conducted to segment small intestine regions using artificial intelligence and neural network algorithms. However, most conventional techniques target organ segmentation from a single image, making it extremely difficult to identify frames where organs change within the image. Despite this, images are still used in clinical settings for diagnosis. Therefore, segmenting small intestine regions based on a single image has limited utility in the diagnosis of small intestine diseases.

[0006] The purpose of the present invention is to provide a neural network-based medical image processing device and method capable of selectively acquiring images of a small intestine region by predicting frames in which an organ changes in an image.

[0007] A neural network-based medical image processing device according to the present invention is a neural network-based medical image processing device that distinguishes a small intestine region from a medical image acquired by a capsule endoscope, the device comprising: a memory equipped with an organ classification algorithm that performs organ classification for the medical image; and a processor that applies the medical image to the organ classification algorithm so that the small intestine region is distinguished; and the organ classification algorithm includes a convolutional neural network algorithm that distinguishes organs included in the medical image into a stomach, a small intestine, and a large intestine so that the small intestine region is distinguished; and a temporal filtering algorithm that is linked to the convolutional neural network algorithm and reduces images misclassified by the convolutional neural network algorithm.

[0008] The above convolutional neural network algorithm may include a ResNet model trained with 2D images, and the above temporal filtering algorithm may include a hybrid temporal filter composed of a Savitzky-Golay filter and a median filter.

[0009] In the learning of the above convolutional neural network algorithm, a plurality of 2D images are read to perform 3-class labeling into the stomach, small intestine, and large intestine, a learning set is created with the labeled plurality of 2D images, and the convolutional neural network algorithm is trained based on the learning set so that the convolutional neural network algorithm can predict the small intestine region.

[0010] In the learning of the above convolutional neural network algorithm, after the labeling, the labeled multiple 2D images are randomly selected and classified into the learning set, verification set, and test set, and each set can be classified to include normal data and abnormal data together.

[0011] In the learning of the above convolutional neural network algorithm, the 2D image ratio of the stomach, small intestine, and large intestine can be adjusted to 1:2:1 so that the imbalance between the organs is adjusted.

[0012] In the above 2D image ratio adjustment, normal and abnormal stomach images can be enlarged by applying horizontal and vertical flips, and normal and abnormal small intestine images and normal and abnormal large intestine images can be downsampled at a set ratio.

[0013] In the above downsampling, the normal small intestine and normal large intestine images can be downsampled at ratios of 2 / 3 and 1 / 3, respectively, and the abnormal small intestine and abnormal large intestine images can be downsampled at ratios of 3 / 4 and 3 / 7, respectively.

[0014] In learning the above convolutional neural network algorithm, the convolutional neural network algorithm can be verified based on the verification set after training the convolutional neural network algorithm.

[0015] The above neural network-based medical image processing device can test the convolutional neural network algorithm and the temporal filtering algorithm based on the test set after learning the convolutional neural network algorithm.

[0016] In the application of the above temporal filtering algorithm, binary classification can be performed.

[0017] In the above binary classification, the small intestine class obtained by the above convolutional neural network algorithm is applied to the Savitzky-Golay filter and the median filter, the results of the Savitzky-Golay filter and the median filter are added and divided, and then a value greater than 1 is mapped to 1 and a value less than 0 is mapped to 0, thereby distinguishing the frame of the small intestine from the frame of the stomach and the large intestine.

[0018] The above organ segmentation algorithm can segment the small intestine region by predicting frames in which the organ changes from the above medical image.

[0019] The above-mentioned organ classification algorithm applies the small intestine class obtained by the above-mentioned convolutional neural network algorithm to the above-mentioned temporal filtering algorithm to designate the temporally filtered probability as a threshold value, and can distinguish the frame of the small intestine from the frame of the stomach and large intestine based on the threshold value.

[0020] The above threshold value may be 0.87.

[0021] The above temporal filtering algorithm can correct the class probability of a frame by using the long-term probability derived from adjacent frames by the above convolutional neural network algorithm.

[0022] Meanwhile, a neural network-based medical image processing method according to the present invention is a neural network-based medical image processing method for distinguishing a small intestine region from a medical image acquired by a capsule endoscope, the method including a step of inputting the medical image into an organ classification algorithm and a step of distinguishing the small intestine region from the medical image by the organ classification algorithm, wherein the organ classification algorithm may include a convolutional neural network algorithm for distinguishing the organs included in the medical image into a stomach, a small intestine, and a large intestine so that the small intestine region is distinguished, and a temporal filtering algorithm linked to the convolutional neural network algorithm for reducing images misclassified by the convolutional neural network algorithm.

[0023] The neural network-based medical image processing device and method according to the present invention can significantly shorten the reading time of a clinician by automatically acquiring only images of the small intestine through organ segmentation.

[0024] In addition, the neural network-based medical image processing device and method according to the present invention has a great advantage when combined with other technologies for diagnosing lesions in the small intestine by identifying the start and end regions of the small intestine with high accuracy.

[0025] The technical effects of the present invention are not limited to the effects mentioned above, and other technical effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0026] Figure 1 is a schematic diagram of a neural network-based medical image processing device according to the present embodiment.

[0027] Figure 2 is a flowchart schematically illustrating a learning method and a testing method of an organ classification algorithm of a neural network-based medical image processing device according to the present embodiment.

[0028] Figure 3 is a flowchart showing the detection process of the organ segmentation algorithm of the neural network-based medical image processing device according to the present embodiment.

[0029] Figure 4 is a conceptual diagram showing the results according to the long-term classification algorithm of the neural network-based image processing device according to the present embodiment.

[0030] Figure 5 is the result data of analyzing the long-term classification algorithm of the neural network-based medical image processing device according to the present embodiment using a gradient weighted class activation map.

[0031] FIG. 6 is a diagram showing the transition error between the stomach and small intestine and the transition error between the small intestine and large intestine in the organ segmentation algorithm of the neural network-based medical image processing device according to the present embodiment.

[0032] Fig. 7 is a flowchart illustrating a neural network-based medical image processing method according to the present embodiment.

[0033] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. However, the present embodiments are not limited to the embodiments disclosed below and may be implemented in various forms. These embodiments are provided only to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention. The shapes of elements in the drawings may be exaggerated for clearer explanation, and elements indicated by the same reference numerals in the drawings represent the same elements.

[0034] Figure 1 is a schematic diagram of a neural network-based medical image processing device according to the present embodiment.

[0035] As illustrated in FIG. 1, a neural network-based medical image processing device (100, hereinafter referred to as a processing device) according to the present embodiment can predict frames in which an organ changes from a medical image (11) acquired from a wireless capsule endoscope (10) and provide only the small intestine image (12), which is the subject of analysis, to a reading doctor (30). At this time, the medical image (10) may be an image provided in real time from the wireless capsule endoscope (10) or a previously stored image provided from a database (not illustrated).

[0036] This processing device (100) can divide the medical image (11) into three sections, namely the stomach, small intestine, and large intestine, and provide only the small intestine image (12) to the interpreting doctor (30). In the conventional case, the medical image (11) could be divided into four sections, namely the esophagus, stomach, small intestine, and large intestine. However, when the organ division of the medical image (11) is performed into four sections, the number of learning data is small, resulting in data imbalance and a decrease in the performance of the algorithm. Therefore, the processing device (100) can divide the organs into three sections, excluding the esophagus section. At this time, the processing device (100) can identify the start and end regions of the small intestine based on the landmarks of the border between the stomach and small intestine and the landmarks of the border between the small intestine and large intestine in the organ division into three sections.

[0037] For this purpose, the processing device (100) may include a memory (110) and a processor (120).

[0038] The memory (110) is equipped with an organ segmentation algorithm (111) for performing organ segmentation of a medical image (11). In addition, the processor (120) can input the medical image (11) into the organ segmentation algorithm (111) to selectively extract a small intestine image (12) from the medical image (11).

[0039] First, the organ segmentation algorithm (111) mounted on the processing device (100) can be prepared by combining a 2D convolutional neural network (CNN) algorithm (111a) and a temporal filtering algorithm (111b). This organ segmentation algorithm (111) can be learned using images rather than videos. Hereinafter, the learning and design method of the organ segmentation algorithm (111) will be described in detail with reference to the attached drawings.

[0040] Fig. 2 is a flowchart schematically illustrating a learning method and a testing method of an organ segmentation algorithm of a neural network-based medical image processing device according to the present embodiment. Fig. 3 is a flowchart illustrating a detection process of an organ segmentation algorithm of a neural network-based medical image processing device according to the present embodiment, and Fig. 4 is a conceptual diagram illustrating results according to an organ segmentation algorithm of a neural network-based image processing device according to the present embodiment.

[0041] As illustrated in FIGS. 2 to 4, in the learning and design of the organ segmentation algorithm (111) according to the present embodiment, a WCE (Wireless Capsule Endoscope) image (13) acquired from a wireless capsule endoscope (10) may be used. However, this is for the purpose of explaining the present embodiment, and the type of learning data is not limited. Here, the WEC image (13) may be provided in JPEG format with a matrix size of 320*320 and 3FPS.

[0042] Afterwards, in the learning and design of the organ classification algorithm (111), organ-specific image labeling is performed (S210). In the image labeling, a reader can manually read the entire target WEC image (13) and perform 3-class labeling into stomach, small intestine, and large intestine. Then, the labeled WEC images are classified into a training set (S1), a validation set (S2), and a test set (S3) (S220). A random selection method can be applied in the image classification. At this time, each set (S1, S2, S3) can be classified so that normal data and abnormal data are included together.

[0043] Thereafter, in the learning and design of the organ segmentation algorithm (111), the training of the 2D convolutional neural network algorithm (111a) is performed based on the learning set (S1) (S230). The 2D convolutional neural network algorithm (111a) may be an algorithm for predicting the probability of a small intestine region. The 2D convolutional neural network algorithm (111a) is trained to distinguish the stomach, small intestine, and target using the ResNet50 model as a backbone network.

[0044] At this time, data enlargement or downsampling can be performed so that the ratio of images of the stomach, small intestine, and large intestine becomes 1:2:1 to adjust for the imbalance between organs. For example, the stomach images of normal and abnormal patients can be enlarged by two times by applying horizontal and vertical flips. In addition, the small intestine and large intestine images of normal patients can be downsampled by 2 / 3 and 1 / 3, respectively, and the small intestine and large intestine images of abnormal patients can be downsampled by 3 / 4 and 3 / 7, respectively. In training this 2D convolutional neural network algorithm (111a), the learning rate can be 0.001, and an ADAM optimizer with cross-entropy loss can be applied.

[0045] Meanwhile, when the training of the 2D convolutional neural network algorithm (111a) is completed, the organ segmentation algorithm (111) can classify the stomach, small intestine, and large intestine from the medical image (11) of the wireless capsule endoscope (10). That is, the organ segmentation algorithm (111) can classify the small intestine by predicting the probability of the boundaries of the stomach, small intestine, and large intestine.

[0046] Thereafter, the organ segmentation algorithm (111) is designed to perform a temporal filtering algorithm (111b). The temporal filtering algorithm may be a hybrid temporal filter composed of a Savitzky-Golay filter and a median filter. Accordingly, the temporal filtering algorithm (111b) enables the boundaries of the small intestine, stomach, and large intestine to be distinguished as a threshold using only the probability of the small intestine.

[0047] And in the application of the temporal filtering algorithm (111b), binary classification can be performed. The temporal filtering algorithm (111b) can add the results of the Savitzky-Golay filter and the median filter, divide them in half, and then map values ​​greater than 1 to 1 and values ​​less than 0 to 0. Here, if the maximum value is less than 1, the value can be divided by the maximum value. For example, the filtering range can be set to 1,001 frames, and after applying the temporal filtering algorithm (111b), the small intestine, stomach, and large intestine can be separated with a threshold value of 0.87. Here, the minimum index of the frame predicting the small intestine can be determined as the starting region of the small intestine, and the maximum index can be determined as the ending region of the small intestine.

[0048] In this way, the organ classification algorithm (111) can detect the transition point of an organ by applying a neural network algorithm to which a temporal filtering algorithm (111b) is applied. That is, the organ classification algorithm (111) classifies an image into the stomach, small intestine, and large intestine through a 2D convolutional neural network algorithm (111a). In addition, the organ classification algorithm (111) can significantly reduce the number of frames misclassified in the 2D convolutional neural network algorithm (111a) by correcting the class probability of a frame using the organ probability derived from adjacent frames through the temporal filtering algorithm (111b).

[0049] And, by specifying the temporally filtered probability for the small intestine as a threshold, the frames of the small intestine mapped to 1 and the frames of the stomach and large intestine mapped to 0 can be distinguished. Accordingly, the organ classification algorithm (111) can detect the boundary between the stomach and small intestine and the transition point between the boundary between the small intestine and large intestine, which have inter-organ dependencies in the image frame.

[0050] In this way, when the learning and design of the long-term classification algorithm (111) is completed, testing of the long-term classification algorithm (111) can be performed based on the test set (S3) (S240).

[0051] For example, in testing the organ segmentation algorithm (111), a gradient-weighted class activation map (Grad-CAM, hereinafter referred to as Grad-CAM), which is an explainable model, can be applied to the ResNet50 model. The Grad-CAM can be extracted from the feature map for the predicted class, then adjusted to the image size of the wireless capsule endoscope (10), 320*320, and can be superimposed on the original image. Accordingly, in testing, the performance of the learned and designed organ segmentation algorithm (111) can be quantitatively analyzed in terms of accuracy, sensitivity, specificity, PPV (Positive Predictive Value), and NPN (Negative Predictive Value).

[0052] Below, the 3-class classification results in the application of the long-term classification algorithm according to this embodiment will be described in detail with reference to the attached drawings.

[0053] FIG. 5 is a result data analyzed using a gradient weighted class activation map for the organ segmentation algorithm of a neural network-based medical image processing device according to the present embodiment, and FIG. 6 is a diagram showing the transition error between the stomach and the small intestine and the transition error between the small intestine and the large intestine in the organ segmentation algorithm of a neural network-based medical image processing device according to the present embodiment.

[0054] As shown in FIGS. 5 and 6, the long-term classification algorithm (111) according to the present embodiment was analyzed using Gradcam, and the color map in FIG. 5 represents a normalized prediction in which red and blue refer to 1 and 0, respectively.

[0055] Here, comparing Figures 5a and 5b reveals that the application results of the organ segmentation algorithm (111) according to the present embodiment are similar to the organ classification process of an endoscopist. That is, the trained and designed organ segmentation algorithm (111) was confirmed to classify organs using structural information such as dark areas captured along wrinkles and track directions and mucosal vascular patterns.

[0056] In addition, it was confirmed that the long-term classification algorithm (111) according to the present embodiment has higher performance compared to the conventional technology.

[0057] MethodOverallaccuracySmall bowelaccuracysensitivityspecificityPPVNPVPrior Art75.10%78.80%92.13%75.09.%78.76%78.89%ResNet5088.00%88.47%94.22%85.00%85.48%92.65%Prior Art+temporal filter-97.90%98.79%96.94%97.32%98.61%ResNet50+temporal filter-99.80%99.60%99.80%99.98%99.55%

[0058] Comparing the organ segmentation algorithm (111) according to the prior art and the present embodiment as shown in Table 1 above, it was confirmed that the organ segmentation algorithm (111) has high performance by combining the 2D convolutional neural network algorithm (111a) of the ResNet50 model and the temporal filtering algorithm (111b).

[0059] In addition, the results were analyzed for cases randomly selected from the test set (S3), and it was confirmed that the temporal filtering algorithm (111b) was strongly effective in distinguishing the small intestine, which is the subject of analysis. In particular, it was confirmed that the number of misclassified images in the 2D convolutional neural network algorithm (111a) was significantly reduced after applying the temporal filtering algorithm (111b). In addition, the problem of many misclassified frames in the large intestine region was solved by deriving an appropriate threshold value of 0.87, and it was shown that the distinction between the small intestine and the large intestine was possible.

[0060] And the time error was calculated as frame error and FPS for each case, and the transition error between the upper and lower intestines was only 38.8±25.8 seconds, and the transition error between the small intestine and the large intestine was only 32.0±19.1 seconds, showing that the organ segmentation algorithm (111) has a very low transition time error.

[0061] Meanwhile, below, a method for processing medical images for processing will be described in detail with reference to the attached drawings.

[0062] Fig. 7 is a flowchart illustrating a neural network-based medical image processing method according to the present embodiment.

[0063] As illustrated in FIG. 7, in the image processing method according to the present embodiment, a long-term classification algorithm (111) is loaded into the memory (110).

[0064] Thereafter, when a medical image (11) for processing is provided from the outside to the processing device (100), the processor (120) can perform organ classification by applying the medical image (11) for processing to an organ classification algorithm (111). Accordingly, the organ classification algorithm (111) can automatically classify only the small intestine frame and provide it to the reading doctor (30).

[0065] More specifically, the processor (120) inputs the medical image (11) for processing into the organ classification algorithm (111) (S710). Accordingly, the organ classification algorithm (111) inputs the medical image for processing into a 2D convolutional neural network algorithm and can classify the stomach, small intestine, and large intestine based on the ResNet50 model.

[0066] The processor (120) then applies the classified small intestine class to a temporal filtering algorithm (111) consisting of a Savitzky-Golay filter and a median filter (S720). Accordingly, the boundaries between the stomach and the small intestine and the small intestine and the large intestine can be distinguished by thresholds.

[0067] Thereafter, the processor (120) can provide the small intestine image (12), which is the analysis target, classified by the organ classification algorithm (111), to the reading doctor (30).

[0068] However, in this embodiment, it is described that the processor (120) directly transmits the small intestine image (13) bent by the organ segmentation algorithm (111) to the reading doctor (30). However, this is for the purpose of explaining this embodiment, and the organ segmentation algorithm (111) can be linked to another algorithm for subsequent processing, for example, determining the lesion area, to enable detection of lesions in the small intestine.

[0069] In this way, the neural network-based medical image processing device and method according to the present invention can significantly shorten the reading time of a clinician by automatically acquiring only images of the small intestine through organ segmentation.

[0070] In addition, the neural network-based medical image processing device and method according to the present invention has a great advantage when combined with other technologies for diagnosing lesions in the small intestine by identifying the start and end regions of the small intestine with high accuracy.

[0071] The embodiments of the present invention described above and illustrated in the drawings should not be construed as limiting the technical concept of the present invention. The scope of protection of the present invention is limited only by the matters set forth in the claims, and those skilled in the art will be able to make various improvements and modifications to the technical concept of the present invention. Accordingly, such improvements and modifications, as long as they are obvious to those skilled in the art, will fall within the scope of protection of the present invention.

Claims

1. In a neural network-based medical image processing device that distinguishes a small intestine region from a medical image acquired by a capsule endoscope, A memory equipped with an organ segmentation algorithm that performs organ segmentation for the above medical images; and A processor is included that applies the above medical image to the above organ segmentation algorithm so that the small intestine region is segmented, The above long-term classification algorithm A convolutional neural network algorithm that divides the organs included in the above medical image into the stomach, small intestine, and large intestine, and separates the small intestine region; A neural network-based medical image processing device characterized in that it includes a temporal filtering algorithm that is linked to the convolutional neural network algorithm and reduces images misclassified by the convolutional neural network algorithm.

2. In paragraph 1, The above convolutional neural network algorithm Includes a ResNet model trained on 2D images, The above temporal filtering algorithm A neural network-based medical image processing device characterized by including a hybrid temporal filter composed of a Savitzky-Golay filter and a median filter.

3. In paragraph 2, In the learning of the above convolutional neural network algorithm, By reading multiple 2D images, 3-class labeling is performed into the stomach, small intestine, and large intestine. A neural network-based medical image processing device characterized in that it creates a learning set with the labeled plurality of 2D images and trains the convolutional neural network algorithm based on the learning set so that the convolutional neural network algorithm predicts the small intestine region.

4. In paragraph 3, In the learning of the above convolutional neural network algorithm, After the labeling, the labeled multiple 2D images are randomly selected and classified into the training set, validation set, and test set, A neural network-based medical image processing device characterized in that each of the above sets is classified to include both normal data and abnormal data.

5. In paragraph 4, In the learning of the above convolutional neural network algorithm, A neural network-based medical image processing device characterized in that the 2D image ratio of the stomach, small intestine, and large intestine is adjusted to 1:2:1 so as to adjust the imbalance between the organs.

6. In paragraph 5, In adjusting the ratio of the above 2D image, Enlarge the normal and abnormal images by applying horizontal and vertical flips, A neural network-based medical image processing device characterized by downsampling normal and abnormal small intestine images and normal and abnormal large intestine images at a set ratio.

7. In paragraph 6, In the above downsampling, The normal small intestine and normal large intestine images were downsampled by 2 / 3 and 1 / 3, respectively. A neural network-based medical image processing device characterized in that abnormal small intestine and abnormal large intestine images are downsampled at ratios of 3 / 4 and 3 / 7, respectively.

8. In paragraph 4, In the learning of the above convolutional neural network algorithm, A neural network-based medical image processing device characterized in that the convolutional neural network algorithm is verified based on the verification set after training the convolutional neural network algorithm.

9. In paragraph 4, After learning the above convolutional neural network algorithm A neural network-based medical image processing device characterized in that the convolutional neural network algorithm and the temporal filtering algorithm are tested based on the test set.

10. In paragraph 2, In the application of the above temporal filtering algorithm, A neural network-based medical image processing device characterized by performing binary classification.

11. In paragraph 10, In the above binary classification, Applying the above-mentioned small class obtained by the above-mentioned convolutional neural network algorithm to the above-mentioned Savitzky-Golay filter and median filter. A neural network-based medical image processing device characterized in that the frame of the small intestine and the frame of the stomach and large intestine are distinguished by adding and dividing the results of the Savitzky-Golay filter and the median filter, and then mapping a value greater than 1 to 1 and a value less than 0 to 0.

12. In paragraph 1, The above long-term classification algorithm A neural network-based medical image processing device characterized in that it predicts frames in which an organ changes from the above medical image and distinguishes the small intestine region.

13. In paragraph 1, The above long-term classification algorithm The above-mentioned class obtained by the above-mentioned convolutional neural network algorithm is applied to the above-mentioned temporal filtering algorithm, and the temporally filtered probability is designated as a threshold value. A neural network-based medical image processing device characterized in that it distinguishes between the frames of the small intestine and the frames of the stomach and large intestine based on the above threshold value.

14. In paragraph 13, The above threshold is A neural network-based medical image processing device characterized by having an accuracy of 0.

87.

15. In paragraph 1, The above temporal filtering algorithm A neural network-based medical image processing device characterized in that the class probability of a frame is corrected using the long-term probability derived from adjacent frames by the above convolutional neural network algorithm.

16. A neural network-based medical image processing method for distinguishing a small intestine region from a medical image acquired by a capsule endoscope, A step of inputting the above medical image into an organ segmentation algorithm; and The above organ segmentation algorithm includes a step of segmenting the small intestine region from the medical image, The above long-term classification algorithm A convolutional neural network algorithm that divides the organs included in the above medical image into the stomach, small intestine, and large intestine, and separates the small intestine region; A medical image processing method characterized in that it includes a temporal filtering algorithm linked to the convolutional neural network algorithm to reduce images misclassified by the convolutional neural network algorithm.

Images