The method and device of domain adaptation between heterogeneous endoscopy

Abstract

The present invention relates to a domain adaptation technique and apparatus for heterogeneous endoscopes. It proposes an AI that randomly extracts retrospectively collected endoscope images, classifies them according to the type of endoscope, and verifies whether the AI ​​developed based on this deep learning can recognize unique image characteristics for each type of endoscope. Furthermore, it presents a method to improve the degraded recognition ability of the AI ​​by utilizing a generative adversarial network-based domain adaptation technique that enables mutual conversion by learning image characteristics, thereby converting images of a specific type of endoscope to have the characteristics of other types.

Description

Technology Field

[0001] The present invention relates to a domain adaptation technique and device between heterogeneous endoscopes. Background Technology

[0002] Deep learning-based artificial intelligence is heavily influenced by the nature of the training data, and if classifiable features exist independently of what is intended in the images to be recognized, it can be a cause of error.

[0003] Gastrointestinal endoscopy images possess unique detailed characteristics depending on the imaging device model. Given that the gastrointestinal endoscopy market is monopolized by a small number of companies, it is necessary to incorporate these specific image characteristics into the artificial intelligence development process from the outset.

[0004] It is necessary to improve the recognition capabilities of artificial intelligence, which may be degraded by the diverse image characteristics that exist depending on the endoscope model, by having the AI ​​recognize the shooting model of a specific endoscope image based on model-specific image characteristics learned from a large volume of endoscope images, and by utilizing domain adaptation technology based on generative adversarial networks to enable the mutual conversion of the unique image characteristics of endoscope models. Prior art literature

[0005] Republic of Korea Published Patent No. 10-2023-0033237 Republic of Korea Published Patent No. 10-2022-0054228 Republic of Korea Registered Patent No. 10-2496672 The problem to be solved

[0006] Accordingly, the present invention was devised to solve the aforementioned conventional problems. According to an embodiment of the present invention, the purpose is to provide a domain adaptation technique and apparatus between heterogeneous endoscopes that can improve the recognition ability of a degraded artificial intelligence by proposing an AI that randomly extracts retrospectively collected endoscopic images, classifies them according to the type of endoscope, and confirms whether the AI ​​developed based on this deep learning can recognize unique image characteristics for each type of endoscope, and by utilizing a generative adversarial neural network-based domain adaptation technique that enables mutual conversion by learning image characteristics to convert images of a specific type of endoscope to have the characteristics of other types.

[0007] According to an embodiment of the present invention, the purpose is to provide a domain adaptation technique and device between heterogeneous endoscopes that can contribute to overcoming the unique market structure of gastrointestinal endoscopes and securing stable performance using image recognition technology based on deep learning, which has recently gained popularity in the IT field.

[0008] Meanwhile, the technical problems to be solved by the present invention are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below. means of solving the problem

[0009] Upper gastrointestinal endoscopy was performed using three types of image processing devices and dedicated endoscopes for each model: Olympus CV-260SL, CV290 (Olympus, Japan), and PENTAX EPK-i (Pentax Medical, Japan).

[0010] Although Olympus’s image processing device can achieve a total output resolution of up to 1920×1080 pixels, the standard for high-resolution television, the actual endoscopic observation area is displayed within 1280×1024 pixels, which is offset to the right of the screen. The remaining area is used to display clinical information related to the examination. In the case of the PENTAX EPK-i model, 1280×1024 pixel image output is possible, and this entire area is used to display the endoscopic observation area.

[0011] Digital monitor signals output from each image processing device were duplicated and captured using a Matrox VIO 7IA OA / G (Matrox, USA), and only images of the endoscopic observation area were extracted from the examination information stored in the Picture Archiving and Communication System (PACS) in accordance with the Digital Imaging and Communication in Medicine (DICOM) standard as Portable Network Graphics (PNG), a lossless compression format, and used for training artificial intelligence.

[0012] From the entire image, only a square portion 1.16 times the radius of the circle circumscribed around the center of the endoscopic observation area was taken and resized to 512×512 pixels, and then a triangle connecting points 70 pixels from the four corners was colored black to standardize the specifications into an octagonal image.

[0013] This process was processed in batches using the Python language and the OpenCV library so that external factors of the images would not affect the learning and verification of the artificial intelligence, ensuring that there were no changes to characteristics such as color tone, sharpness, and aspect ratio (see Fig. 1). A total of 9,949 examination cases with 310,389 images were collected and all were anonymized.

[0015] The first objective of the present invention can be achieved as a domain adaptation device between heterogeneous endoscopes, characterized by comprising: a data collection unit that randomly collects endoscopic images of heterogeneous endoscopes; an endoscope model classification unit that learns the collected endoscopic images and classifies them according to the model; a feature recognition unit that learns image characteristics for each model of the endoscopic images classified according to the model and recognizes image characteristics; and a conversion unit that learns the image characteristics and mutually converts the endoscopic images of different models to have the characteristics of different models.

[0016] In addition, the data collection unit may be characterized by further including a preprocessing unit that standardizes the collected data and randomly distributes training images and verification images.

[0017] In addition, the endoscope type classification unit may be characterized by determining the endoscope type in the collected endoscope images based on a deep learning-based classification network such as EfficientNet, and the feature recognition unit may be characterized by independently learning the endoscope images classified by endoscope type in order to recognize image characteristics.

[0018] In addition, the above-mentioned transformation unit may be characterized by applying a domain adaptation technology based on a generative adversarial neural network that enables mutual transformation by learning image characteristics, thereby learning to transform an image of a specific endoscope model to have image characteristics of at least one other endoscope model.

[0019] In addition, the above transformation unit may be characterized by being based on a domain adaptation method such as CycleGAN.

[0020] The second objective of the present invention can be achieved as a domain adaptation technique between heterogeneous endoscopes, characterized by comprising: a first step of randomly collecting endoscopic images of heterogeneous endoscopes in a data collection unit; a second step in which a preprocessing unit standardizes the collected data and randomly distributes images for training and images for verification; a third step in which an endoscope model classification unit learns the collected endoscopic images and classifies them according to the model; a fourth step in which a feature recognition unit learns image characteristics for each model of the endoscopic images classified according to the model and recognizes image characteristics; and a fifth step in which a conversion unit learns the image characteristics and mutually converts the endoscopic images of different models to have the characteristics of different models.

[0021] And in the third step above, the type of endoscope is identified in the collected endoscope images based on a deep learning-based classification network such as EfficientNet, and in the fourth step above, the endoscope images classified by endoscope type are independently learned to recognize image characteristics based on a deep learning-based classification network such as EfficientNet.

[0022] In addition, in the above 5th step, the method may be characterized by applying a generative adversarial neural network-based domain adaptation technique that enables mutual conversion by learning image characteristics based on a domain adaptation method such as CycleGAN, thereby learning to convert an image of a specific endoscope model to have image characteristics of at least one other endoscope model.

[0023] And the above image characteristics may be characterized as being a gastroesophageal junction.

[0024] The third objective of the present invention can be achieved as a domain adaptation program between heterogeneous endoscopes, characterized by executing a technique according to the aforementioned third objective, as a program read and executed by a computer. Effects of the invention

[0025] According to the domain adaptation technique and apparatus between heterogeneous endoscopes according to an embodiment of the present invention, a method can be proposed to improve the recognition ability of a degraded artificial intelligence by randomly extracting retrospectively collected endoscope images, classifying them according to the endoscope model, and verifying whether a deep learning-based artificial intelligence developed based on this can recognize unique image characteristics for each endoscope model, and by utilizing a generative adversarial neural network-based domain adaptation technique that enables mutual conversion by learning image characteristics to convert images of a specific endoscope model to have the characteristics of other models.

[0026] According to the domain adaptation technique and apparatus between heterogeneous endoscopes according to an embodiment of the present invention, image recognition technology using deep learning-based artificial intelligence, which has recently gained popularity in the IT field, has the advantage of contributing to overcoming the unique market structure of gastrointestinal endoscopes and securing stable performance.

[0027] According to the domain adaptation technique and apparatus between heterogeneous endoscopes according to an embodiment of the present invention, when performance degradation occurs due to reasons such as a lack of database when attempting to recognize a disease or junction in an endoscope of Company A, the decrease in recognition rate can be minimized by converting the endoscope images of Company A into endoscope images of Company B based on a model trained on an endoscope of Company B, which has a larger database. Furthermore, by matching the color tone and image quality of the endoscope images of two different types, the effect is achieved that doctors using the endoscopes can convert and utilize the images into a form they are more familiar with.

[0028] Meanwhile, the effects obtainable from the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present invention belongs from the description below. Brief explanation of the drawing

[0029] The following drawings attached to this specification illustrate preferred embodiments of the present invention and serve to further enhance understanding of the technical concept of the present invention together with the detailed description of the invention; therefore, the present invention should not be interpreted as being limited only to the matters described in such drawings. Figure 1 is an example of standardization during the endoscopic image acquisition process. Figure 2 is an example of image marker using Microsoft Excel, Figure 3 shows the random sampling of images for training and verification of artificial intelligence. FIG. 4 is a schematic diagram according to an embodiment of the present invention, Figures 5a and 5b show the structures of the EfficientNet and CycleGAN algorithms, Figure 6 shows the performance of artificial intelligence for identifying the type of endoscope used for imaging, Figure 7 shows a comparison of AI performance for gastroesophageal junction recognition in the o260 dataset. Figure 8 shows a comparison of AI performance for gastroesophageal junction recognition in the o290 dataset. Figure 9 shows a comparison of AI performance for gastroesophageal junction recognition in the pEPK dataset. Figure 10 shows an example of the cycleGAN transform and class activation map of an AI that recognizes images of the gastroesophageal junction. Figure 11 shows the improvement in artificial intelligence performance on verification data transformed to have the characteristics of the learned images (o260 dataset), Figure 12 shows the improvement in artificial intelligence performance on verification data transformed to have the characteristics of the learned image (o290 dataset), Figure 13 shows the improvement in artificial intelligence performance (pEPK dataset) in verification data transformed to have the characteristics of the learned images. Specific details for implementing the invention

[0030] The above objects, other objects, features, and advantages of the present invention will be easily understood through the following preferred embodiments associated with the accompanying drawings. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided to ensure that the disclosed content is thorough and complete and to ensure that the spirit of the invention is sufficiently conveyed to a person skilled in the art.

[0031] In this specification, when a component is described as being on another component, it means that it may be formed directly on the other component or that a third component may be interposed between them. Also, in the drawings, the thicknesses of the components are exaggerated for the effective description of the technical content.

[0032] The embodiments described herein will be explained with reference to cross-sectional and / or plan views, which are exemplary illustrations of the invention. In the drawings, the thicknesses of films and regions are exaggerated for effective explanation of the technical content. Accordingly, the shapes of the exemplary drawings may be modified by manufacturing techniques and / or tolerances, etc. Accordingly, the embodiments of the invention are not limited to the specific shapes depicted but include variations in shape produced according to the manufacturing process. For example, a region depicted as a right angle may be rounded or have a certain curvature. Accordingly, the regions illustrated in the drawings have properties, and the shapes of the regions illustrated in the drawings are intended to illustrate specific shapes of the regions of the device and are not intended to limit the scope of the invention. Although terms such as first, second, etc., have been used to describe various components in the various embodiments of this specification, these components should not be limited by such terms. These terms are used merely to distinguish one component from another. The embodiments described and illustrated herein also include their complementary embodiments.

[0033] The terms used herein are for describing the embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. As used herein, 'comprises' and / or 'comprising' do not exclude the presence or addition of one or more other components to the mentioned components.

[0034] In describing the specific embodiments below, various specific details have been included to explain the invention more specifically and to aid understanding. However, a reader with sufficient knowledge in the art to understand the invention will recognize that it can be used without these various specific details. In some cases, it is noted in advance that commonly known aspects that are not significantly related to the invention have been omitted to prevent unnecessary confusion in describing the invention.

[0036] According to an embodiment of the present invention, the invention includes deep learning technology for distinguishing between different types of endoscope images, technology for improving recognition rate by domain transforming an image of another endoscope (B) back to the corresponding endoscope (A) for a model trained with one endoscope (A) and then applying it to the model trained with the corresponding endoscope (A), technology for recognizing the junction between the esophagus and the stomach through domain adaptation, and technology for normalizing collected images to have the same resolution and content in order to apply domain adaptation between heterogeneous endoscopes.

[0037] Currently, when there are various types of endoscopes, a large amount of database must be collected for each endoscope. However, according to an embodiment of the present invention, if a high-performance model is created using an endoscope with a sufficient amount of database, and images from other endoscopes are converted to those with a high-performance model through domain adaptation, the decrease in recognition rate can be minimized even if there is not much database.

[0039] First, the configuration, function, and adaptation method of a domain adaptation device between heterogeneous endoscopes according to an embodiment of the present invention will be described.

[0040] A domain adaptation device between heterogeneous endoscopes according to an embodiment of the present invention may be configured to include a data collection unit, a preprocessing unit, an endoscope model classification unit, a characteristic recognition unit, a conversion unit, etc.

[0041] A data collection unit according to an embodiment of the present invention randomly collects endoscopic images of heterogeneous endoscopes. Additionally, a preprocessing unit is configured to standardize the collected data and randomly distribute images for training and images for verification.

[0042] And, the endoscope type classification unit according to an embodiment of the present invention learns the collected endoscope images and classifies them according to the type. The endoscope type classification unit may be configured to determine the endoscope type from the collected endoscope images based on a deep learning-based classification network such as EfficientNet.

[0043] In addition, the feature recognition unit is configured to recognize image characteristics by learning the image characteristics for each model of the endoscopic images classified according to the above models. These image characteristics may be, for example, the gastroesophageal junction.

[0044] The feature recognition unit can independently learn endoscopic images classified by endoscope model to recognize image features based on a deep learning-based classification network such as EfficientNet.

[0045] The conversion unit according to an embodiment of the present invention is configured to learn the image characteristics and mutually convert the endoscope model image to have the characteristics of another model.

[0046] The transformation unit applies a domain adaptation technique based on a generative adversarial neural network that learns image characteristics to enable mutual transformation, thereby learning to transform images of a specific endoscope model to have the image characteristics of at least one other endoscope model.

[0047] In addition, the transformation unit according to an embodiment of the present invention may be based on a domain adaptation method such as CycleGAN.

[0049] Below, the experimental method, statistical analysis, and results according to the experimental example of the present invention will be described.

[0050] 1-A. Experimental Method

[0051] Images collected using image enhancement functions such as dyes that cause artificial color tones, narrowband imaging, or i-scan were excluded. These images were classified by labeling them as o260, o290, and pEPK, respectively, according to the three endoscope models used for imaging: Olympus CV-260SL, CV290, and PENTAX EPK-i. Among these, images in which the Z-line—the boundary between the squamous and columnar epithelium—was identified in the lower esophageal images were separately labeled as gastroesophageal junction images following consensus among the three specialists. Microsoft Excel and its built-in Visual Basic for Application were used to ensure this process was performed accurately (see Fig. 2).

[0052] Using Microsoft Excel, the number of images of the gastroesophageal junction and the remaining images were organized for each case. Random numbers were assigned to these images to randomly sample them, and the training and validation data for the AI ​​were distributed in an 8:2 ratio to ensure that the image sizes were similar and there were no duplicate cases (see Fig. 3). Based on this, two types of AI were developed: one for recognizing the type of endoscope captured in the images and another for distinguishing the gastroesophageal junction images (see Fig. 4). The AI ​​for recognizing the endoscope type was trained on all the images distributed for training. The AI ​​for distinguishing the gastroesophageal junction was independently trained on the training images divided by endoscope type, and each was optimized for its respective purpose. Both AIs were developed based on the EfficientNet-B0 model, which has been verified for efficiency and accuracy (see Fig. 5).

[0053] The size of the input images to the PyTorch-based EfficientNet, which is based on initial variables set through pre-training using ImageNet, was adjusted to 224×224 pixels, and Stochastic Gradient Descent was used with a learning rate of 0.05, momentum of 0.9, and weight decay of 1e-4. Cross-entropy was used as the loss function, and the model that showed the highest accuracy during training for 200 epochs on the image data classified for validation was finally selected.

[0054] The CycleGAN algorithm also learned image characteristics by endoscope model using the same classified images used for training the image recognition AI. The images were adjusted to 256×256 pixels for input, and an Adam optimizer was applied with a learning rate of 0.0002. Based on this, the verification image data was transformed to possess the characteristics of two different models. Since there was no standard to verify the image characteristic transformation, the result of training for 200 epochs was selected as the final model. All transformed images were reviewed to confirm whether there were any issues using them to verify the AI's ability to recognize the gastroesophageal junction.

[0055] All computations were performed on a computer equipped with an NVIDIA RTX2080 (NVIDIA, USA) with 8-gigabyte memory. The performance of the AI ​​for identifying endoscope models was measured by top-1 accuracy based on the maximum softmax value. For the AI ​​recognizing the gastroesophageal junction, performance was evaluated by setting a probability of 0.5 as the threshold, and to qualitatively verify this, a class activation map was generated separately to check whether the gastroesophageal junction was recognized according to the development intent.

[0057] 1-B. Statistical Analysis

[0058] Using only accuracy to evaluate the performance of artificial intelligence that targets the gastroesophageal junction, which accounts for a small proportion of images in upper gastrointestinal endoscopy, makes it difficult to reflect the imbalance between classification targets. To supplement this, the F1 score, which is the harmonic mean of precision and recall calculated by the following formula, was presented together and calculated using the SPSS program (version 26.0, IBM SPSS Statistics for Windows, IBM corp., USA).

[0059] For AI recognizing the gastroesophageal junction, the differences between the ROC curves (Receiver operating characteristic curves) of three AIs specialized for each endoscope model were compared using the DeLong test on validation images classified by endoscope model, using the MedCalc program (version 20.100, MedCalc Software Ltd., Belgium). Statistical significance was determined based on P<0.05.

[0061] 2. Results

[0062] 2,160 examinations containing a total of 57,105 images were randomly selected for the training and validation of the artificial intelligence (see Table 1). The average age of the subjects was 54.7 years, and 970 were female. 831 examinations were performed with the Olympus CV-260L, and 1,423 of the 19,037 examination images were of the gastroesophageal junction. 681 examinations were performed with the Olympus CV-290, and 1,489 of the 19,037 images were of the gastroesophageal junction. 648 examinations were performed with the PENTAX EPK-i, and 1,404 of the 19,031 images were of the gastroesophageal junction. Among the gastroesophageal junction images extracted, findings of reflux esophagitis were found in 605 examinations, Barrett's esophagus was confirmed in 38 examinations, and esophageal varices were confirmed in 14 examinations.

[0063] The AI ​​that estimates the endoscope model used for imaging was able to distinguish between the PENTAX EPK-i and other Olympus models without error. Errors occurred in which 93 images out of o290 were identified as o260, and 14 images out of o260 were identified as o290. The top-1 accuracy of this AI was calculated to be 0.991, and the F1 scores for predicting o260, o290, and pEPK were 0.986, 0.986, and 1.000, respectively, and the area under the receiver operating characteristic curve (ROC) was 0.998, 0.999, and 1.000, respectively (see Fig. 6).

[0064] The AI ​​systems for recognizing the gastroesophageal junction were specialized and named AI-o260, AI-o290, and AI-pEPK based on the endoscope models. When their recognition performance was verified using validation images, all of them showed the highest accuracy and F1 scores in images from the respective specialized models (Table 2). AI-o260 successfully recognized 285 images of the gastroesophageal junction in the validation o260 images, achieving an accuracy of 0.988 and an F1 score of 0.917. AI-o290 demonstrated an accuracy of 0.979 and an F1 score of 0.867 in o290 images, while AI-pEPK demonstrated an accuracy of 0.986 and an F1 score of 0.906 in pEPK images. The AIs showed degraded performance in endoscope images different from the models they were trained on. The results of comparing the AUROCs were statistically significant in all cases except for the comparison of AI-o260 and AI-o290 in the o260 image (see Fig. 7) (see Figs. 8 and 9).

[0065] The three specialists participating in the invention reconfirmed that there was no change in the image marker patterns even after the original images, divided into three groups by endoscope model, were converted to possess the image characteristics of the other two models. Through CAM images, it was confirmed that the AI ​​recognized the squamous-columnar epithelial boundary and distinguished the gastroesophageal junction as intended in the study (see Fig. 10). The AI ​​showed improved performance in images converted to possess the characteristics of the respective endoscope models it was trained on, and the improvements in AUROC were all statistically significant. When o260 images were converted to possess o290 characteristics, the AUROC of AI-290 increased by 0.0056, and when converted to possess pEPK characteristics, the recognition performance of AI-pEPK improved by 0.0182 (see Fig. 11). When the original o290 images were converted to possess o260 and pEPK characteristics, the performance improvements of AI-o260 and AI-pEPK were 0.0134 and 0.0299, respectively (see Fig. 12). When the image characteristics of AI-o260 and AI-o290 in pEPK images were converted to suit each, there was a performance improvement of 0.0215 and 0.0616, respectively (see Fig. 13).

[0067] Table 1. Clinical aspects of test cases distributed as training and validation data for artificial intelligence.

[0068]

[0069] o260, Olympus CV-260SL; o290, Olympus CV-290; pEPK, PENTAX EPK-i

[0070] Table 2. Performance comparison of gastroesophageal junction recognition AI based on validation data

[0071]

[0072] o260, Olympus CV-260SL; o290, Olympus CV-290; pEPK, PENTAX EPK-i; AI, artificial intelligence

[0074] Figure 1 is an example of standardization during the endoscopic image acquisition process. The acquired endoscopic image is standardized to a size of 512×512 pixels by taking only a square portion (green rectangle) with a radius 1.16 times the center of a circle (blue reference line) circumscribing the observation area, and this process was performed in batches using the Python language and the OpenCV library (from left to right: Olympus CV-260SL, CV-290, PENTAX EPK-i).

[0075] Figure 2 is an example of image labeling using Microsoft Excel. Three specialists participating in the study used Microsoft Excel and Visual Basic for Application functions to label the data in order to minimize errors in the process of classifying gastroesophageal junction images and reviewing the results.

[0076] Figure 3 shows the random sampling of images for training and verification of artificial intelligence. Random numbers were assigned to images classified into the gastroesophageal junction and the remainder for each endoscope model, and random sampling was performed based on cases to ensure there was no difference in image size among the three models. The images selected for use in the study were distributed in an 8:2 ratio and used for training and verification, respectively, in the development of two types of artificial intelligence, and were also utilized in image feature transformation experiments using cycleGAN.

[0077] Figure 4 is a schematic diagram according to an embodiment of the present invention, wherein the extracted endoscopic images were distributed in a random ratio of 8:2 for training and verification purposes. The performance of an artificial intelligence trained using training data was tested against verification data and data transformed using cycleGAN to have two different types of endoscopic image characteristics.

[0078] o260, Olympus CV-260SL; o290, Olympus CV-290; pEPK, PENTAX EPK-i; EGJ, esophagogastric junction; GAN, generative adversarial network

[0079] Figures 5a and 5b illustrate the algorithm structures of EfficientNet and CycleGAN.

[0080] As shown in Fig. 5a, EfficientNet independently learned images classified by endoscope type to identify the endoscope type used for imaging and to recognize the gastroesophageal junction.

[0081] As shown in Figure 5b, CycleGAN was trained to transform images of a specific endoscope model to have image characteristics of two other endoscope models.

[0082] MBConv, inverted linear BottleNeck layer with depth-wise separable convolution; BN, batch normalization; FC, fully connected layer; Conv, convolution; GAP, global average pooling; EGJ, esophagogastric junction; GAN, generative adversarial network

[0083] Figure 6 illustrates the performance of an artificial intelligence that identifies the type of endoscope used for imaging. The AI ​​made an error in identifying 93 images in the o260 dataset as having been captured with an Olympus CV-290 and 14 images in the o290 dataset as having been captured with an Olympus CV-260SL. It succeeded in recognizing all images in the pEPK dataset as having been captured with a PENTAX EPK-i.

[0084] o260, Olympus CV-260SL; o290, Olympus CV-290; pEPK, PENTAX EPK-i; AUROC, area under receiver operating characteristic curve

[0085] Figure 7 illustrates a comparison of the performance of AI for gastroesophageal junction recognition on the o260 dataset. When evaluating the AI's ability to recognize the gastroesophageal junction on the verification o260 dataset, the AI ​​trained on the training o260 dataset showed the highest recognition performance (0.992, AI-o260, blue solid line). The AI ​​trained on the o290 dataset lagged behind with a performance of 0.991 (AI-o290, green dotted line), but there was no statistically significant difference. The AI ​​built based on images from EPK-i models from different manufacturers lagged the most (0.969, AI-pEPK, orange dotted line), and the difference was statistically significant.

[0086] o260, Olympus CV-260SL; o290, Olympus CV-290; pEPK, PENTAX EPK-i; AI, artificial intelligence; EGJ, esophagogastric junction; GAN, generative adversarial network

[0087] Figure 8 illustrates a comparison of the performance of AI for gastroesophageal junction recognition on the o290 dataset. When evaluating the AI's ability to recognize the gastroesophageal junction on the verification o290 dataset, the AI ​​trained on the training o290 dataset showed the highest recognition performance (0.990, AI-o290, green dotted line). The AI ​​trained on the o260 dataset lagged behind with a performance of 0.973 (AI-o260, blue solid line), and there was a statistically significant difference. The AI ​​built based on images from EPK-i models from different manufacturers lagged the most (0.934, AI-pEPK, orange dotted line), and the difference was statistically significant.

[0088] o260, Olympus CV-260SL; o290, Olympus CV-290; pEPK, PENTAX EPK-i; AI, artificial intelligence; EGJ, esophagogastric junction; GAN, generative adversarial network

[0089] Figure 9 illustrates a comparison of AI performance for gastroesophageal junction recognition on the pEPK dataset. When evaluating the AI's ability to recognize the gastroesophageal junction on the validation pEPK dataset, the AI ​​trained on the training pEPK dataset showed the highest recognition performance (0.992, AI-pEPK, orange dotted line). The AI ​​trained on the o260 dataset lagged behind with a performance of 0.967 (AI-o260, blue solid line), and there was a statistically significant difference. The AI ​​built based on o290 model images lagged the most (0.932, AI-o290, green dotted line), and the difference was statistically significant.

[0090] o260, Olympus CV-260SL; o290, Olympus CV-290; pEPK, PENTAX EPK-i; AI, artificial intelligence; EGJ, esophagogastric junction; GAN, generative adversarial network

[0091] Figure 10 shows an example of the cycleGAN transformation and Class activation map of an artificial intelligence that recognizes images of the gastroesophageal junction.

[0092] When the AI ​​indicated on the far left recognizes the gastroesophageal junction based on the image on the left, it is displayed in bright blue as the probability of the junction increases, as shown in the color scale on the right. The CAM images generated in this way are superimposed on the original images and arranged on the right. The transformation results using CycleGAN for the original images listed in the top row are arranged at the bottom. The color boundary formed around the squamous-columnar epithelial boundary located in the center of the image suggests that the AI ​​recognizes the gastroesophageal junction in the intended direction.

[0093] o260, Olympus CV-260SL; o290, Olympus CV-290; pEPK, PENTAX EPK-i; CAM, class activation map; EGJ, esophagogastric junction; GAN, generative adversarial network

[0094] Figure 11 shows the improvement in artificial intelligence performance (o260 dataset) in verification data transformed to have the characteristics of the learned images.

[0095] The AI ​​that recognizes the gastroesophageal junction, trained on original images of each model, shows degraded recognition performance on images of other endoscope models (AI-o290, AI-pEPK indicated by dotted lines), but shows significant performance improvement on the validation dataset transformed to have the characteristics of the images trained by the AI ​​(AI-o290, AI-pEPK indicated by solid lines).

[0096] o260, Olympus CV-260SL; o290, Olympus CV-290; pEPK, PENTAX EPK-i; AI, artificial intelligence; EGJ, esophagogastric junction; GAN, generative adversarial network

[0097] Figure 12 shows the improvement in artificial intelligence performance (o290 dataset) on verification data transformed to have the characteristics of the learned images.

[0098] The AI ​​that recognizes the gastroesophageal junction, trained on original images of each model, shows degraded recognition performance on images of other endoscope models (AI-o260, AI-pEPK indicated by dotted lines), but shows significant performance improvement on the validation dataset transformed to have the characteristics of the images trained by the AI ​​(AI-o260, AI-pEPK indicated by solid lines).

[0099] o260, Olympus CV-260SL; o290, Olympus CV-290; pEPK, PENTAX EPK-i; AI, artificial intelligence; EGJ, esophagogastric junction; GAN, generative adversarial network

[0100] Figure 13 shows the improvement in artificial intelligence performance (pEPK dataset) in verification data transformed to have the characteristics of the learned images.

[0101] The AI ​​that recognizes the gastroesophageal junction, trained on original images of each model, shows degraded recognition performance on images of other endoscope models (AI-o260, AI-o290 indicated by dotted lines), but shows significant performance improvement on the validation dataset transformed to have the characteristics of the images trained by the AI ​​(AI-o260, AI-o290 indicated by solid lines).

[0103] In addition, the apparatus and method described above are not limited to the configurations and methods of the embodiments described above; rather, all or part of each embodiment may be selectively combined to allow for various modifications to be made.

Claims

Claim 1 A domain adaptation device between heterogeneous endoscopes, comprising: a data collection unit that randomly collects endoscopic images of heterogeneous endoscopes; an endoscope model classification unit that learns the collected endoscopic images and classifies them according to the model; a feature recognition unit that learns image characteristics for each model of the endoscopic images classified according to the model and recognizes image characteristics; and a conversion unit that learns the image characteristics and mutually converts the endoscopic images of different models to have the characteristics of different models. Claim 2 A domain adaptation device between heterogeneous endoscopes according to claim 1, wherein the data collection unit further includes a preprocessing unit that standardizes the collected data and randomly distributes training images and verification images. Claim 3 A domain adaptation device between heterogeneous endoscopes according to claim 2, characterized in that the endoscope type classification unit identifies the endoscope type in the collected endoscope images based on a deep learning-based classification network such as EfficientNet, and the feature recognition unit independently learns the endoscope images classified by endoscope type to recognize image characteristics. Claim 4 A domain adaptation device between heterogeneous endoscopes, wherein, in claim 3, the conversion unit applies a generative adversarial neural network-based domain adaptation technology that enables mutual conversion by learning image characteristics to convert an image of a specific endoscope model to have image characteristics of at least one other endoscope model. Claim 5 A domain adaptation device between heterogeneous endoscopes, characterized in that, in claim 4, the transformation unit is based on a domain adaptation method such as CycleGAN. Claim 6 A domain adaptation technique between heterogeneous endoscopes, comprising: a first step in which a data collection unit randomly collects endoscopic images of heterogeneous endoscopes; a second step in which a preprocessing unit standardizes the collected data and randomly distributes images for training and images for verification; a third step in which an endoscope model classification unit learns the collected endoscopic images and classifies them according to the model; a fourth step in which a feature recognition unit learns image characteristics for each model of the endoscopic images classified according to the model and recognizes image characteristics; and a fifth step in which a conversion unit learns the image characteristics and mutually converts the endoscopic images of different models to have the characteristics of different models. Claim 7 A domain adaptation technique between heterogeneous endoscopes according to claim 6, characterized in that, in the third step, the type of endoscope is identified in collected endoscopic images based on a deep learning-based classification network such as EfficientNet, and in the fourth step, the endoscopic images classified by type of endoscope are independently learned to recognize image characteristics based on a deep learning-based classification network such as EfficientNet. Claim 8 A domain adaptation technique between heterogeneous endoscopes according to claim 7, characterized in that, in the 5th step above, a generative adversarial neural network-based domain adaptation technique is applied to learn image characteristics based on a domain adaptation method such as CycleGAN to enable mutual conversion, thereby learning to transform an image of a specific endoscope model to have image characteristics of at least one other endoscope model. Claim 9 A domain adaptation technique between heterologous endoscopes characterized in that, in claim 6, the image characteristic is the gastroesophageal junction. Claim 10 delete

Images