Artificial intelligence-based gastric endoscopy image diagnostic support system and method

Abstract

To improve accuracy of a medical video diagnosis result by a user, by providing an evaluation score including reliability and accuracy for a plurality of medical video analyzing algorithms in a process of the user performing diagnosis on medical video.SOLUTION: A system and a method for supporting gastroscopy video diagnosis on the basis of artificial intelligence are disclosed. The system includes a processor for analyzing a video frame of gastroscopy video using at least one or more medical video analyzing algorithms and detecting whether a suspicious lesion part exists in the video frame. When a suspicious lesion part exists in the video frame, the processor calculates position coordinates of the suspicious lesion part. The processor creates display information including the presence or absence of the existence of the suspicious lesion part and the position coordinates of the suspicious lesion part.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an apparatus and method for assisting medical video diagnosis by an automation system. Specifically, when performing a gastroscopy examination by an artificial intelligence-based medical video analysis algorithm, it relates to an apparatus, system and method for assisting in the diagnosis of gastroscopy videos and reducing the risk of lesion omission.

[0002] The present invention is derived from research conducted as part of the Regulatory Free Zone Innovation Business Incubation (R&D) project of the Small and Medium Venture Business Administration and the Korea Institute of Industrial Technology [Problem ID Number: 1425145953, Project Number: P0011352, Project Title: Algorithm-centered Medical Wellness Information Service·Equipment Development Platform Demonstration].

Background Art

[0003] Endoscopic diagnosis is a medical procedure that occurs very frequently for the purpose of regular health checkups. During such endoscopic diagnosis, there is a need for a technology that preprocesses real-time video so that experts can easily identify lesions at the medical site. Recently, technologies have been introduced, such as U.S. Patent Publication No. 2018 / 0253839 “A system and method for detection of suspicious tissue regions in an endoscopic procedure”, which perform a preprocessing process of removing noise from video frames and perform a noise removal preprocessing process and a computer-aided diagnosis (CAD) process in parallel to provide real-time diagnostic support display information.

[0004] In such technologies, the accuracy and reliability of the CAD module are recognized as very important factors.

[0005] Techniques for segmenting (segmentation) or detecting (detection) objects within images, and for classifying (classification) objects within images, are utilized in a variety of applications in image processing. In medical imaging, objects within images are segmented, detected, and classified based on their brightness or intensity values. In this case, the objects can be human organs or lesions.

[0006] In recent years, the introduction of deep learning and convolutional neural networks (CNNs) as artificial neural networks has dramatically improved the performance of automated video processing processes.

[0007] However, on the other hand, recent artificial neural networks such as deep learning and CNNs are almost black boxes internally, and even if the derived results are excellent, users are reluctant to fully accept and adopt them. This reluctance towards artificial neural networks is particularly pronounced in the medical imaging field, which deals with human life.

[0008] Against this backdrop, research into explainable artificial intelligence (X-AI) is being attempted by organizations such as the Defense Advanced Research Projects Agency (DARPA) of the Department of Defense (https: / / www.darpa.mil / program / explainable-artificial-intelligence). However, no visible results have yet emerged.

[0009] In the medical field, a technique that selectively applies multiple segmentation algorithms to segment / detect, classify, and diagnose lesions with complex morphologies has been introduced, such as in the internationally published patent WO2018 / 015414, "METHOD AND SYSTEM FOR ARTIFICIAL INTELLIGENCE BASED MEDICAL IMAGE SEGMENTATION."

[0010] In the aforementioned prior literature, a technique is applied to obtain the final result of video segmentation by comparing pre-trained segmentation algorithms and selecting at least one of them.

[0011] However, since it is not possible to derive descriptive information (explanation) from the aforementioned prior literature regarding the criteria for selectively applying the partitioning algorithm, there is a problem in that it is difficult for clinicians to gain confidence in how clinically useful such partitioning techniques are.

[0012] Furthermore, Korean Registered Patent No. 10-1938992, "CAD System and Method for Generating Diagnostic Reasons," introduced a technique for generating feature vectors by sequentially fusing feature information extracted based on a DNN in order to derive evidence for lesion diagnosis. However, since Korean Registered Patent No. 10-1938992 merely derives feature information from the artificial neural network itself, and does not perform any verification of whether the extracted feature information is clinically useful, there is little basis for claiming that humans will be able to recognize the explanation of the diagnostic results from the artificial neural network.

[0013] Similarly, a similar problem still exists in the diagnostic process of medical imaging, where the process by which artificial intelligence diagnostic systems, which operate like a black box, derive their results cannot be clinically trusted.

[0014] Previously known research (S. Kumar et al., "Adenoma miss rates associated with a 3 minute versus 6 minute colonoscopy withdrawal time: a prospective, randomized trial,") It is known that up to 25% of lesions can be missed during a gastroscopy. This phenomenon is known to occur due to imaging problems, blind spots, and human error. Because doctors often experience fatigue due to the continuous and repetitive nature of the procedure, lesions may not be detected properly. Furthermore, human error can cause lesions to be missed, negatively impacting the medical outcome of the examination. [Prior art documents] [Patent Documents]

[0015] [Patent Document 1] U.S. Patent Publication No. 2018 / 0253839 [Patent Document 2] International Published Patent No. WO2018 / 015414 [Patent Document 3] Korean Registered Patent Publication No. 10-1938992 [Patent Document 4] Korean Registered Patent Publication No. 10-1850385 [Patent Document 5] Korean Registered Patent Publication No. 10-1230871 [Non-patent literature]

[0016] [Non-Patent Document 1] S. Kumar et al., "Adenoma miss rates associated with a 3 minute versus 6 minute colonoscopy withdrawal time: a prospective, randomized trial," Gastrointest. Endosc., vol. 85, no. 6, pp. 1273 1280, Jun. 2017, doi: 10.1016 / j. gie.2016.11.030 (June 2017) [Non-Patent Document 2] DA Corley et al., "Adenoma Detection Rate and Risk of Colorectal Cancer and Death," N Engl J Med., vol. 14, no. 3, pp. 1298 306, 2014, doi: 10.1056 / NEJMoa1309086 (April 2014) [Overview of the project] [Problems that the invention aims to solve]

[0017] Recently, efforts have been ongoing to improve the performance of video segmentation, object detection, and classification techniques by applying deep learning-based artificial intelligence methods. However, a limitation of deep learning-based artificial intelligence is that it is a black box in which the user cannot know whether the results provided by its operation are high performance by chance or whether it has gone through an appropriate decision-making process for the task.

[0018] On the other hand, rule-based training or learning methods, which are easy to explain, are limited in their use because they cannot achieve the same level of results as deep learning. Therefore, there is active research into deep learning-based artificial intelligence that can provide descriptive information (explanation) while having improved performance. In the practical application of image processing using artificial neural networks, particularly in the medical imaging field, descriptive information about the basis for diagnosis and classification is necessary, but conventional technologies are still unable to derive such descriptive information.

[0019] In the aforementioned prior document WO2018 / 015414, it is impossible to derive descriptive information (explanation) regarding which elements affect the improvement of the final segmentation performance. Even if a clinician provides clinically meaningful feedback during the segmentation process, there is no way to confirm whether this feedback has actually been appropriately applied to the deep learning system.

[0020] An object of the present invention is to improve the accuracy of medical video diagnosis results by a user by providing an evaluation score including reliability and accuracy for a plurality of medical video analysis algorithms during the process of the user diagnosing medical videos.

[0021] An object of the present invention is to provide recommendation information as descriptive information during the process of a user deriving a final diagnosis result using an artificial intelligence medical video analysis algorithm, and to provide information that quantifies the clinical usefulness of the user for the medical video analysis algorithm.

[0022] An object of the present invention is to generate and provide an optimized combination of a plurality of artificial intelligence medical video diagnosis results as display information for each real-time video frame.

[0023] An object of the present invention is to provide an optimized combination of a plurality of artificial intelligence medical video diagnosis results that can efficiently display diagnosis results that are highly likely to be diagnosed, highly likely to be missed, or have a high risk in the current video frame.

[0024] An object of the present invention is to provide a user interface and a diagnostic computing system that enable medical staff to confirm and consider in real time during an endoscopy by automatically detecting and presenting diagnosis results that are highly likely to be diagnosed, highly likely to be missed, or have a high risk in the current video frame.

[0025] The objective of this invention is to improve operational efficiency and diagnostic accuracy by training an artificial intelligence algorithm to identify polyps, ulcers, and various gastrointestinal diseases that users might miss based on AI medical image diagnosis results for each real-time video frame of gastric endoscopy images, and then applying this to an AI diagnostic support system.

[0026] According to previous research (DA Corley et al., "Adenoma Detection Rate and Risk of Colorectal Cancer and Death"), a 1.0% increase in the adenoma detection rate is known to correlate with a 3.0% decrease in the incidence of colorectal cancer. Therefore, the present invention aims to increase the lesion detection rate, eliminate gastric cancer risk factors early, and lower the incidence of gastric cancer. Furthermore, it aims to contribute to reducing the causes of gastric cancer by enabling physicians to detect and treat more lesions than before, and also to contribute to reducing the frequency of examinations.

[0027] The objective of this invention is to automatically detect diseases that are easily missed by the user during a gastroscopy and to indicate their location along the gastrointestinal pathway (gastric endoscopy pathway), thereby enabling the user to easily review the findings in real time during the gastroscopy and generating a report that can be reviewed by other examiners afterward, all with simple operation.

[0028] In the diagnostic support technology for gastrointestinal endoscopy images to which the present invention applies, there is a difficulty in detecting various lesions in the gastrointestinal tract when they are located together with the gastrointestinal wall and wrinkles, as they are easily missed if they do not have a color difference from the surrounding tissues and are small in size. Therefore, the objective of the present invention is to provide a method that can detect various lesions in real time using artificial intelligence to further improve the detection rate of gastrointestinal diseases, along with providing their position on the gastrointestinal endoscopy route, so that other examiners can reconfirm those lesions later. [Means for solving the problem]

[0029] The present invention was derived as a means to solve the problems of the prior art, and one embodiment of the present invention is a gastroscopy image diagnostic support system which includes a computing system, the computing system including a receiving module, memory or database, processor and user display. The receiving module receives or receives medical images, and the memory or database stores at least one medical image analysis algorithm having a diagnostic function for medical images (gastric endoscopy images).

[0030] The processor analyzes video frames of the gastroscopy image using at least one medical image analysis algorithm to detect whether there are areas suspected to be lesions within the video frames. If there are areas suspected to be lesions within the video frames, the processor calculates the position coordinates of the suspected lesions. The processor then generates display information that includes whether or not there are areas suspected to be lesions and the position coordinates of the suspected lesions.

[0031] If a suspected lesion exists within a video frame, the user display will visually distinguish the suspected lesion on the video frame based on the display information, and will display the position coordinates of the suspected lesion in a way that visually links them to the suspected lesion.

[0032] The processor can calculate the location of the suspected lesion along the gastroscopy path, and can generate display information including whether or not the suspected lesion is present, the position coordinates of the suspected lesion, and the location of the suspected lesion along the gastroscopy path.

[0033] The user display can show the location of a suspected lesion along the gastroscopy route, based on the display information, in a way that visually links it to the suspected lesion.

[0034] The processor can track the position of the video frame displaying the currently examined area along the gastroscopy path, and can calculate the position of the suspected lesion along the gastroscopy path based on the position of the video frame along the gastroscopy path and the position coordinates of the suspected lesion.

[0035] The processor can calculate the location of suspected lesions along the gastroscopy path based on pre-examination medical images, including the three-dimensional anatomical structure of the patient being examined.

[0036] An AI-based medical video (gastroscopy video) analysis algorithm can be trained using each video frame along with a label containing the display of the detected lesion, the position coordinates of the lesion within the video frame, and the position of the lesion along the gastroscopy path, as training data. Therefore, the processor can use the medical video analysis algorithm to calculate the position of suspected lesions along the gastroscopy path within the video frame.

[0037] In an embodiment of the present invention, a receiving module can receive or be inputted at least one gastroscopy video from at least one gastroscopy video acquisition module. Here, the processor can use at least one medical video analysis algorithm to detect whether there is a suspected lesion in each of the video frames of at least one gastroscopy video. The processor can generate display information for each of the video frames of at least one gastroscopy video, including whether or not there is a suspected lesion and the position coordinates of the suspected lesion.

[0038] A gastroscopy image diagnostic support method according to one embodiment of the present invention is performed by a gastroscopy image diagnostic support system including a processor and a user display, and can utilize at least one medical image analysis algorithm having an analysis function for gastroscopy stored in memory or a database within the gastroscopy image diagnostic support system.

[0039] The method of the present invention includes the steps of: receiving or inputting gastroscopy images; a processor analyzing video frames of the gastroscopy images using at least one medical image analysis algorithm to detect whether there are areas suspected to be lesions within the video frames; the processor calculating the position coordinates of the areas suspected to be lesions if such areas exist within the video frames; the processor generating display information including whether or not there are areas suspected to be lesions and the position coordinates of the areas suspected to be lesions; the user display displaying the areas suspected to be lesions on the video frames based on the display information if such areas exist within the video frames; and the user display displaying the position coordinates of the areas suspected to be lesions in a manner that visually links them with the areas suspected to be lesions. [Effects of the Invention]

[0040] According to the present invention, for each real-time video frame of a gastrointestinal endoscopy image, an artificial intelligence algorithm is trained to identify polyps, ulcers, and various gastrointestinal diseases that the user might miss based on the AI ​​medical image diagnosis results. By applying this to an AI diagnostic support system, operational efficiency and diagnostic accuracy can be improved.

[0041] According to the present invention, it is possible to detect lesions early and prevent situations in which they could spread to cancer or other diseases. Since the labels include not only the size of lesions but also their position on the gastrointestinal endoscope and are used as learning data, the present invention not only automatically detects even very small lesions that are easily missed, thereby increasing the lesion detection rate, but it is also possible to extract their position on the gastrointestinal endoscope image path.

[0042] According to the present invention, the lesion detection rate can be increased, risk factors for gastric cancer can be eliminated early, and the incidence of gastric cancer can be reduced. Furthermore, by enabling physicians to find and treat more lesions than before, it can contribute to reducing the causes of gastric cancer and also contribute to reducing the frequency of examinations.

[0043] According to the present invention, diseases that are easily missed by the user during a gastroscopy can be automatically detected and their location along the gastrointestinal pathway (gastric endoscopy pathway) can be shown. This allows the user to review the findings in real time during the gastroscopy, and a report can be easily generated that can be reviewed by other examiners afterward.

[0044] According to the present invention, it is possible to provide optimized content within the artificial intelligence medical image diagnosis results for each real-time video frame of the endoscopic image.

[0045] According to the present invention, it is possible to provide optimized content for each real-time video frame among multiple artificial intelligence medical image diagnostic results.

[0046] According to the present invention, an optimized combination of multiple artificial intelligence medical image diagnostic results can be generated and provided as display information for each real-time video frame.

[0047] According to the present invention, it is possible to provide an optimized combination of multiple artificial intelligence medical video diagnostic results that can efficiently display diagnostic results that are likely to be diagnosed, likely to be missed, or have a high risk in the current video frame.

[0048] According to the present invention, a user interface and diagnostic computing system can be realized that automatically detects and presents diagnostic results that are likely to be diagnosed, likely to be missed, or have a high risk in the current video frame, allowing medical staff to review and consider them in real time during an endoscopic examination. [Brief explanation of the drawing]

[0049] [Figure 1] This figure shows a multi-client structure gastric endoscopy image diagnostic support system and peripheral equipment according to one embodiment of the present invention. [Figure 2] This figure shows a single-client structure gastric endoscopy image diagnostic support system and peripheral equipment according to one embodiment of the present invention. [Figure 3] This figure shows the workflow of a gastric endoscopy image diagnostic support system according to one embodiment of the present invention. [Figure 4] This figure shows an example of a video in which gastroscopy images and display information are displayed together, according to one embodiment of the present invention. [Modes for carrying out the invention]

[0050] In addition to the aforementioned objectives, other objectives and features of the present invention will become apparent from the description of embodiments with reference to the accompanying drawings. Preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the present invention, if it is determined that a specific description of a related known configuration or function may obscure the gist of the invention, such detailed description will be omitted.

[0051] The rapidly developing deep learning / CNN-based artificial neural network technology is being considered for applications in the imaging field, specifically for distinguishing visual elements that are difficult for the human eye to differentiate. The application areas of such technology are expected to expand to diverse fields such as security, medical imaging, and non-destructive testing.

[0052] For example, in the field of medical imaging, cancer cells may not be immediately diagnosed as cancerous in a biopsy state, and it may only be determined whether they are cancerous after follow-up monitoring from a pathological perspective. While it is difficult for the human eye to definitively diagnose whether a cell is cancerous based on medical imaging alone, there is an expectation that applying artificial neural network technology can yield more accurate predictions than observation with the human eye.

[0053] However, even though some studies have shown that artificial neural network technology can produce prediction / classification / diagnosis results superior to the human eye, a problem arises: the lack of descriptive information about the prediction / classification / diagnosis results obtained through the application of artificial neural network technology makes it difficult to incorporate and adopt them in medical settings.

[0054] This invention was derived with the intention of improving the performance of classifying / predicting objects in images that are difficult to classify with the naked eye by applying artificial neural network technology. Furthermore, in order to improve the classification / prediction performance of artificial neural network technology, it is extremely important to obtain descriptive information about the internal operation that leads to the generation of the final diagnostic result based on the classification / prediction process of artificial neural network technology.

[0055] This invention can present performance indicators and quantified clinical usefulness indicators for each of multiple medical image analysis algorithms based on artificial neural networks. This allows for the provision of descriptive information about the process by which the final diagnostic result is derived based on the classification / prediction process of the artificial neural network, and provides a reference for whether human users will adopt the classification / prediction / diagnosis results of the artificial neural network.

[0056] When conventional artificial neural networks are applied to the diagnosis of medical images, they may overfit to a given task, resulting in high statistical accuracy but low accuracy at some clinically important diagnostic points. A significant number of conventional artificial neural networks are in this situation, leading to frequent situations where clinicians have difficulty trusting the diagnostic results of medical images to which these networks have been applied. This danger is even more evident from the fact that IBM's Watson solution, a widely known artificial neural network, has exposed problems such as overfitting to patient racial information included in the trained data, resulting in a significant decrease in accuracy with new patient datasets of different racial groups.

[0057] Therefore, it is crucial to maximize the excellent diagnostic potential of artificial neural networks, provide quantifiable indicators for clinicians to accept such diagnostic results, and ensure pathways that provide direct feedback to clinicians in generating such quantifiable indicators.

[0058] The aforementioned U.S. Published Patent No. 2018 / 0253839, “A system and method for detection of suspicious tissue regions in an endoscopic procedure,” International Published Patent No. WO2018 / 015414, “METHOD AND SYSTEM FOR ARTIFICIAL INTELLIGENCE BASED MEDICAL IMAGE SEGMENTATION,” and Korean Registered Patent No. 10-1938992, “CAD system and method for generating diagnostic reasoning explanations,” disclose basic configurations for AI-based diagnosis of endoscopic images, namely, an endoscopic image acquisition module, an image capture and image processing module, a transmission / reception interface (module) for transmitting acquired / captured endoscopic images to a computing system equipped with an analysis engine, and a memory or database where AI / artificial neural network-based image analysis algorithms / engines are stored.

[0059] In this invention, the basic concepts and structure of data storage means, calculation means, and artificial neural networks, as well as the transmission and reception interface for transmitting input data (images), which can be clearly understood from the prior art and the prior art in this field, are necessary for the realization of the invention. However, a detailed explanation of these basic matters may obscure the gist of the invention. Therefore, within the structure of this invention, any information that became publicly known to those skilled in the art before the filing of this application will be explained in this specification as part of the structure of the invention as necessary. However, if it is determined that facts that are obvious to those skilled in the art may obscure the gist of the invention, the explanation may be omitted.

[0060] Furthermore, any matters omitted in this specification may be explained by informing those skilled in the art that they have become publicly known through prior art referenced in this application, such as U.S. Published Patent No. 2018 / 0253839, “A system and method for detection of suspicious tissue regions in anendoscopic procedure,” International Published Patent No. WO2018 / 015414, “METHOD AND SYSTEM FOR ARTIFICIAL INTELLIGENCE BASED MEDICAL IMAGE SEGMENTATION,” and Korean Registered Patent No. 10-1938992, “CAD system and method for generating diagnostic reasoning.”

[0061] Below, a medical image diagnostic support device and method according to one embodiment of the present invention will be described in detail with reference to Figures 1 to 4.

[0062] Figure 1 shows a multi-client structure gastric endoscopy image diagnostic support system and peripheral equipment according to one embodiment of the present invention.

[0063] The first gastroscopy image acquisition module 132 can either transmit gastroscopy images acquired in real time to the artificial intelligence workstation 120 in real time, or transmit captured images of gastroscopy images to the artificial intelligence workstation 120.

[0064] The second gastroscopy image acquisition module 134 can either transmit gastroscopy images acquired in real time to the artificial intelligence workstation 120 in real time, or transmit captured images of gastroscopy images to the artificial intelligence workstation 120.

[0065] Here, the artificial intelligence workstation 120 may include an input / receive interface module (not shown) that receives gastrointestinal images (or captured images) from the first gastrointestinal image acquisition module 132 and the second gastrointestinal image acquisition module 134.

[0066] The artificial intelligence workstation 120 can transmit received / acquired video frames of gastroscopy images to the artificial intelligence server 110. Here, the images transmitted to the artificial intelligence server 110 can be in a standardized format such as JPEG or MPEG. The artificial intelligence server 110 may also include an input / receive interface module (not shown) that receives video frames in a standardized format.

[0067] The artificial intelligence server 110 can detect and determine lesions within video frames / images using an artificial intelligence algorithm (medical image analysis algorithm) 112.

[0068] A processor (not shown) within the artificial intelligence server 110 inputs given video data into the artificial intelligence algorithm 112, receives the analysis results of the artificial intelligence algorithm 112, and in this process can control the data transfer process between the processor and the memory or storage (not shown) where the artificial intelligence algorithm 112 is stored.

[0069] The output interface module (not shown) of the artificial intelligence server 110 can transmit the analysis results of the artificial intelligence algorithm 112 to the artificial intelligence workstation 120. The information transmitted may include whether a suspected lesion was detected in the video frame, the coordinates where the suspected lesion was detected, the probability that the suspected lesion is a lesion, and the location of the suspected lesion on the gastrointestinal or gastroscopy pathway.

[0070] The artificial intelligence workstation 120 can display the analysis results of the artificial intelligence algorithm 112 on the user display 122. The information displayed on the user display 122 may include whether a suspected lesion has been detected within the video frame of the currently displayed gastroscopy image, visualization of the suspected lesion within the video frame so that it is visually separated (e.g., highlighted, displayed in box form), the position coordinates of the suspected lesion within the video frame, and the position of the suspected lesion on the gastrointestinal tract or gastroscopy path.

[0071] Here, the user display 122 can simultaneously display multiple gastroscopy images in real time. One gastroscopy video frame can be displayed individually in each window on the screen.

[0072] If a user can immediately identify and address a potential lesion using the gastroscopy images during a real-time examination, there are no major problems. However, if a user misses a potential lesion during a real-time examination, it has been virtually impossible to re-diagnose that missed lesion using conventional technology.

[0073] In this invention, captured video frames can be reviewed by other users at a later date, and even if the endoscope has already advanced, previous video frames can be recalled to re-diagnose any missed lesion risk areas. Furthermore, by providing the position coordinates of the risk area within the video frame and its position along the gastroscopy path after the endoscopic examination is completed, the location of the risk area can be identified, enabling subsequent measures to be taken.

[0074] In Figure 1, for the sake of explanation, an embodiment is shown in which the artificial intelligence workstation 120 and the artificial intelligence server 110 are separated. However, this is only one embodiment of the present invention, and it will be obvious to those skilled in the art that, according to other embodiments of the present invention, the artificial intelligence workstation 120 and the artificial intelligence server 110 can be implemented while remaining coupled within a single computing system.

[0075] Figure 2 shows a single-client structure gastric endoscopy image diagnostic support system and peripheral equipment according to one embodiment of the present invention.

[0076] The gastroscopy image acquisition module 232 can either transmit gastroscopy images acquired in real time to the artificial intelligence workstation 220 in real time, or transmit captured images of gastroscopy images to the artificial intelligence workstation 220.

[0077] Here, the artificial intelligence workstation 220 may include an input / receive interface module (not shown) that receives gastrointestinal video (or captured video) from the gastrointestinal video acquisition module 232.

[0078] The artificial intelligence workstation 220 can detect and identify lesions within video frames / images using an artificial intelligence algorithm (medical image analysis algorithm) 212.

[0079] A processor (not shown) in the artificial intelligence workstation 220 inputs given video data to the artificial intelligence algorithm 212, receives the analysis results of the artificial intelligence algorithm 212, and in this process can control the data transfer process between the processor and the memory or storage (not shown) where the artificial intelligence algorithm 212 is stored.

[0080] The output interface module (not shown) of the artificial intelligence workstation 220 can generate the analysis results of the artificial intelligence algorithm 212 as display information and transmit the display information to the user display 222. The information transmitted may include whether a suspected lesion was detected in the video frame, the coordinates where the suspected lesion was detected, the probability that the suspected lesion is a lesion, and the location of the suspected lesion on the gastrointestinal or gastroscopy route.

[0081] The artificial intelligence workstation 220 can display the analysis results of the artificial intelligence algorithm 212 on the user display 222. The information displayed on the user display 222 may include whether a suspected lesion was detected within the video frame of the currently displayed gastroscopy image, a visualization that visually distinguishes the suspected lesion within the video frame (e.g., highlighting, displaying the suspected lesion in a box), the position coordinates of the suspected lesion within the video frame, and the position of the suspected lesion on the gastrointestinal tract or gastroscopy path.

[0082] Figure 3 shows the workflow of a gastric endoscopy image diagnostic support system according to one embodiment of the present invention.

[0083] A gastrointestinal endoscopy image diagnostic support system according to one embodiment of the present invention includes a computing system, the computing system includes a receiving module, memory or database, processor, and user display. The receiving module receives or receives medical images, and the memory or database stores at least one medical image analysis algorithm 312 having a diagnostic function for medical images (gastric endoscopy images).

[0084] The gastroscopy image acquisition module 332 can transmit the gastroscopy images acquired in real time to the gastroscopy image diagnostic support system, or the gastroscopy image capture module 334 can capture the gastroscopy images and transmit them to the gastroscopy image diagnostic support system.

[0085] The processor can perform image processing 320, including cropping to remove black edges from gastrointestinal images and / or captured images, rotation / tilting, and correction of image brightness values.

[0086] The processor analyzes video frames of the gastroscopy image using at least one medical image analysis algorithm 312 to detect whether there are areas suspected to be lesions within the video frames. If there are areas suspected to be lesions within the video frames, the processor calculates the position coordinates of the areas suspected to be lesions. The processor then generates an analysis result 314 that includes whether or not there are areas suspected to be lesions and the position coordinates of the areas suspected to be lesions. Based on the analysis result 314, the processor generates display information that will be displayed together with the gastroscopy image.

[0087] The user display displays the analysis results 314 along with the gastroscopy image (322). Specifically, if there is a suspected lesion within the video frame, the user display displays the suspected lesion on the video frame in a way that visually distinguishes it based on the display information (322), and displays the position coordinates of the suspected lesion in a way that visually links them to the suspected lesion (322).

[0088] The processor can calculate the location of the suspected lesion on the gastroscopy path and generate display information including the presence or absence of the suspected lesion, the position coordinates of the suspected lesion, and the location of the suspected lesion on the gastroscopy path. Here, the processor can calculate the location of the suspected lesion on the gastroscopy path based on information from sensors on the gastroscopy equipment and / or the analysis results 314 of the artificial intelligence algorithm 312.

[0089] The user display can show the location of the suspected lesion on the gastroscopy route based on the display information, so as to be visually linked to the suspected lesion (322).

[0090] The processor can track the position of the video frame displaying the currently examined area along the gastroscopy path, and can calculate the position of the suspected lesion along the gastroscopy path based on the position of the video frame along the gastroscopy path and the position coordinates of the suspected lesion.

[0091] The processor can calculate the location of suspected lesions along the gastroscopy path based on pre-examination medical images, including the three-dimensional anatomical structure of the patient being examined.

[0092] The user can complete the endoscopic examination process by making a final confirmation of the display information shown along with the endoscopic image (324), and by acknowledging the lesion risk area in the display information as a lesion, denying it as not being a lesion, or, if acknowledged as a lesion, taking subsequent measures for the lesion, or by creating a report to take further measures.

[0093] The AI-based medical video (gastroscopy video) analysis algorithm 312 can be trained using each video frame along with a label containing the display of the detected lesion, the position coordinates of the lesion within the video frame, and the position of the lesion on the gastroscopy path as training data. Therefore, the processor can use the medical video analysis algorithm 312 to calculate the position of suspected lesions on the gastroscopy path within the video frame and provide it as an analysis result 314.

[0094] In embodiments of the present invention, the main means for identifying the current position on the path shown by the endoscopic image can primarily rely on learning and inference from the endoscopic image.

[0095] Here, by relying on learning and inference from endoscopic images to identify the current position on the (gastroscopy or colonoscopy) path, the labels of the endoscopic images used for learning can include, separately for each frame, the position of the endoscope (gastroscopy or colonoscopy) on the path and information on detected / confirmed lesions (images actually confirmed by biopsy).

[0096] In another embodiment of the present invention, learning and inference from endoscopic images are the main means for identifying the current position on the colonic pathway. However, by additionally combining this with a means for estimating the progression speed of endoscopic image frames through image analysis to identify the current position, the current position can be identified more accurately.

[0097] Furthermore, while it is generally difficult to take CT images before endoscopic procedures and the current position within the colonic pathway must be identified solely by relying on endoscopic images, if it is possible to take CT images before endoscopic procedures, in other embodiments of the present invention, the current position can also be identified in relation to a 3D model of the endoscopic examination target (such as the stomach or colon) that is reconstructed based on CT images taken before the procedure.

[0098] Here, a 3D model of the colon based on CT images can be realized by combining it with the applicant's existing patents, such as virtual endoscopic imaging technology (Korean Registered Patent No. 10-1850385 or Korean Registered Patent No. 10-1230871).

[0099] Furthermore, in other embodiments of the present invention, when identifying the current position on the endoscopic path within an endoscopic examination target (such as the stomach or large intestine), the method does not rely solely on image analysis, but can also be corrected (compensated) in relation to a sensor provided on the endoscope or endoscopic device (a sensor capable of detecting the length of time the endoscope has been inserted into the human body).

[0100] In an embodiment of the present invention, a receiving module can receive or be inputted at least one gastroscopy video from at least one gastroscopy video acquisition module. Here, the processor can use at least one medical video analysis algorithm 312 to detect whether there is a suspected lesion in each of the video frames of at least one gastroscopy video. The processor can generate display information for each of the video frames of at least one gastroscopy video, including whether there is a suspected lesion and the position coordinates of the suspected lesion.

[0101] Another embodiment of the present invention provides a medical image diagnostic support method that is executed by a processor within a diagnostic support system (computing system) that assists in the diagnosis of medical images, and is executed based on program instructions loaded into the processor.

[0102] A gastroscopy image diagnostic support method according to one embodiment of the present invention is performed by a gastroscopy image diagnostic support system including a processor and a user display, and can utilize at least one medical image analysis algorithm having an analysis function for gastroscopy stored in memory or a database within the gastroscopy image diagnostic support system.

[0103] The method of the present invention includes the steps of: receiving or inputting gastroscopy images; a processor analyzing video frames of the gastroscopy images using at least one medical image analysis algorithm to detect whether there are areas suspected to be lesions within the video frames; the processor calculating the position coordinates of the areas suspected to be lesions if such areas exist within the video frames; the processor generating display information including whether or not there are areas suspected to be lesions and the position coordinates of the areas suspected to be lesions; a user display displaying the areas suspected to be lesions on the video frames based on the display information if such areas exist within the video frames; and the user display displaying the position coordinates of the areas suspected to be lesions in a manner that visually links them with the areas suspected to be lesions.

[0104] Herein, the method of the present invention may further include the step of the processor calculating the location of the suspected lesion on the gastroscopy path.

[0105] The step of generating the display information in the method of the present invention allows the processor to generate the display information including whether or not there is a suspected lesion, the position coordinates of the suspected lesion, and the position of the suspected lesion on the gastroscopy path.

[0106] The step in the method of the present invention in which the user display displays the position coordinates of the suspected lesion in a manner that visually links them with the suspected lesion, means that the user display can display the position of the suspected lesion on the gastroscopy path in a manner that visually links it with the suspected lesion, based on the display information.

[0107] The step of receiving or inputting gastroscopy images in the method of the present invention may involve receiving or inputting at least one gastroscopy image from at least one gastroscopy image acquisition module.

[0108] The step of detecting whether a suspected lesion exists within the video frame of the method of the present invention can be performed by using the at least one medical video analysis algorithm to detect whether a suspected lesion exists in each of the at least one video frame of the gastrointestinal endoscopy image.

[0109] The step of generating the display information in the method of the present invention can generate the display information for each of the video frames of at least one gastroscopy image, including whether or not there is a suspected lesion and the position coordinates of the suspected lesion.

[0110] Figure 4 shows an example of an image in which gastric endoscopy images and display information are displayed together, according to one embodiment of the present invention. The display information may include whether or not a suspected lesion is present, the position coordinates of the suspected lesion (position coordinates within the current video frame), and the position of the suspected lesion along the gastroscopy path.

[0111] In the video frame of the gastroscopy image, areas suspected of being lesions are visualized to be visually distinguishable from other parts of the video frame, as shown in Figure 4. Here, as shown in Figure 4, these areas can be represented by visualization elements such as markers / boxes, or they can be represented by highlighting.

[0112] Furthermore, location information and the probability that a lesion is actually a lesion (a probability inferred by artificial intelligence) are included in the display information, and can be visualized so that the user can intuitively understand the proximity or relationship to the visualization element of the area suspected of being a lesion.

[0113] In the learning process of the artificial intelligence analysis algorithm for gastrointestinal endoscopy images according to one embodiment of the present invention, the training input data includes the following. The gastrointestinal endoscopy images used as training input data consist of images with black backgrounds of different sizes, depending on the resolution supported by the image acquisition device (gastric endoscopy image acquisition module). In order to use only gastrointestinal endoscopy image information, endoscopic partial extraction (cropping) is performed prior to carrying out the learning. During the learning stage, the position on the gastrointestinal endoscopy path is learned together (included in the label information) along with information for detecting lesions, etc., and finally, learning can be carried out for the coordinate values ​​where lesions etc. are located, the probability of lesions, and the position results on the path.

[0114] After training, during the inference process for real-time video, the analyzed results are displayed on the user's screen using visually distinguishable visualization elements. To further reduce the user's risk of leakage, an alarm sound can be used to attract the user's attention when a hazardous area is detected. Different alarm sounds can be used to further concentrate the user's attention depending on the type of hazardous area, the probability that the hazardous area is a lesion, and whether the hazardous area is located in a blind spot in the field of view, resulting in a high risk of leakage.

[0115] When generating training data, data augmentation can be performed to eliminate overfitting caused by specific biases in the data (such as hue, brightness, resolution, or tilting of endoscopic equipment). Data augmentation can be achieved by rotating / tilting, translating, symmetricalizing, or correcting hue / brightness / resolution of image data.

[0116] Furthermore, various methods can be used to prevent overfitting, such as weighted value regulation, dropout addition, and network capacity adjustment (reduction).

[0117] In the embodiments shown in Figures 1 to 4, a real-time image acquisition module acquires real-time endoscopic images from an endoscopic image diagnostic acquisition module / endoscopic equipment. The real-time image acquisition module transmits the real-time endoscopic images to a diagnostic support system. The diagnostic support system includes at least two artificial intelligence algorithms and applies at least two artificial intelligence algorithms to the real-time endoscopic images to generate display information containing diagnostic information. The diagnostic support system transmits the display information to a user system, which can either overlay the display information on the real-time endoscopic images or display the real-time endoscopic images and the display information together.

[0118] Real-time endoscopic images can be divided into individual image frames. These endoscopic image frames can be received or input by a receiving module.

[0119] The diagnostic support system (computing system) includes a receiving interface module, a processor, a transmission interface module, and memory / storage. The processor includes submodules whose functions are internally embodied by hardware or software. The processor may include a first submodule that extracts context-based diagnostic requirements, a second submodule that selects artificial intelligence analysis results to be displayed in the diagnostic results generated by applying an artificial intelligence diagnostic algorithm to endoscopic image frames, and a third submodule that generates display information to be displayed on the user system screen.

[0120] Multiple artificial intelligence diagnostic algorithms are stored in memory or a database (not shown) within the diagnostic computing system and can be applied to endoscopic image frames under the control of the processor to generate diagnostic results for the endoscopic image frames.

[0121] In the embodiments shown in Figures 1 to 4, multiple artificial intelligence diagnostic algorithms are stored in memory or a database (not shown) inside the diagnostic computing system and driven by the control of a processor. However, in other embodiments of the present invention, multiple artificial intelligence diagnostic algorithms may also be stored in memory or a database (not shown) outside the diagnostic computing system. When multiple artificial intelligence diagnostic algorithms are stored in memory or a database (not shown) outside the diagnostic computing system, the processor can control the memory or database (not shown) outside the diagnostic computing system via a transmission module so that the multiple artificial intelligence diagnostic algorithms are applied to endoscopic image frames and diagnostic results are generated for the endoscopic image frames. The generated diagnostic results are then transmitted to the diagnostic computing system via a receiving module, and the processor can generate display information based on the diagnostic results.

[0122] The processor analyzes endoscopic video frames, which are video frames of medical images, to extract diagnostic requirements for the endoscopic video frames. Based on the diagnostic requirements, the processor selects multiple diagnostic application algorithms from among multiple medical image analysis algorithms to perform a diagnosis on the endoscopic video frames. The processor then applies these diagnostic application algorithms to the endoscopic video frames to generate display information containing the diagnostic results for the endoscopic video frames. This process is performed by the processor for each endoscopic video frame.

[0123] The processor can analyze endoscopic image frames and extract contextually relevant diagnostic requirements corresponding to the characteristics of the endoscopic image frames. Based on these contextually relevant diagnostic requirements, the processor can select multiple diagnostic application algorithms to perform diagnoses on the endoscopic image frames.

[0124] The processor can select a combination of multiple diagnostic application algorithms based on contextual diagnostic requirements. The processor can apply the combination of multiple diagnostic application algorithms to the endoscopic image frame to generate display information containing the diagnostic results for the endoscopic image frame.

[0125] A combination of multiple diagnostic application algorithms may include a first diagnostic application algorithm that is preferentially recommended for endoscopic image frames based on contextually-based diagnostic requirements, and a second diagnostic application algorithm that is recommended based on complementary diagnostic requirements derived within the contextually-based diagnostic requirements based on the characteristics of the first diagnostic application algorithm.

[0126] Contextual diagnostic requirements may include at least one of the following: the body part of the human body included in the endoscopic image frame, the organ of the human body, the relative position of the endoscopic image frame within the organ of the human body, the probability of a lesion occurring in relation to the endoscopic image frame, the risk of a lesion in relation to the endoscopic image frame, the difficulty of identifying a lesion in relation to the endoscopic image frame, and the type of target lesion. Once the organ to which the endoscopic image frame relates is identified, for example, if the endoscopic image frame relates to a colonoscopy image, information about whether the image currently displayed in the image frame is the beginning, middle, or end of the colonoscopy image can be identified along with its relative position within the colon (entrance, middle, or end of the organ). In the case of a gastroscopy image, information about whether the image currently displayed in the image frame is the beginning (e.g., esophagus), middle (entrance to the gastrointestinal tract), or end of the gastroscopy image can be identified along with its relative position along the gastroscopy path.

[0127] This allows for the extraction of context-based diagnostic requirements based on the types of lesions / diseases most likely to occur at the identified location and site, the types of lesions / diseases that are not easily identifiable with the naked eye and are likely to be missed by medical personnel, diagnostic information for lesions / diseases that are not easily identifiable visually within the current video frame, and the types of lesions / diseases that are high in risk / mortality and require attention among those that may occur at the location within the human organs shown in the current video frame. Here, context-based diagnostic requirements may also include information about the types of target lesions / diseases that should be given priority consideration in relation to the current video frame based on the information described above.

[0128] The display information may include endoscopic image frames, diagnostic results selectively overlaid on the endoscopic image frames, information on the diagnostic application algorithm that generated the diagnostic results, and an evaluation score for the diagnostic application algorithm. The process for calculating the evaluation score for the diagnostic application algorithm can be the same as the process for calculating the evaluation score in the embodiments shown in Figures 1 and 2 described above.

[0129] While artificial intelligence diagnostic algorithms can apply prioritization when applying diagnoses in order of highest evaluation score, there are several additional variables that need to be considered.

[0130] If the first-priority AI algorithm detects only a portion of the possible lesions in the endoscopic image, and the second-priority AI algorithm detects what the first-priority algorithm failed to detect, the diagnostic results of both the first-priority and second-priority AI algorithms can be displayed together. Furthermore, a menu can be provided that allows the user to select the final AI algorithm to apply based on these criteria. To assist the user in their selection, the diagnostic results of multiple AI algorithms and explanations of why those results are displayed together can also be shown.

[0131] For example, suppose lesions A1 and A2 are known as the most likely types of lesions to occur within the current video frame, and lesion B is known as a lesion that is less likely to occur than A1 and A2 but is difficult to identify visually and may be missed. The artificial intelligence diagnostic algorithm X that obtains the highest evaluation score for lesions A1 and A2 can be selected as the first diagnostic algorithm to be recommended preferentially, as it will have the highest overall evaluation score. Here, it is possible that the first diagnostic algorithm obtains the highest evaluation score for lesions A1 and A2, but obtains an evaluation score below the standard value for lesion B. Here, lesion B, in which the first diagnostic algorithm performs below the standard value, can be designated as a complementary diagnostic requirement. The second diagnostic algorithm can be selected as the artificial intelligence diagnostic algorithm Y that obtains the highest evaluation score for lesion B, which is the complementary diagnostic requirement. The combination of the first and second diagnostic algorithms can be selected such that the combination has a high evaluation score for overall diagnostic information reliability, accuracy, etc., while not missing diagnostic information for specific lesions / diseases or having weak diagnostic performance for specific lesions / diseases. Therefore, the second diagnostic application algorithm can be designed with logical conditions for selecting a diagnostic application algorithm such that the AI ​​diagnostic application algorithm that best performs in the complementary diagnostic requirements where the first diagnostic application algorithm is weak is selected, rather than an AI diagnostic application algorithm with uniformly high overall evaluation scores.

[0132] In the above embodiment, we gave an example of a case where two diagnostic application algorithms are selected. However, if a combination of three or more diagnostic application algorithms shows better performance based on the evaluation score, an embodiment in which three or more diagnostic application algorithms are selected and applied can also be realized by the matters described in this patent application.

[0133] The embodiments shown in Figures 1 to 4 present diagnostic results to which AI diagnostic algorithms with high internal evaluation scores have been applied, allowing the user to select the diagnostic result to which the AI ​​diagnostic algorithm with the highest evaluation score has been applied. The embodiments in Figures 1 to 4 disclose a configuration derived with the aim of displaying diagnostic results quickly for real-time endoscopic images. Therefore, in the embodiments in Figures 1 to 4, after preferentially selecting combinations of AI diagnostic algorithms to be displayed for the currently displayed video frame based on contextually relevant diagnostic requirements, the diagnostic results for these combinations are generated as display information and provided to the user along with the video frame.

[0134] Here, context-based diagnostic requirements can include the types of lesions / diseases that are most likely to occur in the current video frame, the types of lesions / diseases that are likely to occur in the current video frame but are difficult to identify visually and are likely to be missed by medical staff, and the types of lesions / diagnoses that are high in risk / mortality and require attention among the lesions / diagnoses that may occur in the current video frame. In addition, context-based diagnostic requirements can include the types of target lesions / diseases that must not be missed in the current video frame, and the priority of target lesions / diseases, based on the type and characteristics of the lesions / diseases.

[0135] When used in a hospital, the diagnostic result derivation and display information of the present invention is displayed by a user system having a user interface that can display artificial intelligence diagnostic support results after receiving and analyzing endoscopic data. Based on user input, the system can decide whether to confirm the diagnostic result, change the diagnostic result, or accept or reject the diagnostic result.

[0136] The processor can concatenate the display information with the endoscopic image frame and store it in the database. Here, the database may be an internal database of the diagnostic computing system and may be stored therein as a medical record for the patient.

[0137] The processor can generate external storage data in which the display information and endoscopic image frames are concatenated, and can transmit the external storage data to an external database via a transmission module so that it is stored in the external database. Here, the external database can be a PACS database or a database implemented based on the cloud.

[0138] Here, the multiple medical image analysis algorithms are artificial intelligence algorithms that use an artificial neural network, and the processor can generate evaluation scores based on diagnostic requirements / contextual diagnostic requirements as descriptive information for each of the multiple medical image analysis algorithms.

[0139] The diagnostic support system of the present invention may internally include at least two or more artificial intelligence diagnostic algorithms. Endoscopic video data is transmitted to the diagnostic support system from three or more endoscopes. The diagnostic support system applies at least two or more artificial intelligence diagnostic algorithms to each frame of the endoscopic video data to generate a diagnostic result. The diagnostic support system generates display information by concatenating the diagnostic results for each frame of the endoscopic video data. Here, the display information may be generated including identification information of the hospital where the endoscopic video data was generated (Hospital A). The display information may also be generated including identification information given to each endoscopic device in each hospital (Endoscope 1, Endoscope 2, Endoscope 3).

[0140] The system of the present invention transmits the generated display information to a cloud-based database, where the endoscopic equipment that generated the endoscopic image data and the hospital that generated it are identified, and the endoscopic image data and display information are stored. The display information is generated and stored by linking diagnostic information to each frame of the endoscopic image data. The diagnostic information generated for each frame of the endoscopic image data can be automatically generated based on evaluation scores and contextual diagnostic requirements, as described in the embodiments of Figures 1 to 4.

[0141] When the present invention is applied in a cloud environment, endoscopic image data and diagnostic results can be received by the hospital's user system using equipment connected via a wireless communication network, and the artificial intelligence diagnostic support results can be displayed on the user system.

[0142] Display information stored in a cloud database can be provided to a hospital designated by the patient, allowing the patient to receive their endoscopic image data and diagnostic information at a hospital convenient to them, and to receive interpretation of the diagnosis and subsequent diagnoses from a doctor at that hospital.

[0143] In conventional medical diagnostic computer terminals, diagnostic results are generated using the results of an artificial intelligence algorithm. During the process of generating these diagnostic results, comments from the medical team can be added.

[0144] In a medical image diagnostic support device / system to which the present invention is applied, the evaluation score, I-Score, is transmitted from the computing system to the medical team's diagnostic computer terminal. The evaluation score, I-Score, is reflected in the generation of the diagnostic result, and the final diagnostic statement can be generated. In one embodiment of the present invention, the computing system can generate a diagnostic statement along with the evaluation score, I-Score, and transmit it to the medical team's computing system. Here, the diagnostic statement generated by the computing system can be created using a diagnostic result based on a diagnostic application algorithm with a high evaluation score, I-Score.

[0145] The computing system can select recommended diagnostic results using an internally calculated evaluation score called the I-Score, and since the evaluation score is displayed along with the recommended diagnosis, it can provide a user interface that allows the radiologist to evaluate / confirm the diagnostic confidence of the recommended diagnosis (for example, a recommended diagnostic algorithm that matches the radiologist's diagnosis). The computing system's processor can select a first and second diagnostic result as recommended diagnostic results from among multiple diagnostic results based on the evaluation score. The processor can generate display information including the evaluation score for the first diagnostic algorithm, the first diagnostic result, the evaluation score for the second diagnostic algorithm, and the second diagnostic result.

[0146] The computing system can generate an evaluation score based on the confidence score of the diagnostic algorithm, the accuracy score of the diagnostic algorithm, and the evaluation confidence score of the diagnostician who provided the feedback. The processor can generate confidence scores for each of the multiple medical image analysis algorithms, accuracy scores for each of the multiple medical image analysis algorithms, and the user's evaluation confidence score for each of the multiple medical image analysis algorithms as sub-evaluation items based on multiple diagnostic results and user feedback on the multiple diagnostic results, and generate the evaluation score based on the sub-evaluation items.

[0147] For example, the criteria for generating evaluation scores can be embodied as follows: (Equation 1) I-Score = ax (confidence score of the AI ​​algorithm) + bx (accuracy score of the AI ​​algorithm) + cx (confidence score of the diagnostician's evaluation of the AI ​​algorithm)

[0148] The confidence score for each algorithm can be assigned by the diagnosing physician. That is, if the first diagnostic result is judged to be more accurate than the second diagnostic result, a higher confidence score can be assigned to the first diagnostic result.

[0149] The accuracy score of an algorithm can be determined by the extent to which the diagnostician accepts the diagnostic results of each algorithm, without a separate scoring process. For example, if the first diagnostic result presents 10 suspected lesion locations, but the diagnostician accepts 9 of them, the accuracy score can be given as 90 / 100.

[0150] In yet another embodiment where algorithm accuracy scores are assigned, it can be assumed that accurate results are obtained through a biopsy or the like. In this case, the accuracy of the diagnostic results of each diagnostic algorithm can be shown by comparing them to the accurate results from the biopsy. If the user inputs the accurate results from the biopsy into the computing system, the computing system can calculate the accuracy score of each diagnostic algorithm by comparing each diagnostic result with the accurate results from the biopsy (reference).

[0151] The diagnostic physician's evaluation confidence score can be provided as a confidence score for the diagnostic physician's evaluation. In other words, the more experienced and experienced the diagnostic physician is in the relevant clinical field, the higher the evaluation confidence score will be. The evaluation confidence score can be calculated by considering the diagnostic physician's years of experience, their specialty, whether they are a resident, and their experience in the relevant clinical field.

[0152] The computing system can continuously learn evaluation score calculation criteria through its internal artificial intelligence algorithms and update them according to an internally determined schedule. The processor assigns weights to each of the sub-evaluation items, which are the confidence score for each of the multiple medical image analysis algorithms, the accuracy score for each of the multiple medical image analysis algorithms, and the user's evaluation confidence score for each of the multiple medical image analysis algorithms. Based on multiple diagnostic results and user feedback on those results, the system can update the weights of each sub-evaluation item so that they are adjusted according to the target requirements.

[0153] An example of a target requirement is adjusting the correlation between the user's confidence and accuracy for each algorithm so that they match. For example, a diagnostician's confidence score for the first and second diagnostic algorithms may differ even if they have the same accuracy score. Here, if the confidence scores still differ with a certain trend even after removing the general error of the diagnostician's evaluation, it can be said that the diagnostician's confidence in the first and second diagnostic algorithms is different. For example, if the first and second diagnostic algorithms both produce accurate diagnostic results for 9 out of a total of 10 suspected lesion locations, resulting in an accuracy score of 90 / 100, but only the first diagnostic algorithm accurately diagnoses the severity of the lesion, while the second diagnostic algorithm fails to do so, then the reason for the difference in the diagnostician's confidence can be explained. Possible means of adjusting the correlation between accuracy and confidence to match include adjusting the weighting values ​​for each sub-item or subdividing the selection criteria for target lesions for accuracy assessment. In this case, possible methods include classifying lesions based on criteria such as the mildness / severity of the diagnosed lesion, its position relative to the center of the medical image, and the difficulty of identifying the lesion (difficult in areas where bones, organs, and blood vessels are intricately mixed), and then assigning other weightings to the diagnostic accuracy of the lesion in each area.

[0154] The computing system may include a function that allows it to automatically assign a number of applicable artificial intelligence algorithms internally based on the video. To determine which artificial intelligence algorithms are applicable to the video, the computing system can classify one or more videos using a separate video classification artificial intelligence algorithm within the recommendation and diagnostic system, and then apply a number of artificial intelligence algorithms.

[0155] In one embodiment of the present invention, multiple medical image analysis algorithms can utilize an artificial neural network. Here, evaluation scores and sub-evaluation items are generated as descriptive information for each diagnostic algorithm, and the computing system can provide information so that the evaluation scores and sub-evaluation items are fed back to the entity that generates the diagnostic algorithms and used to improve the diagnostic algorithms. Here, in the case of an artificial neural network that uses recently studied relevance scores and confidence levels, the evaluation scores and sub-evaluation items are linked to the relevance score or confidence level of the artificial neural network, and statistical analysis is performed so that the evaluation scores and sub-evaluation items can influence the improvement of the diagnostic algorithms.

[0156] Such an embodiment of the present invention is designed to provide the advantages obtained by the present invention while making the greatest possible changes to the conventional medical imaging diagnostic sequence.

[0157] In yet another embodiment of the present invention, a computing system can independently select a plurality of diagnostic application algorithms and apply each of the plurality of diagnostic application algorithms to medical images to generate a plurality of diagnostic results. In this case, the computing system transmits not only information about the selected diagnostic application algorithms but also the plurality of diagnostic results based on the diagnostic application algorithms to the medical team's diagnostic computer terminal, where the medical team's diagnostic computer terminal can display the results of applying the artificial intelligence algorithms (diagnostic application algorithms) to the medical images.

[0158] In this case, one embodiment of the present invention can provide the advantages obtained by the present invention even when the computing power of the medical team's diagnostic computer terminal is not large, for example, when it is a mobile device or an older computing system. Here, in one embodiment of the present invention, the entity that applies the artificial intelligence algorithm to the medical images is the computing system, the computing system functions as a type of server, and the medical team's diagnostic computer terminal can also operate as a thin-client-based system. Here, in one embodiment of the present invention, the medical team's diagnostic computer terminal can feed back feedback indicators that the medical team has entered for multiple diagnostic results or multiple diagnostic application algorithms to the computing system. The feedback indicators can be linked to each of the evaluation targets, i.e., each of the multiple diagnostic results or multiple diagnostic application algorithms, and stored in memory or a database in the computing system.

[0159] As described above, in one embodiment of the present invention, the step in which the selected algorithm is applied may be performed in a clinician's diagnostic system, and multiple diagnostic results may be transmitted to a computing system. In another embodiment of the present invention, the step in which the selected algorithm is applied may be performed entirely within a computing system, and then the results may be displayed in the clinician's diagnostic system.

[0160] A method for assisting the diagnosis of medical images according to one embodiment of the present invention can be embodied as a program instruction form that can be executed by various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the medium may be specifically designed and configured for the present invention or may be publicly known and usable by those skilled in the computer software art. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

[0161] However, the present invention is not limited to or restricted by the embodiments. The same reference numerals shown in each figure indicate the same component. The lengths, heights, sizes, widths, etc. disclosed in the embodiments and drawings of the present invention may be exaggerated for illustrative purposes.

[0162] As described above, the present invention has been explained based on specific matters such as concrete components and limited embodiments and drawings. However, these are provided only to aid in a more general understanding of the present invention, and the present invention is not limited to the above embodiments. A person with ordinary skill in the art to which the present invention belongs can make various modifications and variations from this description.

[0163] Therefore, the concept of the present invention should not be limited to the embodiments described, and it can be said that not only the claims described later, but also all variations equivalent to or similar to these claims, fall within the scope of the concept of the present invention. [Explanation of Symbols]

[0164] 110 Artificial Intelligence Server (Computing System) 112, 212 Artificial Intelligence Algorithms 120, 220 Artificial Intelligence Workstations (Computing Systems) 122, 222 user displays 132, 134, 232 Gastric Endoscopy Image Acquisition Module

Claims

1. A diagnostic support system that assists in the diagnosis of medical images, wherein the diagnostic support system includes a computing system. The computing system is adjacent A receiving module that receives or inputs gastroscopy images as the aforementioned medical images, A memory or database for storing at least one medical image analysis algorithm having an analysis function for the aforementioned gastroscopy, Processor and User display and Includes, The processor analyzes the video frames of the gastroscopy image using at least one medical image analysis algorithm based on artificial intelligence, and as a result of the analysis, detects whether there is a suspected lesion within the video frame. If the processor finds a suspected lesion within the video frame, it calculates the position coordinates of the suspected lesion within the video frame. The processor tracks the position of the video frame displaying the currently examined area along the gastroscopy path. The processor calculates the position of the suspected lesion on the gastroscopy path based on the position of the video frame on the gastroscopy path and the position coordinates of the suspected lesion. The processor generates display information including the presence or absence of the suspected lesion, the position coordinates of the suspected lesion, and the position of the suspected lesion on the gastroscopy path. The user display, if a suspected lesion exists within the video frame, displays the suspected lesion on the video frame based on the display information so as to be visually separated, and displays the position coordinates of the suspected lesion and its position on the gastroscopy path so as to be visually linked to the suspected lesion. A system for supporting the diagnosis of gastric endoscopy images.

2. The gastroscopy image diagnostic support system according to claim 1, wherein the processor calculates the position of the area suspected of being a lesion on the gastroscopy path based on pre-examination medical images including the three-dimensional anatomical structure of the patient to be examined.

3. The receiving module receives or is input to at least one gastroscopy image from at least one gastroscopy image acquisition module. The processor uses the at least one medical video analysis algorithm to detect whether there is a suspected lesion in each of the at least one video frames of the gastroscopy images, The gastroscopy video diagnostic support system according to claim 1, wherein the processor generates display information for each of the video frames of at least one gastroscopy image, including whether or not there is a suspected lesion and the position coordinates of the suspected lesion within the video frame.

4. A method for supporting gastric endoscopy image diagnosis performed by a gastric endoscopy image diagnosis support system including a processor and a user display, The stage where gastroscopy images are received or input, The process includes: using at least one medical video analysis algorithm based on artificial intelligence having an analysis function for the gastroscopy stored in the memory or database of the gastroscopy video diagnostic support system, the processor analyzes the video frames of the gastroscopy video, and as a result of the analysis, detects whether there is a suspected lesion in the video frames; The processor, if a suspected lesion exists within the video frame, calculates the position coordinates of the suspected lesion within the video frame. The processor tracks the position of the video frame displaying the currently examined area along the gastroscopy path. The processor calculates the position of the suspected lesion on the gastroscopy path based on the position of the video frame on the gastroscopy path and the position coordinates of the suspected lesion; The processor generates display information including whether or not there is a suspected lesion, the position coordinates of the suspected lesion, and the position of the suspected lesion on the gastroscopy path. The user display, if a suspected lesion exists within the video frame, displays the suspected lesion on the video frame based on the display information so as to be visually separated, and displays the position coordinates of the suspected lesion and its position on the gastroscopy path so as to be visually linked to the suspected lesion. A method for supporting gastric endoscopy image diagnosis, including gastric endoscopy.

5. The step of receiving or inputting the gastroscopy image involves receiving or inputting at least one gastroscopy image from at least one gastroscopy image acquisition module, The step of detecting whether there is a suspected lesion in the video frame involves using the at least one medical video analysis algorithm to detect whether there is a suspected lesion in each of the at least one video frame of the gastroscopy image, The gastroscopy video diagnostic support method according to claim 4, wherein the step of generating the display information is to generate the display information for each of the video frames of at least one gastroscopy video, including whether or not there is a suspected lesion and the position coordinates of the suspected lesion within the video frame.

Images