Method for Attention Mechanism-Based Instrument-Aware Endoscopic Lesion Detection and Visual Stabilization, and Computing System Thereof

Abstract

The present invention relates to an attention mechanism-based instrument-responsive endoscopic lesion detection method capable of visual stabilization and a computing system for performing the same. More specifically, the invention relates to an attention mechanism-based instrument-responsive endoscopic lesion detection method capable of visual stabilization and a computing system for performing the same, which detects lesions without separate post-processing by identifying the contextual relationship between a lesion and a medical instrument using an attention mechanism-based neural network model, and corrects and tracks coordinates by applying a speed-sensitive adaptive filter in which the smoothing coefficient is dynamically varied according to the movement speed of the lesion object, thereby improving the reliability of lesion detection even in environments of interference caused by medical instruments, suppressing micro-tremors of objects occurring during the display process, and maximizing visual stability by minimizing tracking delay.

Description

Technology Field

[0001] The present invention relates to an attention mechanism-based instrument-responsive endoscopic lesion detection method capable of visual stabilization and a computing system for performing the same. More specifically, the invention relates to an attention mechanism-based instrument-responsive endoscopic lesion detection method capable of visual stabilization and a computing system for performing the same, which detects lesions without separate post-processing by identifying the contextual relationship between a lesion and a medical instrument using an attention mechanism-based neural network model, and corrects and tracks coordinates by applying a speed-sensitive adaptive filter in which the smoothing coefficient is dynamically varied according to the movement speed of the lesion object, thereby improving the reliability of lesion detection even in environments of interference caused by medical instruments, suppressing micro-tremors of objects occurring during the display process, and maximizing visual stability by minimizing tracking delay. Background Technology

[0002] Gastrointestinal endoscopy is a key diagnostic tool for the early detection of lesions in the gastrointestinal mucosa, playing a crucial role in the early diagnosis of malignant tumors of the digestive system, including gastric and colorectal cancer, and in improving patient survival rates. Recently, driven by advancements in artificial intelligence and computer vision technologies, active research is being conducted on Computer-Aided Detection (CADe) systems that combine object recognition technology with endoscopic images to assist operators in diagnosis.

[0003] Conventional AI-based lesion detection technology for endoscopic image analysis primarily employs object detection models based on Convolutional Neural Networks (CNNs). It operates by estimating the location of lesions using predefined anchor boxes and eliminating duplicate detections through post-processing operations such as Non-Maximum Suppression. However, this CNN-based approach has limitations: its ability to recognize global issues across the entire image is restricted, and the training data is biased toward refined, magnified images of large lesions with excellent visibility. This results in a domain discrepancy with distant and wide-angle images from actual examination environments, leading to a significant decrease in detection sensitivity for clinically important small lesions, such as micropolyps or flat polyps. In addition, the unique characteristics of the endoscopic environment, where medical instruments frequently intervene on the screen, are not fully reflected during the learning phase, making it easy for false or false detections to occur in situations where the instrument is obscured or in close proximity. Furthermore, there is a problem where the visual fatigue of the operator is increased due to the fine shaking and flickering of the bounding box during the display process of detected objects.

[0004] For example, Patent Document 1 discloses a technology that detects target lesions and non-target medical devices from endoscopic images and then forcibly deletes or displays lesion markings through rule-based post-processing. However, this is merely a form that adds rule-based post-processing logic to a convolutional neural network model, so it cannot flexibly respond to various clinical situations and has limitations such as being vulnerable to screen shaking due to a simple frame-by-frame counting method. Additionally, Patent Document 2 discloses a configuration that preprocesses an input image and then sequentially performs a lesion location detection model and a type classification model in a dual manner. However, as the processing steps are separated, bottlenecks and latency occur during the inference process, making it difficult to meet the requirements of an endoscopic examination environment that demands real-time performance of 30fps or more. Furthermore, Patent Document 3 discloses a technology that uses a Generative Adversarial Network (GAN) to generate virtual lesion images and utilize them for training in order to solve the problem of insufficient training data for convolutional neural network-based models. However, artificially generated virtual data cannot perfectly mimic physical contexts such as the fine texture or lighting reflection of actual mucosa, which can cause false positives during actual clinical application. Moreover, the very reliance on such data augmentation demonstrates the limitations of low data efficiency, which structurally requires large-scale training data for convolutional neural network-based approaches.

[0005] As such, conventional technologies focus solely on simple pathological classification or the detection of distinct large lesions, failing to meet the essential clinical demand for the early detection of microscopic lesions that are easily overlooked by operators. Consequently, there is a growing need to grasp the global context of the image to directly predict lesions in real-time without separate post-processing, and to maintain robust detection reliability by adapting to interference from medical instruments. Furthermore, it is necessary to significantly reduce visual fatigue among medical staff by preventing screen jitter and tracking delays. Therefore, there is a need for a new computing system based on an attention mechanism that can reliably detect microscopic lesions and provide visual stability even in environments with instrument interference. Prior art literature

[0006] Korean Registered Patent 10-2607977 ("Device and method for analyzing gastrointestinal endoscopic images using an endoscopic image analysis AI model", published Nov. 30, 2023) Korean Registered Patent 10-2388535 ("Method and device for analyzing endoscopic images based on artificial intelligence", published April 22, 2022) Korean Registered Patent 10-2766249 (Device and method for image learning for lesion detection, published Nov. 29, 2022) The problem to be solved

[0007] The present invention aims to provide a lesion detection method capable of attention mechanism-based instrument-responsive endoscope lesion detection and visual stabilization, and a computing system for performing the same, which detects lesions without separate post-processing by identifying the contextual relationship between a lesion and a medical device using an attention mechanism-based neural network model, and corrects and tracks coordinates by applying a speed-sensitive adaptive filter in which the smoothing coefficient is dynamically varied according to the movement speed of the lesion object, thereby improving the reliability of lesion detection even in environments of interference by medical devices, suppressing micro-tremors of objects occurring during the display process, and maximizing visual stability by minimizing tracking delay. means of solving the problem

[0008] To solve the above problems, one embodiment of the present invention is a method for detecting lesions based on endoscopic images performed in a computing system comprising one or more processors and one or more memories, comprising: an image reception step of receiving a real-time image stream from an endoscopic device; a lesion detection step of detecting a lesion object using a neural network model having an attention mechanism-based encoder-decoder structure for a current frame of the image stream received in the image reception step, thereby deriving a detection result including the location and class of the lesion object; and a tracking update step of updating tracking information for the lesion object based on the detection result of the lesion detection step. The present invention provides a lesion detection method comprising: an overlay display step for overlaying information about the lesion object tracked in the tracking update step onto the image stream and outputting it to a display unit; wherein no separate post-processing step is performed between the lesion detection step and the tracking update step, and the neural network model is a model in which the contextual relationship between the lesion and one or more other objects other than the lesion is learned based on a training dataset in which the lesion and one or more other objects other than the lesion are labeled.

[0009] In some embodiments of the present invention, in the lesion detection step, the neural network model comprises: an encoder that calculates global interactions for features derived from the current frame through an attention operation; and a decoder that predicts the location and class of the lesion object based on a learnable object query, wherein the decoder may not utilize an anchor box.

[0010] In some embodiments of the present invention, the neural network model may be a full-transformer architecture combining a Vision Transformer (ViT)-based backbone and a transformer decoder.

[0011] In some embodiments of the present invention, other objects other than the lesion include medical devices, and the training dataset includes training data in which the medical devices are labeled, so that the neural network model is trained to distinguish the lesion and the medical devices as distinct classes, and when the medical device is identified within the current frame in the lesion detection step, the detection reliability of the lesion object can be adaptively adjusted based on the attention weight between the object query corresponding to the lesion and the object query corresponding to the medical device in the multi-head attention layer of the neural network model.

[0012] In some embodiments of the present invention, the training dataset comprises: a first training dataset in which at least one of a lesion or a medical device is labeled; and a second training dataset in which no label is assigned to an object including a lesion and a medical device, and the ratio of the first training dataset to the second training dataset within the training dataset may be in the range of 7:3 to 9.5:0.5.

[0013] In some embodiments of the present invention, the tracking update step may perform detailed steps of: updating tracking information by associating a lesion object with an existing track or creating a new track, wherein when associating a lesion object with an existing track, a first matching step in which a first detection object having a preset high reliability standard or higher among the detection results is first matched with the existing track; and a second matching step in which, for an existing track not matched in the first matching step, a second detection object having a preset low reliability standard or higher among the detection results is second matched with a second detection object having a high reliability standard or lower than the high reliability standard.

[0014] In some embodiments of the present invention, the overlay display step comprises the original coordinates (X) of the current frame for the lesion object tracked in the tracking update step. t ) and the final corrected coordinates (XC) corrected in the previous frame t-1Using ), the smoothing coefficient (α) according to the movement speed of the lesion object t By applying a velocity-sensitive adaptive filter in which ) is dynamically varied, the final corrected coordinates (XC) of the current frame t A coordinate correction step in which ) is calculated; and a final display step in which a visual identifier including a bounding box indicating the location of the lesion object is generated based on the final corrected coordinates calculated in the coordinate correction step and overlaid on the image stream; are performed, and the velocity-sensitive adaptive filter uses the following [Equation 1] to calculate the final corrected coordinates (XC t It can be a filter that produces ).

[0015] [Equation 1]

[0016]

[0017] In some embodiments of the present invention, the overlay display step comprises the original coordinates (X t ) and the final corrected coordinates (XC) of the previous frame mentioned above t-1 The difference of ) is the time interval between frames ( Divided by t), the raw velocity (v) of the above-mentioned lesion object t A raw velocity calculation step for calculating ); the raw velocity (v t The pre-set velocity smoothing coefficient (α) in ) v By applying ) and the operation of [Equation 2] below, the reference movement speed (V t Further performing a detailed step of a velocity smoothing step that calculates ), and the velocity-sensitive adaptive filter, wherein a smoothing coefficient (α) that is dynamically calculated based on the reference moving speed t ) can be applied.

[0018] [Equation 2]

[0019]

[0020] (here, α v is a preset value between 0 and 1, V t-1 corresponds to the reference movement speed calculated in the previous frame.)

[0021] In some embodiments of the present invention, the smoothing coefficient is calculated based on a cutoff frequency determined in proportion to the magnitude of the moving speed, wherein if the magnitude of the moving speed is smaller than a preset speed standard, the smoothing coefficient (α t Since ) is calculated as a relatively small value, the reflection ratio of the final correction coordinates of the previous frame becomes relatively high, thereby suppressing jittering on the lesion object, and when the magnitude of the movement speed is greater than the speed standard, the smoothing coefficient (α t Since ) is calculated as a relatively large value, the reflection ratio of the original coordinates of the current frame is relatively high, thereby reducing the tracking lag for the lesion object.

[0022] To solve the above-mentioned problem, one embodiment of the present invention comprises a computing system that performs an endoscopic image-based lesion detection method, the computing system comprising one or more processors and one or more memories, the computing system comprising: an image receiving unit that receives a real-time image stream from an endoscopic device; a lesion detection unit that derives a detection result including the location and class of said lesion object by detecting a lesion object using a neural network model having an attention mechanism-based encoder-decoder structure for a current frame of the image stream received by the image receiving unit; and a tracking update unit that updates tracking information for said lesion object based on the detection result of the lesion detection unit. The computing system includes an overlay display unit that overlays information about the lesion object tracked in the tracking update step onto the image stream and outputs it to a display unit; wherein the detection result of the lesion detection unit is transmitted to the tracking update unit without performing separate post-processing, and the neural network model is a model in which the contextual relationship between the lesion and the other object other than the lesion is learned based on a training dataset in which the lesion and one or more other objects other than the lesion are labeled. Effects of the invention

[0023] According to one embodiment of the present invention, the endoscopic image-based lesion detection method and computing system can improve the reliability of lesion detection and reduce the rate of lesion oversight by the operator even in clinical environments where interference by medical devices is frequent, by integrally recognizing the global context of the image and the contextual relationship between objects through an attention mechanism-based neural network model.

[0024] According to one embodiment of the present invention, the computing system adopts a structure in which an image receiving unit, a lesion detection unit, a tracking update unit, and an overlay display unit are organically combined on a single processing pipeline, thereby configuring the system so that no separate post-processing procedure intervenes between the detection stage and the display stage, so as to stably achieve real-time performance of 30fps or more.

[0025] According to one embodiment of the present invention, the neural network model can effectively resolve the problem of real-time performance impairment caused by anchor box dependency and non-maximum inhibition post-processing of conventional convolutional neural network-based methods by directly predicting the location and class of lesion objects based on learnable object queries through a hybrid architecture combining a convolutional neural network-based backbone and a transformer encoder-decoder structure or a vision transformer-based full-transformer architecture.

[0026] According to one embodiment of the present invention, the neural network model calculates attention weights between a lesion object query and a medical device object query in a multi-head attention layer and adaptively adjusts the detection reliability of the lesion object, thereby enabling the effect of flexibly responding to the intervention situation of a medical device without relying on a rule-based post-processing method.

[0027] According to one embodiment of the present invention, the training dataset adopts a structure in which a first training data group labeled with lesions and medical devices and a second training data group not labeled with lesions and medical devices are combined at a predetermined ratio, thereby enhancing the ability of a neural network model to identify atypical areas such as normal mucosa, light reflection, mucus, or bubbles as no-object states and effectively reducing the frequency of false positives in a clinical environment.

[0028] According to one embodiment of the present invention, the neural network model is based on the inherent data efficiency of a transformer structure that integrally calculates the mutual associations between all locations within an image through attention operations. Unlike conventional convolutional neural network-based models that required the acquisition of large-scale training data, this model can achieve the effect of stably securing clinically meaningful detection sensitivity by inferring the semantic identity of an object from the global context of the entire image using only a relatively small amount of training data.

[0029] According to one embodiment of the present invention, the tracking update unit adopts a multi-stage association tracking structure that sequentially performs first-stage matching and second-stage matching for high-reliability objects and low-reliability objects classified according to reliability scores, thereby stably maintaining the track ID of the same lesion object even in environments of temporary interference by medical devices or partial concealment by mucosal folds, and thus can exert the effect of suppressing the flickering phenomenon of the visual identifier.

[0030] According to one embodiment of the present invention, the tracking update unit integrates a camera motion compensation function, an external shape embedding-based matching function, and a width-height separation state estimation function, thereby producing a tracking result that is robust even to displacement across the entire screen and changes in the shape of atypical polyps caused by the operator's endoscope manipulation, and thus can minimize the track ID switching phenomenon.

[0031] According to one embodiment of the present invention, the overlay display unit applies a speed-sensitive adaptive filter in which the smoothing coefficient is dynamically varied based on the movement speed of the lesion object, thereby suppressing jittering for a stationary lesion object and reducing tracking lag for a moving lesion object, so as to simultaneously achieve adaptive visual stabilization within a single algorithm structure. Brief explanation of the drawing

[0032] Figure 1 schematically illustrates the generational development process of object recognition technology and details regarding conventional technology. FIG. 2 schematically illustrates the overall configuration of a computing system that performs an endoscopic image-based lesion detection method according to one embodiment of the present invention. FIG. 3 schematically illustrates the steps of an endoscopic image-based lesion detection method according to one embodiment of the present invention. FIG. 4 schematically illustrates a full-transformer architecture of a neural network model having an attention mechanism-based encoder-decoder structure according to one embodiment of the present invention and a hybrid architecture according to another embodiment of the present invention. FIG. 5 schematically illustrates the process in which the detection reliability of a lesion object is adaptively adjusted based on the attention weights between a lesion object query and a medical device object query in a multi-head attention layer of a neural network model according to one embodiment of the present invention. FIG. 6 schematically illustrates the detailed execution flow of the first matching step and the second matching step performed in the tracking update step according to one embodiment of the present invention. FIG. 7 schematically illustrates the detailed execution steps of the overlay display step according to one embodiment of the present invention. FIG. 8 schematically illustrates the comparison of the coordinate behavior of a lesion object bounding box before and after the application of a velocity-sensitive adaptive filter according to one embodiment of the present invention. FIG. 9 schematically illustrates the output results of detection reliability by class of lesion objects according to one embodiment of the present invention. FIG. 10 schematically illustrates an overlay screen in which detection reliability is adaptively adjusted and output according to whether a medical device is identified within the current frame according to one embodiment of the present invention. FIG. 11 schematically illustrates an example of an endoscopic image including a lesion object that is a target detected by a computing system according to one embodiment of the present invention. FIG. 12 illustrates, in an exemplary manner, the internal configuration of a computing device according to one embodiment of the present invention. Specific details for implementing the invention

[0033] Hereinafter, various embodiments and / or aspects are disclosed with reference to the drawings. For illustrative purposes, numerous specific details are disclosed in the following description to aid in a general understanding of one or more aspects. However, it will also be recognized by those skilled in the art that these aspects may be practiced without such specific details. The following description and the accompanying drawings describe specific exemplary aspects of one or more aspects in detail. However, these aspects are exemplary, and some of the various methods in the principles of the various aspects may be used, and the description is intended to include all such aspects and their equivalents.

[0034] In addition, various aspects and features will be presented by a system that may include multiple devices, components and / or modules, etc. It should also be understood and recognized that various systems may include additional devices, components and / or modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in relation to the drawings.

[0035] As used herein, terms such as "examples," "examples," "aspects," "examples," etc., may not be interpreted as implying that any aspect or design described is better or more advantageous than other aspects or designs.

[0036] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that the relevant feature and / or component is present, but not to exclude the presence or addition of one or more other features, components and / or groups thereof.

[0037] Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0038] Furthermore, in the embodiments of the present invention, all terms used herein, including technical or scientific terms, unless otherwise defined, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the embodiments of the present invention.

[0039] Figure 1 schematically illustrates the generational development process of object recognition technology and details regarding conventional technology.

[0040] Gastrointestinal endoscopy is a key diagnostic procedure that enables the rapid identification of lesions such as polyps, adenomas, and early-stage cancers occurring on the gastrointestinal mucosa within the operator's field of view, allowing for early resection or further examination. Particularly in the case of colonoscopy, as clinical reports accumulate indicating that a single microscopic polyp overlooked during the examination process can later develop into advanced cancer, AI-based Computer-Aided Detection (CADe) technology to assist the operator's visual interpretation has been continuously developing.

[0041] As shown in Fig. 1, such object recognition technology has evolved through three major generations in accordance with the development trends in the fields of artificial intelligence and computer vision, and the field of medical imaging diagnosis is also following this trend.

[0042] FIG. 1(a) illustrates a rule-based CADe technology corresponding to the first generation. The rule-based image processing method illustrated in FIG. 1(a) is a method for identifying objects by implementing features predefined by humans in the form of code. Specifically, it may be structured to derive detection results according to fixed matching standards from structured inputs by utilizing classical computer vision algorithms such as boundary detection, color threshold filtering, and optical character recognition. However, since this first-generation method is capable of recognition only within the predefined rule range, it has limitations in that it is vulnerable to unstructured environmental changes and is difficult to apply directly to the internal biological environment, which is full of unstructured elements.

[0043] Figure 1(b) illustrates an exemplary second-generation CADe technology based on a Convolutional Neural Network (CNN). The second-generation CADe technology illustrated in Figure 1(b) can operate by extracting local feature maps of an image through a multilayer feature extractor in which a CNN model repeatedly performs convolution operations on an input image, placing anchor boxes of various predefined sizes and ratios across the entire image, calculating the probability that an object exists within each anchor box, and additionally performing a Non-Maximum Suppression (NMS) post-processing operation to remove objects detected redundantly, thereby deriving a final detection result. For example, models such as YOLO (You Only Look Once), SSD (Single Shot Detector), and Faster R-CNN fall into this category, and many currently commercialized endoscope AI solutions are based on this second-generation technology.

[0044] However, these CNN-based second-generation CADe technologies have structural limitations in the receptive range of convolutional filters, making it difficult to comprehensively recognize the global context of the entire image. Additionally, since detection results depend on the shape and size of predefined anchor boxes, it is difficult to flexibly reflect irregular polyps with inconsistent shapes. Furthermore, a separate NMS post-processing step must be added to remove duplicate detections, resulting in a trade-off between real-time performance and detection accuracy. In endoscopic examination environments requiring real-time processing of 30fps or more, these post-processing operations can act as a cause of processing delay.

[0045] Meanwhile, conventional technology may also exhibit additional limitations in terms of application in actual clinical settings. More specifically, during endoscopic examinations, medical instruments frequently appear on the screen, often partially obscuring lesions or being located in close proximity to them. However, conventional CNN-based models fail to adequately reflect the contextual relationship between these instruments and lesions during the training phase, which can lead to misidentifying parts of the instruments as lesions or, conversely, missing actual lesions located near the instruments.

[0046] Furthermore, the areas in clinical practice where AI-assisted diagnostic assistance is urgently required are not large polyps or advanced cancers that can be easily identified by the naked eye of a skilled operator, but rather microscopic or flat polyps, or small lesions located at a distance that are easily overlooked during rapid passage. However, publicly available standard endoscopic training datasets (e.g., Kvasir, CVC) and proprietary datasets used in conventional technologies are mostly composed of refined images captured by endoscopists—either through close-ups or from the clearest angles—to ensure diagnostic clarity. Consequently, the absolute amount of training data regarding microscopic lesions, which are clinically critical for detection, is insufficient. As a result, predefined anchor boxes are also biased and optimized for large objects; consequently, while conventional CNN-based models demonstrate relatively superior detection performance for large, easily identifiable lesions, they exhibit limitations in that their detection sensitivity falls short of clinical expectations for microscopic lesions that are critical for clinical detection.

[0047] In other words, conventional object recognition-based endoscopic lesion detection technology currently faces a complex set of limitations, including a structural inability to comprehensively understand the context of the entire image, hindered real-time performance resulting from reliance on anchor boxes and NMS post-processing, false positives in clinical settings caused by the failure to reflect the contextual relationship between medical instruments and lesions, and reduced detection capabilities for microscopic lesions stemming from bias in training data. Consequently, there is a need for a lesion detection technology capable of comprehensively resolving these issues.

[0048] Hereinafter, a lesion detection method according to one embodiment of the present invention and a computing system for performing the same will be described in detail.

[0049] FIG. 2 schematically illustrates the overall configuration of a computing system that performs an endoscopic image-based lesion detection method according to one embodiment of the present invention.

[0050] According to one embodiment of the present invention, a computing system that performs an endoscopic image-based lesion detection method, comprising one or more processors and one or more memories, may include: an image receiving unit (1100) that receives a real-time image stream from an endoscopic device; a lesion detection unit (1200) that detects a lesion object using a neural network model having an attention mechanism-based encoder-decoder structure for a current frame of the image stream received by the image receiving unit (1100), thereby deriving a detection result including the location and class of the lesion object; a tracking update unit (1300) that updates tracking information for the lesion object based on the detection result of the lesion detection unit (1200); and an overlay display unit (1400) that overlays information about the lesion object tracked in the tracking update step onto the image stream and outputs it to a display unit.

[0051] At this time, the detection result of the lesion detection unit (1200) is transmitted to the tracking update unit (1300) without performing separate post-processing, and the neural network model may be a model in which the contextual relationship between the lesion and one or more other objects other than the lesion is learned based on a training dataset in which the lesion and one or more other objects other than the lesion are labeled.

[0052] As illustrated in FIG. 2, a computing system for performing an endoscopic image-based lesion detection method according to one embodiment of the present invention may be configured to include an image receiving unit (1100) that receives a real-time image stream from an endoscopic device, a lesion detection unit (1200) that derives a detection result of a lesion object for a current frame transmitted from the image receiving unit (1100), a tracking update unit (1300) that updates tracking information based on the detection result of the lesion detection unit (1200), and an overlay display unit (1400) that overlays the tracking information derived from the tracking update unit (1300) onto the image stream and outputs it to a display.

[0053] In one embodiment of the present invention, the computing system may be implemented in a form including one or more processors and one or more memories. More specifically, the computing system may be implemented as a general-purpose computing device such as a separate workstation, personal computer, laptop, or tablet placed in an endoscopy examination room, as well as integrated into a system-on-chip or edge device embedded in the control module of the endoscopy device.

[0054] Furthermore, the computing system is not limited to a physical hardware form and can be implemented as a centralized or distributed computing structure utilizing virtualized resources on a server or cloud infrastructure located remotely from the endoscope device. For example, the computing system can be understood as a concept including a local server within a medical institution, a cloud server in an external data center, or an edge server that processes distributed data over a network. In this case, the computing system (1000) may be equipped with a dedicated accelerator (GPU, NPU, FPGA, etc.) or use a dedicated communication protocol to prevent bottlenecks in high-resolution image data and ensure real-time performance of 30fps or more.

[0055] Consequently, the configuration referred to as the computing system (1000) in this specification may include one or more of hardware, software, or technical means organically combined with the same that can perform a lesion detection algorithm and visual stabilization processing according to the present invention.

[0056] The above image receiving unit (1100) corresponds to a configuration that receives a real-time image stream captured by the endoscope device and provides input data for subsequent processing. More specifically, the image receiving unit (1100) can extract a current frame, which is a unit image to be analyzed, from the received image stream and transmit it to the lesion detection unit (1200). Meanwhile, as illustrated in FIG. 2, in one embodiment of the present invention, the image receiving unit (1100) can transmit the current frame to the lesion detection unit (1200) and simultaneously transmit the real-time image stream directly to the overlay display unit (1400). In this case, the original image stream to be displayed to the user can be transmitted to the overlay display unit (1400) separated from the inference processing path of the neural network model, and the overlay display unit (1400) can output separately calculated tracking information by synthesizing it over the original image, thereby minimizing the latency perceived by the user. At this time, a time difference in terms of frame timing may occur between the original video stream directly transmitted from the video receiving unit (1100) to the overlay display unit (1400) and the tracking information in which delays have accumulated during the computational processing of the lesion detection unit (1200) and the tracking update unit (1300). To correct this, in one embodiment of the present invention, at least one of the video receiving unit (1100) or the overlay display unit (1400) may further include a synchronization buffer to match the frame timing between the two paths. More specifically, the synchronization buffer may operate to synthesize the original video captured at the same time and the corresponding tracking information at the overlay display unit (1400) based on a timestamp or frame identifier assigned to each frame. Through this, the visual misalignment between the video moving rapidly according to the operator's endoscope operation and the visual identifier (e.g., bounding box or label) displayed thereon can be effectively minimized.

[0057] Meanwhile, in another embodiment of the present invention, the image transmission path from the image receiving unit (1100) to the overlay display unit (1400) may be implemented in a form that passes through the lesion detection unit (1200) and the tracking update unit (1300).

[0058] The above-described lesion detection unit (1200) is configured to detect a lesion object using a neural network model having an attention mechanism-based encoder-decoder structure for the current frame received from the above-described image receiving unit (1100). More specifically, the above-described lesion detection unit (1200) can derive a detection result including the location and class of the lesion object through the above-described neural network model and transmit it to the above-described tracking update unit (1300). At this time, the above-described neural network model corresponds to a model in which the contextual relationship between the lesion and other objects other than the lesion is learned based on a training dataset in which the lesion and one or more objects other than the lesion are labeled, and accordingly, the detection result for the lesion object can be derived while integrally recognizing the presence and distribution of medical devices within the image stream. Furthermore, a configuration that understands the global context of an image based on this attention mechanism can provide robustness that effectively excludes atypical noise environments, such as mucosal folds, light reflections, or mucus unique to the endoscopic environment, without mistaking them for lesions.

[0059] Meanwhile, in one embodiment of the present invention, the detection result derived from the lesion detection unit (1200) can be directly transmitted to the tracking update unit (1300) without undergoing a separate post-processing step. More specifically, redundant detection removal operations, such as non-maximum suppression, which are generally required in conventional convolutional neural network-based detection models, are not performed in the computing system of the present invention. This may be because the decoder of the neural network model itself has a structure that directly predicts lesion objects one-to-one through learnable object queries. Due to this direct prediction structure, the processing path between the lesion detection unit (1200) and the tracking update unit (1300) can be effectively shortened, and bottlenecks that may occur in a real-time processing environment can be substantially minimized.

[0060] The tracking update unit (1300) is configured to update tracking information for the lesion object based on the detection result received from the lesion detection unit (1200). More specifically, the tracking update unit (1300) can update tracking information by mutually matching existing track information maintained in the previous frame with the detection result of the current frame, thereby maintaining a Track ID for the same lesion object in a temporally continuous manner or creating a new track for a newly appearing lesion object. Meanwhile, the tracking update unit (1300) can transmit the updated tracking information to the overlay display unit (1400).

[0061] The overlay display unit (1400) is configured to generate a visual identifier (e.g., bounding box, label) indicating the location of the lesion object based on tracking information received from the tracking update unit (1300), and to generate an overlay screen by overlaying the visual identifier onto a real-time video stream transmitted from the image receiving unit (1100). The overlay screen generated by the overlay display unit (1400) can be transmitted to the display connected to the computing system and displayed to the operator in real time. In particular, the overlay display unit (1400) may be equipped with a filter module that performs coordinate correction based on the amount of change in the object's movement coordinates, thereby correcting the jittering phenomenon of the identifier to reduce visual obstruction and visual fatigue of the operator. A more detailed explanation of such a filter module will be provided through the drawings described later.

[0062] In other words, the endoscopic image-based lesion detection method according to one embodiment of the present invention is configured such that the image reception step (S100), the lesion detection step (S200), the tracking update step (S300), and the overlay display step (S400) are sequentially performed as a single processing procedure, and no separate post-processing procedure is intervened between adjacent steps, thereby minimizing data transmission delay between each step and stably ensuring real-time performance of 30fps or more.

[0063] FIG. 3 schematically illustrates the steps of an endoscopic image-based lesion detection method according to one embodiment of the present invention.

[0064] According to one embodiment of the present invention, an endoscopic image-based lesion detection method performed in a computing system comprising one or more processors and one or more memories may include: an image reception step of receiving a real-time image stream from an endoscopic device; a lesion detection step of detecting a lesion object using a neural network model having an attention mechanism-based encoder-decoder structure for a current frame of the image stream received in the image reception step, thereby deriving a detection result including the location and class of the lesion object; a tracking update step of updating tracking information for the lesion object based on the detection result of the lesion detection step; and an overlay display step of overlaying information about the lesion object tracked in the tracking update step onto the image stream and outputting it to a display unit.

[0065] At this time, no separate post-processing step is performed between the lesion detection step and the tracking update step, and the neural network model may be a model in which the contextual relationship between the lesion and one or more other objects other than the lesion is learned based on a training dataset in which the lesion and one or more other objects other than the lesion are labeled.

[0066] As illustrated in FIG. 3, an endoscopic image-based lesion detection method according to one embodiment of the present invention may be performed by including an image reception step, a lesion detection step, a tracking update step, and an overlay display step. The process of performing each step will be described in more detail below.

[0067] The above image reception step is performed by the image reception unit (1100) and corresponds to a step of receiving a real-time image stream from an endoscope device. More specifically, in the image reception step, the image stream captured and transmitted from the endoscope device can be separated into a current frame, which is a unit image in an analyzable form, and transmitted to a subsequent step. In one embodiment of the present invention, the image stream received in the image reception step may correspond to a real-time image of 30fps or higher, and this can be used as an input for real-time analysis in a subsequent lesion detection step.

[0068] The above lesion detection step is performed by the above lesion detection unit (1200) and corresponds to a step of deriving a detection result including the location and class of the lesion object by detecting a lesion object using a neural network model having an attention mechanism-based encoder-decoder structure for the current frame of the video stream received in the above video reception step. More specifically, the above neural network model corresponds to a model in which the contextual relationship between the lesion and one or more other objects other than the lesion is learned based on a training dataset in which the lesion and one or more other objects other than the lesion are labeled. That is, in the above lesion detection step (S200), the detection result for the lesion object can be derived by not simply relying on local visual features within the video to detect the lesion, but by integrally recognizing the global context of the entire video and the relationship with other objects existing together within the video. Meanwhile, in one embodiment of the present invention, the location of the lesion object may be expressed in the form of bounding box coordinates (e.g., [x, y, w, h]), and the class of the lesion object may include information indicating whether there is a lesion or the type of lesion (e.g., adenoma, hyperplastic polyp, etc.).

[0069] The tracking update step (S300) is performed by the tracking update unit (1300) and corresponds to a step of updating tracking information for the lesion object based on the detection result of the lesion detection step (S200). More specifically, in the tracking update step (S300), existing track information maintained up to the previous frame and the detection result derived from the current frame are matched with each other; if it is determined to be the same lesion object, the Track ID of the object is maintained continuously in time, and a new track can be created for new detection results that do not match. Meanwhile, in one embodiment of the present invention, the tracking information may include frame-by-frame location coordinates, Track ID, and the active / lost status of the track for each lesion object.

[0070] The overlay display step (S400) is performed by the overlay display unit (1400) and corresponds to a step of overlaying information about the lesion object tracked in the tracking update step (S300) onto the video stream and outputting it to the display unit. More specifically, in the overlay display step (S400), a visual identifier (e.g., bounding box, label, etc.) indicating the location of the lesion object is generated based on the tracking information, and the visual identifier is composited onto the original video stream received in the video reception step (S100) to form an overlay screen, after which it can be output to the display (20).

[0071] Meanwhile, a separate post-processing step may not be performed between the lesion detection step (S200) and the tracking update step (S300) according to one embodiment of the present invention. That is, duplicate detection removal operations such as NMS, which are generally required in conventional Convolutional Neural Network (CNN)-based object detection methods, are not performed in the present invention. This is because the neural network model used in the lesion detection step (S200) itself has a structure that directly predicts lesion objects one-to-one through learnable object queries. With this configuration, the detection result of the lesion detection step (S200) can be transmitted directly to the tracking update step (S300) without processing delay due to separate post-processing, and processing bottlenecks can be effectively minimized in an endoscopic examination environment requiring real-time processing of 30fps or more.

[0072] In other words, the endoscopic image-based lesion detection method according to one embodiment of the present invention detects a lesion while integrally recognizing the contextual relationship between the lesion and its surrounding objects through an attention mechanism-based neural network model, directly links the detection result to a tracking step without a separate post-processing step, and outputs the tracked result by overlaying it onto a real-time image stream; by organically performing a series of steps on a single pipeline, the reliability of lesion detection can be improved even in an endoscopic examination environment where interference from medical devices is frequent, while simultaneously minimizing processing delay.

[0073] FIG. 4 schematically illustrates a full-transformer architecture of a neural network model having an attention mechanism-based encoder-decoder structure according to one embodiment of the present invention and a hybrid architecture according to another embodiment of the present invention.

[0074] According to one embodiment of the present invention, in the lesion detection step, the neural network model comprises: an encoder that calculates global interactions for features derived from the current frame through an attention operation; and a decoder that predicts the location and class of the lesion object based on a learnable object query, wherein the decoder may not utilize an anchor box.

[0075] Meanwhile, according to one embodiment of the present invention, the neural network model may be a Full-Transformer architecture combining a Vision Transformer (ViT)-based backbone and a Transformer decoder.

[0076] In addition, according to another embodiment of the present invention, the neural network model may be a hybrid architecture combining a convolutional neural network (CNN)-based backbone and a transformer encoder-decoder structure.

[0077] FIG. 4(b) illustrates an exemplary data processing flow when a neural network model according to one embodiment of the present invention is implemented as a Full-Transformer architecture combining a Vision Transformer (hereinafter ViT)-based backbone and a Transformer decoder.

[0078] In one embodiment of the present invention, the full-transformer architecture illustrated in FIG. 4(b) can calculate global interactions by having a ViT-based backbone divide the input image into multiple image patches when a current frame is input, convert each patch into an embedding vector, and then perform an attention operation on the converted patch sequence. That is, the ViT-based backbone can integrally perform the feature extraction process and the global context calculation process within a single module. Subsequently, the global feature sequence derived through the aforementioned process can be transmitted to a transformer decoder, and the transformer decoder can output the location and class prediction results of the lesion object based on a learnable object query.

[0079] In other words, since the full-transformer architecture applies the attention mechanism from the first stage of image processing, it can overcome the structural constraints of the receptive field inherent in the convolution operations of the CNN backbone and reflect the context of the entire image in a more direct and integrated manner.

[0080] FIG. 4(a) exemplarily illustrates the data processing flow when a neural network model according to another embodiment of the present invention is implemented as a hybrid architecture combining a convolutional neural network (CNN)-based backbone and a transformer encoder-decoder structure.

[0081] In another embodiment of the present invention, the hybrid architecture illustrated in FIG. 4(a) can operate in a flow in which, when a current frame is input, features of the image are first extracted through a CNN-based backbone, the extracted features are passed to a transformer encoder to calculate global interactions for the features through attention operations, and the encoded features are passed to a transformer decoder to output the location and class prediction results of the lesion object based on a learnable object query.

[0082] More specifically, the above CNN-based backbone may correspond to a convolutional neural network structure such as ResNet, and can be responsible for generating a feature map that compresses local visual information by repeatedly performing convolution operations on the input image. Meanwhile, the above Transformer encoder can derive an encoding that reflects global context by calculating the interrelationships between all locations within the image after adding position embeddings to the feature map and performing a self-attention operation.

[0083] Both of the above architectures share a common structure in which a transformer decoder directly predicts the location and class of a lesion object based on a learnable object query. More specifically, the learnable object query corresponds to a type of learnable vector that is trained during the learning process to contain prior information regarding which region within the image is likely to contain what type of object. The transformer decoder can output the bounding box coordinates (e.g., [x, y, w, h]) and class (e.g., [lesion]) of the lesion object corresponding to each query as a prediction result by combining the object query and features received from the encoder through a cross-attention operation. Meanwhile, the decoder does not utilize a predefined anchor box, and accordingly, it is possible to operate using a direct prediction method that does not require separate post-processing operations such as non-maximum suppression (NMS).

[0084] To enable a clearer understanding of the operating principle of the neural network model according to one embodiment of the present invention, the processing process in a situation where a medical instrument appears within an image and approaches a lesion is described by example. For instance, one can assume a situation where a polyp exists within the current frame, and a medical instrument, such as biopsy forceps, enters the screen and gradually approaches the polyp due to manipulation by the operator. In the case of conventional convolutional neural network-based detection methods, objects are identified based only on local information contained within the limited receptive area of ​​the convolutional filter; consequently, as the medical instrument enters the screen, the consistency of features extracted from that area may deteriorate, or parts of the instrument's external shape may be interpreted as textures similar to the lesion, leading to false detections. Furthermore, conventional detection methods fail to recognize the relationship between the medical instrument and the lesion in an integrated manner, making it difficult to adaptively adjust detection results according to the progress of the procedure. Additionally, they have limitations in that appropriate control of the visual identifier displayed to the operator is virtually impossible without a separate rule-based post-processing step.

[0085] On the other hand, the encoder of the neural network model according to one embodiment of the present invention can integrally encode features corresponding to the occluded lesion region together with contextual information throughout the image, such as mucosal tone, vascular patterns, boundaries with surrounding tissues observed at other locations within the image, and the location and shape of a medical device superimposed thereon, during the process of calculating the interrelationships between all locations within the image through attention operations. That is, the encoder can encode features in a form that reflects the spatial and contextual relationship between the two objects, while identifying the appearance of the medical device not as mere noise but as a separate, meaningful object distinct from the lesion.

[0086] Subsequently, the decoder can directly predict the location and class of the lesion object and the medical device object, respectively, by combining the learnable object query and the encoding features received from the encoder through cross-attention operations. More specifically, the object query, which learns the contextual relationship between the two objects from images in which various medical devices and various types of lesions appear together during the learning process, can determine the lesion and the medical device as distinct classes during the inference stage, while simultaneously calculating multi-head attention weights between the object queries corresponding to the two objects. The attention weights calculated at this time tend to gradually increase as the medical device approaches the lesion, as specifically explained in Figure 5 below. This change in attention weights can be seamlessly linked to the adaptive adjustment of detection reliability for the lesion object and the control of the output of the visual identifier displayed to the operator within the same neural network model pipeline.

[0087] That is, according to one embodiment of the present invention, through a full-transformer architecture combining a ViT-based backbone and a transformer decoder, a neural network model that integrally learns the global context of the entire image and the contextual relationships between objects directly predicts the location and class of lesion objects and medical device objects based on learnable object queries, thereby effectively minimizing the problems of missing detection and false detection in occluded environments that occurred in conventional convolutional neural network-based methods, and achieving the effect of deriving stable lesion detection results without a separate post-processing step.

[0088] FIG. 5 schematically illustrates the process in which the detection reliability of a lesion object is adaptively adjusted based on the attention weights between a lesion object query and a medical device object query in a multi-head attention layer of a neural network model according to one embodiment of the present invention.

[0089] According to one embodiment of the present invention, other objects other than the lesion include medical devices, and the training dataset includes training data in which the medical devices are labeled, so that the neural network model is trained to distinguish the lesion and the medical devices as distinct classes, and when the medical device is identified within the current frame in the lesion detection step, the detection reliability of the lesion object can be adaptively adjusted based on the attention weight between the object query corresponding to the lesion and the object query corresponding to the medical device in the multi-head attention layer of the neural network model.

[0090] Meanwhile, in one embodiment of the present invention, the training dataset comprises: a first training dataset group in which at least one of a lesion or a medical device is labeled; and a second training dataset group in which no label is assigned to an object including a lesion and a medical device; and the ratio of the first training dataset group to the second training dataset group within the training dataset may be in the range of 7:3 to 9.5:0.5.

[0091] More preferably, the ratio of the first training data group to the second training data group may be 8:2.

[0092] As illustrated in FIG. 5, a neural network model according to one embodiment of the present invention may be configured with an operation flow in which attention weights between object queries corresponding to lesions are calculated in a multi-head attention layer included in the decoder of the neural network model, and the detection reliability of lesion objects is adaptively adjusted based on the calculated attention weights. Below, the neural network model of the present invention is trained based on certain training data, and the manner in which the trained neural network model operates during the inference stage is explained in more detail.

[0093] In one embodiment of the present invention, the training dataset used for training the neural network model may be configured to include training data in which various forms of labeling are assigned to objects included in an image. More specifically, the training dataset may include, for example, training data in which only lesions are included in an image and labeling is assigned to the lesions; training data in which lesions and medical devices are included together in an image and labeling is assigned to both objects; training data in which microscopic lesions (e.g., microscopic polyps, flat polyps, etc.) occupying an area less than or equal to a preset reference ratio relative to the screen area in an image and labeling is assigned to them; or training data corresponding to a normal mucosal environment in which neither lesions nor medical devices are included in the image. However, the types of labeling are not limited to the above examples, and depending on the characteristics of the clinical environment that the neural network model of the present invention intends to recognize, additional labeling for various incidental objects that may appear in mucosal images, such as mucus, foam, residual stool, and light reflection areas, may be included.

[0094] In one embodiment of the present invention, the neural network model may possess a plurality of learnable object queries within the decoder during the learning phase, and as learning progresses, each object query may be updated to contain prior information regarding visual patterns in which objects of a specific class appear in an image and contextual relationships between the objects.

[0095] For example, the object query described above can evolve through the learning process into a vector that performs semantic query functions such as "where is the area with a high probability of a lesion?", "where is the area where a medical device appears?", and "what is the spatial relationship between the lesion and the medical device in the current frame?". That is, the object query functions as a vector that concisely represents visual and contextual patterns repeatedly observed in the training data, and can be used to predict the location and class of the object corresponding to each query by performing cross-attention with image features received from the encoder during the inference stage. As described in relation to Fig. 4, through this object query-based direct prediction mechanism, the present invention can flexibly detect micro-lesions of irregular shape without relying on anchor boxes of a fixed size in advance.

[0096] Through the learning process described above, the neural network model according to one embodiment of the present invention is trained to distinguish between lesions and medical devices as distinct classes, and at the same time, is trained to recognize the spatial and contextual relationships between the lesions and medical devices through the results of attention operations between the two object queries. Accordingly, in the inference stage, the neural network model can determine not only whether a lesion is present, but also whether a medical device is in close proximity to a lesion and a procedure is in progress within the current frame, or whether the medical device is located in a position unrelated to the lesion and merely appears on the screen.

[0097] As illustrated in FIG. 5, when a lesion and a medical device are identified together within the current frame during the inference stage, the multi-head attention layer of the neural network model may calculate an attention weight between the lesion object query (A) and the medical device object query (B). More specifically, if a medical device is close to the lesion and a procedure or interference situation occurs, the attention weight between the two object queries may be calculated as a relatively high value, and in this case, the neural network model may adaptively down-adjust (suppress) the detection confidence of the lesion object in the corresponding frame. On the other hand, if the medical device is located far from the lesion and the contextual association between the two objects is low, the attention weight is calculated as a relatively low value, and in this case, the neural network model may maintain the detection confidence of the lesion object without separate confidence adjustment.

[0098] With this configuration, the neural network model according to one embodiment of the present invention can independently adjust detection reliability within a single integrated model based on contextual information between objects inherent in the learning phase, unlike conventional rule-based post-processing methods such as IoU-based forced deletion or stopping display after frame-by-frame counting. That is, when a medical device is close to a lesion, detection reliability is suppressed, thereby minimizing the phenomenon where the operator is repeatedly exposed to unnecessary visual warnings; conversely, when the medical device appears in an area unrelated to the lesion, lesion detection is maintained normally, allowing for flexible response to various situations in the clinical environment.

[0099] Meanwhile, the detection reliability that is adaptively down-adjusted as described above can be consistently linked to the visual identifier output processing of the overlay display step described later. More specifically, in one embodiment of the present invention, the overlay display step may operate to suppress the output of a visual identifier (e.g., a bounding box or a label) for a lesion object when the detection reliability of the lesion object, which is adaptively down-adjusted based on the attention weight of the multi-head attention layer, is less than a preset display threshold. With this configuration, the visual identifier for the lesion object displayed within the operator's field of vision naturally disappears when the medical device is performing a procedure based on the lesion, thereby effectively preventing the phenomenon in which the visual identifier acts as a factor obstructing the operator's field of vision during a procedure requiring precise manipulation.

[0100] Meanwhile, in one embodiment of the present invention, the training dataset may include: a first training data group in which at least one of a lesion or a medical device is labeled; and a second training data group in which no label is assigned to an object including a lesion and a medical device.

[0101] That is, the first training data group corresponds to a set of training data in which lesions, medical devices, or both are included in the image and labels are assigned to the corresponding objects, and the second training data group corresponds to a set of training data in which normal mucosa, light reflections, mucus, bubbles, or residual stool are distributed and no separate object labels are assigned, as a background image in which lesions and medical devices are not included.

[0102] In particular, a training dataset according to one embodiment of the present invention may include a second training data group composed of pure background images in which no lesions or medical devices exist, within a range of 5 to 30% of the total training dataset's data amount. This configuration induces the neural network model to naturally learn and discriminate the visual context of an 'object-free' state, thereby effectively reducing the frequency of misidentifying normal mucosa or light-reflecting areas as lesions during the inference stage. That is, the lesion detection method and the computing system performing the same according to the present invention can effectively suppress unnecessary false detections even in complex mucosal environments and improve clinical reliability as a diagnostic assistance system.

[0103] Meanwhile, in one embodiment of the present invention, some of the training data included in the first training data group may be configured to include training data labeled for micro-lesions that are easily overlooked by operators in a clinical environment. More specifically, the micro-lesions may be defined as lesions occupying an area of ​​less than 10% of the total screen area within the image, and in a more preferred embodiment of the present invention, the micro-lesions may be defined as lesions occupying an area within the range of 0.1% to 5% of the total screen area within the image. Even more preferably, the micro-lesions may be defined as lesions occupying an area within the range of 0.5% to 3% of the total screen area within the image, and micro-lesions corresponding to this area range are regions where conventional convolutional neural network-based models have difficulty detecting, and may correspond to key regions where the attention mechanism-based neural network model of the present invention demonstrates clinically meaningful detection performance improvement based on attention operations on the global context of the entire image.

[0104] That is, in one embodiment of the present invention, the first training data group may further include training data labeled for micro-lesions occupying an area of ​​3% or less of the total screen area in the image.

[0105] With this configuration, in one embodiment of the present invention, the neural network model is trained to determine a normal mucosa, a light-reflecting area, a mucus or foam area as a no-object state based on the background learning data, thereby reducing the frequency of false positives in which normal tissue is detected as a lesion.

[0106] For reference, conventional object detection models based on convolutional neural networks required the acquisition of balanced, large-scale training data for all classes to ensure sufficient detection performance due to their structural characteristics. However, there was a limitation in that for classes such as microlesions, where the absolute quantity is insufficient in public datasets and clinical settings, separate data augmentation (e.g., GAN-based virtual image generation) or the construction of additional training data in the tens of thousands was necessary.

[0107] However, the neural network model according to one embodiment of the present invention can infer the semantic identity of an object from the global context of the entire image due to the characteristics of the transformer structure, which integrally calculates the mutual association between all locations within the image through attention operations. Therefore, the detection rate of micro-lesions can be improved even in environments where the absolute amount of training data is relatively small, and it can have practical data efficiency that can effectively learn the contextual relationship between micro-lesions and medical devices.

[0108] In addition, due to the characteristics of the transformer structure, which integrally calculates the interrelationships between all locations within an image through attention operations, the neural network model can infer the identity of an object from the global context of the entire image and the semantic relationships between objects within the image; thus, compared to conventional convolutional neural network-based models that rely on predefined anchor boxes, it can demonstrate robust detection performance even for micro-lesions of various sizes and shapes.

[0109] To verify the micro-lesion detection performance of the neural network model according to one embodiment of the present invention, the applicant trained the neural network model of the present invention based on a training dataset configured to include training data labeled for micro-polyps occupying less than 3% of the total screen area within the first training data group. Subsequently, a comparative experiment was performed with a neural network model corresponding to a full-transformer architecture, a comparative neural network model #1 corresponding to a hybrid architecture combining a convolutional neural network-based backbone and a transformer encoder-decoder, and a comparative neural network model #2 based on a conventional convolutional neural network-based model (e.g., a YOLO-based model). The results of the comparative experiment are described in more detail below according to the ratio of lesion area.

[0110] In one embodiment of the present invention, in a comparative experiment targeting the detection of micropolyps occupying less than 3% of the total screen area within an image, the neural network model according to one embodiment of the present invention achieved a detection performance level of approximately 0.75 based on the F1-score. Meanwhile, Comparative Neural Network Model #1, which corresponds to a hybrid architecture adopting a convolutional neural network-based backbone, achieved an F1-score of 0.65 under the same conditions, showing a significant performance degradation compared to the F1-score of approximately 0.79 obtained under general conditions without area limitations. Furthermore, the conventional convolutional neural network-based Comparative Neural Network Model #2 achieved an F1-score of approximately 0.53 to 0.70 under the same conditions, consistently exhibiting a downward trend in micro-detection sensitivity attributed to the convolutional structure of the backbone. That is, it was confirmed that the full-transformer architecture-based neural network model according to one embodiment of the present invention stably maintains a clinically meaningful level of detection performance even for microlesions, whereas the detection performance of comparative neural network model #1 or comparative neural network model #2, which rely on a convolutional neural network structure in the backbone, consistently deteriorates as the area of ​​the lesion in the image decreases.

[0111] Furthermore, in a comparative experiment targeting the detection of microscopic polyps occupying less than 1% of the total image area, the neural network model based on a full-transformer architecture according to one embodiment of the present invention achieved a detection performance of approximately 0.62 based on an F1-score, confirming that it maintains robust detection performance at a clinically applicable level even in environments where the area of ​​the lesion within the image is extremely small, less than 1%. On the other hand,

[0112] Comparison Neural Network Model #1, which corresponds to a hybrid architecture adopting a convolutional neural network-based backbone, yielded results with an F1-score of approximately 0.35 under the same conditions. This resulted in detection performance that was equivalent to or even lower than the F1-score of approximately 0.24 to 0.52 obtained by the conventional convolutional neural network-based comparison Neural Network Model #2. As the area of ​​the lesion within the image becomes extremely small, a tendency is observed where the degradation in detection performance of the hybrid architecture model relying on the convolutional neural network structure at the backbone stage and the conventional convolutional neural network-based model expands significantly. This can be understood as being due to the structural characteristics of the backbone, which relies on the limited receptive field of the convolutional filter. On the other hand, the full-transformer architecture, which can infer the semantic identity of an object from the global context of the entire image based on an attention mechanism from the backbone stage, can stably maintain its detection performance even in environments where the area of ​​the lesion within the image is extremely small.

[0113] In other words, the results of the comparative experiment described above quantitatively support the fact that the neural network model according to one embodiment of the present invention provides clinically significant improvement in detection performance in the most clinically important micro-lesion regions, that is, in regions where conventional technology has virtually faced detection limits. This confirms that the clinically friendly effect of "reduction in the operator's lesion oversight rate" pursued by the present invention can be substantially realized.

[0114] In other words, according to one embodiment of the present invention, the reliability of lesion detection within a single integrated model is adaptively adjusted based on attention weights between lesion object queries and medical device object queries calculated in the multi-head attention layer, thereby enabling flexible response to intervention situations involving surgical instruments without the need for separate rule-based post-processing. Furthermore, based on the inherent data efficiency of the transformer structure, robust learning is possible even with a relatively small amount of micro-lesion training data. Consequently, the burden of constructing large datasets required in conventional convolutional neural network-based models and the reliance on GAN-based virtual data augmentation can be minimized, and the ability to effectively detect lesions of various sizes and shapes without the need for separate tuning of predefined anchor boxes can be achieved.

[0115] FIG. 6 schematically illustrates the detailed execution flow of the first matching step and the second matching step performed in the tracking update step according to one embodiment of the present invention.

[0116] According to one embodiment of the present invention, in the tracking update step, tracking information is updated by associating a lesion object with an existing track or creating a new track, wherein when associating a lesion object with an existing track, a first matching step is performed to first match a first detection object having a preset high reliability standard or higher among the detection results with the existing track; and for an existing track that is not matched in the first matching step, a second matching step is performed to second match a second detection object having a preset low reliability standard or higher among the detection results with a second detection object having a high reliability standard or lower than the high reliability standard.

[0117] As illustrated in FIG. 6, a tracking update step (S300) according to one embodiment of the present invention may be initiated by receiving existing track information maintained up to the previous frame and the lesion detection result of the current frame. It may be configured as a flow in which a first matching and a second matching are sequentially performed by branching according to the reliability score of each detection object included in the detection result.

[0118] In one embodiment of the present invention, a 'Detection Object' refers to an individual detection result derived from the current frame by a neural network model in the lesion detection step (S200). More specifically, each detection object may be configured to include attributes such as bounding box coordinates (e.g., [x, y, w, h]), class information (e.g., lesion, medical device, etc.), a confidence score, and an appearance embedding vector. That is, the detection object corresponds to a detection result derived only once within a single frame and does not possess an identifier (ID) on the time axis. In contrast, a 'Lesion Object' refers to a tracking target identified and tracked as the same entity across the time axis. More specifically, it may correspond to a unit in which detection objects repeatedly detected across multiple frames are associated as the same entity and integrated into a single track. That is, the tracking update step (S300) essentially corresponds to the operation of associating one-time 'detection objects' newly derived in each frame with a 'lesion object (i.e., track)' having temporal continuity, or creating a new track for a detection object that is not associated with any track.

[0119] In one embodiment of the present invention, 'Track' corresponds to a data unit that stores the temporal tracking history of the same lesion object, and tracking information can be managed in memory in the form of a set of said tracks. More specifically, said tracking information can be stored, for example, in a table-shaped data structure, where each row corresponds to one lesion object (track), and each column may include attributes such as the unique track ID of the corresponding track, location coordinates in the most recent frame, shape embedding vector, the active status of the track (e.g., Active, LOST, Removed, etc.), the number of cumulative matching frames, and the time of the last matching. However, the storage format of said tracking information is not limited to the above example, and various data structures capable of expressing equivalent information (e.g., object list, key-value mapping, etc.) may be applied.

[0120] In one embodiment of the present invention, 'existing track' refers to a track that is already registered and maintained within the tracking information through an accumulation process up to immediately before the current frame, and 'new track' refers to a track that is newly created as it is determined to have newly appeared in the current frame. Meanwhile, among the active states of the track, 'LOST state' refers to a state in which the track has not been matched with any detected object for a certain period but is not immediately discarded and is retained in memory for the possibility of future re-identification.

[0121] Based on the definitions of terms above, the detailed execution flow of the present step illustrated in Fig. 6 will be explained in more detail.

[0122] In one embodiment of the present invention, the tracking update step (S300) may first determine whether the reliability score of each detection object derived from the current frame is greater than or equal to a preset high reliability standard. More specifically, detection objects whose reliability score is greater than or equal to the preset high reliability standard may be classified into a first detection object (high reliability object) group and used as input for a subsequent first matching step. Meanwhile, for detection objects whose reliability score is less than the preset high reliability standard, it may be additionally determined whether their reliability score is greater than or equal to a preset low reliability standard. Detection objects whose reliability score is greater than or equal to the preset low reliability standard but less than the preset high reliability standard may be classified into a second detection object group and used as input for a second matching step described later, and detection objects whose reliability score falls below even the preset low reliability standard may be considered as background noise and excluded from subsequent processing.

[0123] In the first matching step, a first matching may be performed between the first detection object and existing track information maintained from the previous frame. More specifically, the first matching may be performed based on the similarity (e.g., IoU, appearance embedding similarity, etc.) between the location and external features of each first detection object and the last location and external features possessed by the existing track. As illustrated in FIG. 6, depending on the result of the first matching, the first detection object that succeeded in matching may be used as an input to update the tracking information of the matched existing track. Meanwhile, for the first detection object that was not matched with any existing track in the first matching, it is determined that a new lesion object has appeared on the screen, and a new track may be created.

[0124] Subsequently, in the second matching step, a second matching with the previously classified second detection object may be performed for the existing track that remained unmatched in the first matching step (hereinafter referred to as the 'unmatched existing track'). That is, even if a second detection object was excluded from the candidates for the first matching due to a relatively low reliability score, if sufficient positional and external consistency with the unmatched existing track is confirmed, an additional opportunity to associate it with the corresponding existing track may be granted. As illustrated in FIG. 6, depending on the result of the second matching, the second detection object that succeeded in matching may be used to update the tracking information of the matched existing track. Meanwhile, the second detection object that was not matched even in the second matching may be excluded from subsequent processing, and the unmatched existing track that ultimately was not matched with any detection object may be switched to a LOST state.

[0125] In one embodiment of the present invention, the track that has been switched to the LOST state is not immediately discarded but is retained in memory for a preset period, and can be preserved as a candidate group that can be restored with the same track ID if the same lesion object is detected again in a subsequent frame. As shown in FIG. 6, after both the first matching and the second matching are completed, the tracking information for the first detected object and the second detected object that succeeded in matching is finally updated, thereby terminating this step.

[0126] Meanwhile, in one embodiment of the present invention, the first and second matching may be performed using position prediction and cost-based allocation techniques commonly used in the field of multi-object tracking. More specifically, a motion model based on a Kalman filter is applied to each existing track, and a predicted bounding box in the current frame may be calculated from the position and velocity information of the track accumulated up to the previous frame. Subsequently, the consistency between the predicted bounding box and the position of each detected object derived in the current frame may be quantified through positional similarity indicators such as Intersection over Union (IoU), and in the case where shape embedding-based matching is applied together according to one embodiment of the present invention, a cost matrix may be constructed by weighted combination of cosine similarity between shape embeddings. Finally, by applying a bipartite matching algorithm, such as the Hungarian Algorithm, to the cost matrix, an optimal matching combination between each existing track and each detected object can be derived. That is, the first and second matching differ only in the groups of detected objects (high-reliability objects or low-reliability objects) used as input, but the principle of performing the matching can commonly utilize the conventional location prediction and cost-based allocation methods described above.

[0127] Through this multi-stage matching structure, the track disconnection phenomenon that frequently occurs in conventional single threshold-based object tracking methods can be effectively minimized. More specifically, even if the same lesion object is temporarily obscured by a medical device or partially concealed behind mucosal folds, causing a temporary drop in the reliability score, the track ID of the object can be maintained through the second matching stage, thereby effectively suppressing the flickering of the visual identifier displayed to the operator.

[0128] Meanwhile, although not illustrated in FIG. 6, the tracking update step (S300) according to one embodiment of the present invention may further include a configuration to reflect the specificity of the endoscopic examination environment.

[0129] Specifically, the tracking update step (S300) may include a camera motion compensation function that corrects the displacement of the entire screen caused by the movement of the entire image together with the operation of the endoscope body. Specifically, the camera motion compensation function is a function that estimates the global motion between consecutive frames based on image processing and calculates coordinates corrected by the corresponding motion component in the existing track position prediction step. Through this, the accuracy of track position prediction for a lesion object stationary on the actual mucosa can be improved even in situations where the entire screen shakes significantly due to the operator rapidly manipulating the endoscope.

[0130] In addition, the tracking update step (S300) according to one embodiment of the present invention may include a function that utilizes a weighted combination of positional similarity (e.g., IoU) and external similarity (e.g., cosine similarity between external embeddings) when extracting external embeddings for each detected object and existing track, and calculating the matching costs of the first and second matching. Specifically, even in situations where it is difficult to determine whether an object is the same based on simple location-based matching alone due to many mucosal folds or frequent intervention by surgical instruments, the accuracy of the matching can be improved by considering the similarity between external embeddings together. This can be applied equally to the re-identification operation in which a track preserved in the previously described LOST state is restored to the same track ID as a detected object that reappears later, and the track ID can be restored by determining it to be the same object only when the similarity between external embeddings is greater than or equal to a preset threshold.

[0131] Furthermore, in the tracking update step (S300) according to one embodiment of the present invention, the state vector of the Kalman filter used for predicting the position of the existing track can be configured to include width (w) and height (h) as independent state variables, along with the center coordinates and area of ​​the bounding box. That is, unlike the aspect ratio-based state vector used in conventional multi-object tracking algorithms, by estimating the width and height separately, it is possible to track changes in the external shape of polyps that are irregular and change significantly depending on the viewpoint more precisely.

[0132] That is, according to one embodiment of the present invention, detection objects derived from the current frame are classified stepwise into high-reliability objects and low-reliability objects according to their reliability scores, and by sequentially performing first-stage matching and second-stage matching using Kalman filter-based location prediction and Hungarian algorithm-based cost allocation, the effect of stably maintaining the track ID for the same lesion object can be achieved even in environments of temporary interference by surgical instruments or partial concealment caused by mucosal folds. In addition, the tracking update step (S300) according to one embodiment of the present invention adopts a multi-stage association tracking structure in which existing tracks that were not matched in the first-stage matching are not immediately discarded but are given an additional opportunity for second-stage matching, and existing tracks that were not matched are preserved in a LOST state for a certain period to keep open the possibility of future re-identification. This effectively minimizes the track break phenomenon that frequently occurred in sections where the reliability score temporarily dropped in conventional single threshold-based object tracking methods.

[0133] Furthermore, the tracking update step (S300) according to one embodiment of the present invention can derive robust tracking results that sufficiently reflect the specific characteristics of an endoscopic examination environment filled with frequent intervention of surgical instruments and atypical noise of mucosal folds by integrating a camera motion correction function that corrects a situation where the entire screen moves together according to the operation of the endoscope body, an external shape embedding-based matching function that improves the accuracy of matching by weightedly combining positional similarity and external similarity, and a width-height separation state estimation function that can precisely track changes in the shape of atypical polyps. Through this, the flickering and ID switching phenomena of the visual identifier finally displayed to the operator can be effectively suppressed, thereby reducing the visual fatigue of the operator, and as the continuity of the track is stably maintained even for microscopic lesions with temporarily reduced reliability, the operator's ability to recognize 'small lesions that are clinically most important' can be substantially improved.

[0134] FIG. 7 schematically illustrates the detailed execution steps of the overlay display step according to one embodiment of the present invention, and FIG. 8 schematically illustrates the comparison of the coordinate behavior of the lesion object bounding box before and after the application of a speed-sensitive adaptive filter according to one embodiment of the present invention.

[0135] According to one embodiment of the present invention, the overlay display step (S400) comprises the original coordinates (X) of the current frame for the lesion object tracked in the tracking update step (S300). t ) and the final corrected coordinates (XC) corrected in the previous frame t-1 Using ), the smoothing coefficient (α) according to the movement speed of the lesion object t By applying a velocity-sensitive adaptive filter in which ) is dynamically varied, the final corrected coordinates (XC) of the current frame tDetailed steps of a coordinate correction step (S430) in which ) is calculated; and a final display step (S440) in which a visual identifier including a bounding box indicating the location of the lesion object is generated based on the final correction coordinates calculated in the coordinate correction step (S430) and overlaid on the image stream.

[0136] At this time, the above velocity-sensitive adaptive filter uses the following [Equation 1] to obtain the above final correction coordinates (XC t It can be a filter that produces ).

[0137] [Equation 1]

[0138]

[0139] In addition, according to one embodiment of the present invention, the overlay display step comprises the original coordinates (X t ) and the final corrected coordinates (XC) of the previous frame mentioned above t-1 Dividing the difference of ) by the inter-frame time interval (Δt) gives the raw velocity (v) of the above-mentioned lesion object. t A raw speed calculation step (S410) for calculating ); the raw speed (v t The pre-set velocity smoothing coefficient (α) in ) v By applying ) and the operation of [Equation 2] below, the reference movement speed (V t Further detailed steps of the speed smoothing step (S420) that calculates ) can be performed.

[0140] At this time, the speed-sensitive adaptive filter has a smoothing coefficient (α) that is dynamically calculated based on the reference moving speed. t ) can be applied.

[0141] [Equation 2]

[0142]

[0143] According to one embodiment of the present invention, the smoothing coefficient is calculated based on a cutoff frequency determined in proportion to the magnitude of the moving speed, wherein if the magnitude of the moving speed is smaller than a preset speed standard, the smoothing coefficient (α t Since ) is calculated as a relatively small value, the reflection ratio of the final correction coordinates of the previous frame becomes relatively high, thereby suppressing jittering on the lesion object, and when the magnitude of the movement speed is greater than the speed standard, the smoothing coefficient (α t Since ) is calculated as a relatively large value, the reflection ratio of the original coordinates of the current frame is relatively high, thereby reducing the tracking lag for the lesion object.

[0144] As illustrated in FIG. 7, an overlay display step (S400) according to one embodiment of the present invention may be performed by including a raw velocity calculation step (S410), a velocity smoothing step (S420), a coordinate correction step (S430), and a final display step (S440). Below, the process of performing each detailed step and the operating principle of the velocity-sensitive adaptive filter utilized therein will be explained in detail.

[0145] In order to more clearly understand the detailed flow of the overlay display step (S400) according to one embodiment of the present invention, the main coordinate variables used in this step are defined first. In one embodiment of the present invention, 'original coordinates (X t )’ refers to the coordinate values ​​of the lesion object derived through the tracking update step (S300) in the current frame (e.g., the center coordinates, width, and height of the bounding box), and corresponds to the raw coordinate values ​​prior to the application of the smoothing process described later. ‘The final corrected coordinates (XC of the previous frame) t-1)’ refers to the coordinate values ​​used for displaying visual identifiers, for which smoothing correction has already been completed through the processing of the stage seen in the previous frame. ‘The final corrected coordinates of the current frame (XC t )' in this step is the original coordinates (Xt) and the final corrected coordinates (XC) of the previous frame. t-1 It is a coordinate value newly calculated using ) as input, and corresponds to a coordinate value used as the display position of a visual identifier in the current frame.

[0146] The above raw velocity calculation step (S410) corresponds to a step of calculating the inter-frame movement speed of the lesion object. More specifically, in the above raw velocity calculation step (S410), the original coordinates (X of the current frame) t ) and the final corrected coordinates of the previous frame (XC t-1 Dividing the difference of ) by the time interval (Δt) between the frames, the raw velocity (v) of the lesion object t ) can be calculated. That is, the above raw velocity (v t ) is a value that quantitatively expresses how much displacement the lesion object has caused in the current frame compared to the previous frame, and can be used as a basic input value for the dynamic calculation of the smoothing coefficient to be described later.

[0147] The above speed smoothing step (S420) is the raw speed (v) calculated in the above raw speed calculation step (S410). t This corresponds to the step of performing additional smoothing processing on the raw velocity (v). More specifically, the raw velocity (v t Since ) is a value calculated from coordinate displacement over short time intervals between single frames, it may temporarily have large values ​​due to minute vibrations of the mucosal surface or minute variability in neural network inference. This raw velocity (v t Temporary fluctuations of ) remain as is in the smoothing coefficient (α tIf reflected in the calculation of ), a problem may arise where the smoothing effect itself becomes unstable. To prevent this, in the velocity smoothing step (S420), the original velocity (v t The pre-set velocity smoothing coefficient (α) in ) v By applying ) and the operation of [Equation 2] below, the reference movement speed (V t ) can be produced.

[0148] [Equation 2]

[0149]

[0150] Here, the preset velocity smoothing coefficient (α v ) corresponds to a preset value between 0 and 1, and V t-1 corresponds to the reference movement speed calculated in the previous frame. That is, the above [Equation 2] is the raw speed (v) of the current frame. t ) and the reference movement speed of the previous frame (V t-1 It can be composed of the operation of an exponential moving average in the form of a weighted average of ), through which a stable reference moving speed (V) with temporary speed fluctuations effectively mitigated. t ) can be derived.

[0151] The above coordinate correction step (S430) uses the reference moving speed (V) calculated in the above speed smoothing step (S420). t Smoothing coefficient (α) based on ) t Dynamically calculates ), and through a velocity-sensitive adaptive filter applying it, obtains the final corrected coordinates (XC) of the current frame. t This corresponds to the step of deriving ). More specifically, the velocity-sensitive adaptive filter utilized in the coordinate correction step (S430) obtains the final corrected coordinates (XC) of the current frame by the operation of [Equation 1] below. t ) can be produced.

[0152] [Equation 1]

[0153]

[0154] Here, αt is a smoothing coefficient having a value between 0 and 1, which can be dynamically calculated based on the reference movement speed (Vt). That is, [Equation 1] is the original coordinates (X) of the current frame t ) and the final corrected coordinates of the previous frame (XC t-1 ) smoothing coefficient (α t It is configured in a form that combines using ) as weights, and the smoothing coefficient (α t The reflection ratio of both coordinates can be dynamically adjusted depending on the value of ).

[0155] In one embodiment of the present invention, the smoothing coefficient (α t ) is the above reference movement speed (V t It can be calculated based on a cutoff frequency determined in proportion to the magnitude of ). More specifically, the cutoff frequency is the smoothing coefficient (α t As an intermediate variable determining the behavior of ), the above reference moving speed (V t The larger ) is, the higher the value is set, and the above reference movement speed (V t The smaller ) is, the lower the value can be set. Based on this cutoff frequency, the smoothing coefficient (α t ) may exhibit the following behavior.

[0156] In one embodiment of the present invention, the reference moving speed (V t If the size of ) is smaller than the preset speed standard, that is, if the lesion object is substantially stationary or moving very slowly on the screen, the smoothing coefficient (α t ) may be calculated as a relatively small value. In such cases, the final corrected coordinates (XC of the previous frame) in [Equation 1] t-1 (1-α), which is the reflection ratio of ) t As ) becomes relatively higher, the original coordinates of the current frame (X tFine coordinate fluctuations included in ) are effectively attenuated, thereby effectively suppressing fine shaking phenomena on a stationary lesion object.

[0157] On the other hand, the above reference movement speed (V t If the magnitude of ) is greater than the above speed standard, that is, if the lesion object moves rapidly on the screen due to the operator's endoscopic manipulation, the above smoothing coefficient (α t ) may be calculated as a relatively large value. In such cases, the original coordinates of the current frame (X in [Equation 1]) t As the reflection ratio αt of ) becomes relatively higher, the location of the rapidly changing actual lesion object is immediately reflected in the visual identifier, thereby effectively reducing tracking lag caused by smoothing processing.

[0158] That is, the speed-sensitive adaptive filter according to one embodiment of the present invention has a smoothing coefficient (α t It is characterized by the fact that ) is not fixed to a single fixed value but is dynamically varied according to the movement speed of the lesion object.

[0159] The final display step (S440) above corresponds to a step of generating a visual identifier including a bounding box indicating the location of the lesion object based on the final corrected coordinates (XCt) of the current frame calculated in the coordinate correction step (S430), and overlaying it on the image stream. More specifically, in the final display step (S440), a bounding box is generated with the final corrected coordinates (XCt) as the center coordinates and size, and additionally, a label including class information (e.g., lesion type) or a reliability score of the lesion object is generated together and composited onto the image stream, after which it can be output to the display.

[0160] Figure 8 illustrates a comparison of the coordinate behavior of a lesion object bounding box before and after the application of a velocity-sensitive adaptive filter according to an embodiment of the present invention. As shown in the left graph of Figure 8, in the case of a conventional technology in which the visual stabilization technology of the present invention is not applied, it can be observed that the y-coordinate of the bounding box for the same lesion object fluctuates with significant noise and waveform jumps as the frame progresses, and this manifests as box flickering and jitter phenomena of the bounding box on the screen displayed to the operator. On the other hand, as shown in the right graph of Figure 8, when the velocity-sensitive adaptive filter according to an embodiment of the present invention is applied, the change in the y-coordinate of the bounding box for the same lesion object is derived in the form of a smooth curve, and thereby, it can be observed that the movement of the bounding box displayed to the operator is visualized in a smooth and stable form.

[0161] That is, according to one embodiment of the present invention, through a speed-sensitive adaptive filter that applies a smoothing coefficient (αt) dynamically calculated based on the movement speed of a lesion object, jittering is effectively suppressed for a lesion object in a stationary state, and adaptive visual stabilization is simultaneously achieved in which tracking lag is minimized for a lesion object in a moving state. Through this, the behavior of the track ID stably maintained in the tracking update step (S300) and the visual identifier smoothly corrected in this step (S400) are combined, thereby maximizing the visual stability of the overlay screen finally displayed to the operator and effectively reducing the visual fatigue of the operator due to a long endoscopic examination.

[0162] Below, we will explain in more detail how the difference in coordinate behavior shown in Fig. 8 manifests as visual differences in actual clinical environments.

[0163] In one embodiment, a situation can be assumed where the operator precisely observes a micropolyp captured in the center of the screen while the endoscope is stopped. In this case, since the lesion on the actual mucosa is effectively in a stationary state, it is natural for the bounding box displayed to the operator to also be stationary. However, as illustrated in the left graph of Fig. 8, in the case of the prior art, minute variability accumulates in the frame-by-frame inference results of the neural network model, causing the y-coordinate of the bounding box to constantly vibrate up and down by a few pixels, even for a lesion that is actually stationary. This causes the bounding box to appear as if it is shaking slightly in the operator's field of view, acting as a factor that hinders the operator's visual concentration during the precise observation phase.

[0164] In contrast, when a speed-sensitive adaptive filter according to one embodiment of the present invention is applied, the reference moving speed (V) calculated in the speed smoothing step (S420) t ) converges to a very small value, and accordingly, in the coordinate correction step (S430), the smoothing coefficient (α t ) is also calculated as a relatively small value. Thus, in [Equation 1], the final corrected coordinates (XC of the previous frame) t-1 As the reflection ratio of ) becomes very high, the original coordinates of the current frame (X t The minute vibration components included in ) are effectively dampened. As a result, as shown in the graph on the right in Fig. 8, the bounding box coordinates in the stationary section can be maintained in a substantially flat shape, and the operator can see a bounding box stably fixed on the lesion without minute vibrations.

[0165] In another embodiment, a situation can be assumed where a newly emerging lesion moves rapidly from the edge of the screen toward the center while the operator rapidly advances the endoscope along the colon lumen. In this case, the bounding box displayed to the operator must immediately follow the rapid positional change of the actual lesion; however, if smoothing is applied uniformly and strongly, a tracking lag phenomenon may occur where the bounding box lags behind the actual lesion's position. This tracking lag can cause a visual mismatch, where the bounding box only partially encloses the actual lesion or points to a mucosal area outside the lesion, making it difficult for the operator to intuitively perceive the exact location of the lesion.

[0166] In contrast, when a speed-sensitive adaptive filter according to one embodiment of the present invention is applied, the reference moving speed (V) calculated in the speed smoothing step (S420) t ) is derived as a large value, and accordingly, in the coordinate correction step (S430), the smoothing coefficient (α t ) is also calculated as a relatively large value. Thus, in [Equation 1], the original coordinates of the current frame (X t As the reflection rate of ) increases, the actual location of the rapidly changing lesion can be immediately reflected in the visual identifier. As a result, even in the section where the coordinates rise sharply in the graph on the right in Fig. 8, the bounding box can be displayed in a form that smoothly and immediately follows the movement trajectory of the actual lesion, allowing the operator to intuitively recognize the location of the lesion even during rapid examination.

[0167] Meanwhile, according to one embodiment of the present invention, the smoothing coefficient (α t ) is the above reference movement speed (V t) and calculated based on the detection reliability of the lesion object derived in the lesion detection step, wherein if the detection reliability is less than a preset reliability standard, the smoothing coefficient (α) is calculated in comparison to the case where the detection reliability is greater than or equal to the reliability standard. t ) is calculated as a relatively small value, so the final corrected coordinates (XC of the previous frame) t-1 The reflection ratio of ) may be relatively high.

[0168] This configuration corresponds to a configuration that is organically combined with the characteristic of the neural network model according to one embodiment of the present invention, which adaptively adjusts the detection reliability of a lesion object based on contextual situations within the image, such as the proximity of a medical device, as described in FIG. 5. That is, when a temporary reliability fluctuation occurs due to the proximity of a medical device or passing through atypical areas such as mucosal folds, the original coordinates (X) of the current frame derived by the neural network model t In situations where the reliability of ) temporarily decreases, the corresponding original coordinates (X t It can be determined that there is a relatively high probability that the smoothing coefficient (α) contains coordinate fluctuations resulting from uncertainty in the neural network inference step. In such cases, the smoothing coefficient (α) t By ensuring that ) is calculated as a relatively small value, the original coordinates (X) of the current frame, which have relatively low reliability, are utilized. t Lowering the reflection ratio of ) and the final correction coordinates (XC) that have been stably tracked until now t-1 Coordinate correction can be performed in a direction that increases the reflection ratio of ). Through this, the positional stability of the visual identifier can be effectively maintained even in sections where detection reliability temporarily drops.

[0169] On the other hand, in cases where the detection reliability is maintained at a sufficiently high level because the medical device is located in an area unrelated to the lesion or there are no significant interfering factors within the image, the above smoothing coefficient (α t ) is the above reference movement speed (V tIt can be calculated to follow the normal behavior according to ), and thereby the tracking responsiveness of normal times can be maintained.

[0170] That is, according to one embodiment of the present invention, the detection reliability derived from the contextual relationship between objects in an image by the neural network model is consistently utilized up to the calculation of the smoothing coefficient in the subsequent visual stabilization step, thereby forming a single pipeline in which adaptive reliability adjustment in the detection step and adaptive coordinate smoothing in the display step are organically combined. With this configuration, even in situations where the detection reliability temporarily fluctuates due to temporary interference by medical devices or partial occlusion by mucosal folds, the position of the visual identifier displayed to the operator is effectively stabilized, thereby enhancing the visual consistency of the display screen.

[0171] That is, the speed-sensitive adaptive filter according to one embodiment of the present invention can operate continuously without a separate mode switch. More specifically, in the section where the lesion was moving rapidly, a large smoothing coefficient (α t Immediate tracking occurs by applying ), and as the lesion stops, the reference movement speed (V t As ) gradually decreases, the smoothing coefficient (α t In addition, it naturally decreases, thereby allowing for a smooth transition to a stable display state where fine vibrations are effectively suppressed immediately after stopping. That is, the speed-sensitive adaptive filter according to one embodiment of the present invention adaptively behaves within a single algorithm structure in both stationary and moving situations, thereby enabling it to flexibly respond to various manipulation patterns of the operator during an inspection.

[0172] FIG. 9 schematically illustrates the output result of detection reliability by class of a lesion object according to one embodiment of the present invention, FIG. 10 schematically illustrates an overlay screen in which detection reliability is adaptively adjusted and output according to whether a medical device is identified within the current frame according to one embodiment of the present invention, and FIG. 11 schematically illustrates an example of an endoscopic image including a lesion object that is a target detected by a computing system according to one embodiment of the present invention.

[0173] Hereinafter, with reference to FIGS. 9 to 11, the output results derived by the computing system (1000) according to one embodiment of the present invention in an actual clinical environment and the characteristics of the image to be processed will be explained in more detail.

[0174] FIG. 9 illustrates an example of a class-specific detection confidence output result derived by the lesion detection unit (1200) according to an embodiment of the present invention for a single lesion object. As shown in FIG. 9, the neural network model according to an embodiment of the present invention can simultaneously output a probability value in the form of confidence that the lesion object corresponds to each of a plurality of predefined classes, along with a bounding box indicating the location of the single lesion object detected in the image. More specifically, in the example of FIG. 9, the neural network model can output a multi-class probability distribution for the detected lesion object in the form of a confidence of 88% for the Adenoma class, a confidence of 10% for the Hyperplastic class, and a confidence of 2% for the Sessile Serrated class.

[0175] In one embodiment of the present invention, the 'detection reliability' utilized in the tracking update step (S300) and the overlay display step (S400) may correspond to either the highest value among the reliability values ​​for each class shown in FIG. 9 (i.e., 88% of the adenoma class in the example above) or the sum of the reliability values ​​for all lesion classes (i.e., the overall probability that the object corresponds to any lesion). Meanwhile, in one embodiment of the present invention, the class corresponding to the highest value among the reliability values ​​for each class is determined as the representative class of the lesion object and may be used as a label for the visual identifier to be described later. However, the types of classes are not limited to the example in FIG. 9, and depending on the characteristics of the clinical environment to which the present invention is applied, a simple binary classification of lesion / non-lesion or a more detailed pathological classification system may be adopted.

[0176] Meanwhile, in one embodiment of the present invention, if the detection reliability is below a preset display threshold, the output of the visual identifier for the corresponding lesion object in the overlay display unit (1400) may be suppressed. This configuration can be combined with a situation in which the detection reliability is adaptively lowered based on the attention weight of the multi-head attention layer as the medical device approaches the lesion as described in FIG. 5, and can be utilized to prevent unnecessary visual identifiers from being displayed within the operator's field of vision during the procedure using the medical device.

[0177] FIG. 10 illustrates an example of a result in which the computing system (1000) according to an embodiment of the present invention outputs an overlay screen including an adaptively adjusted detection reliability based on whether a medical device is identified. As shown in the left image of FIG. 10, in a situation where no medical device appears in the image and a single polyp is clearly exposed on the mucous membrane, the neural network model can derive a high detection reliability of 0.91 for the polyp, and accordingly, the overlay display unit (1400) can synthesize and output a visual identifier including a solid line bounding box and a label 'Polyp 0.91' onto the image stream.

[0178] On the other hand, as illustrated in the right image of FIG. 10, when a surgical instrument (e.g., biopsy forceps) enters from the right side of the screen and approaches the polyp, as described in FIG. 5, the attention weight between the lesion object query and the medical instrument object query in the multi-head attention layer of the neural network model is calculated to be a relatively high value, and as a result, the detection reliability for the polyp may be adaptively lowered. In this case, in one embodiment of the present invention, the overlay display unit (1400) may operate to suppress the output of a visual identifier (e.g., bounding box and label) for the lesion object when the adaptively lowered detection reliability is below a preset display threshold. Accordingly, the 'Polyp' label and bounding box displayed on the polyp in the left image of Fig. 10 disappear from the operator's field of view in the right image of Fig. 10, and when the operator manipulates the medical instrument to enter the precision procedure stage close to the lesion, a separate visual identifier is not obstructed from the operator's field of view for precision manipulation. Through this, the lesion detection method and the computing system performing the same according to the present invention can effectively reduce the operator's visual fatigue and reduce the possibility of careless procedure errors that may occur due to the visual field being distracted by the visual identifier.

[0179] Meanwhile, in another embodiment of the present invention, the visual identifier may be configured such that its display method varies according to the magnitude of the detection reliability, instead of suppressing the output of the visual identifier itself. More specifically, in this other embodiment, the visual identifier may be displayed as a solid bounding box and a relatively dark label when the detection reliability is above a preset standard, and as a dotted bounding box and a relatively light label when the detection reliability is below the standard. However, the variable display method of the visual identifier is not limited to the above example, and various forms capable of visually distinguishing the detection reliability, such as changes in color, adjustment of transparency, and changes in label size, may be applied.

[0180] In other words, in the lesion detection method of the present invention, the overlay display step may adaptively lower the detection reliability of the lesion object based on the attention weight between the object query corresponding to the lesion and the object query corresponding to the medical device in the multi-head attention layer of the neural network model when the medical device is identified within the current frame in the lesion detection step. At this time, the overlay display limit may be configured to output a visual identifier for the lesion object onto the image stream when the adaptively lowered detection reliability is greater than or equal to a preset display threshold, and to suppress the output of the visual identifier for the lesion object when the adaptively lowered detection reliability is less than the display threshold.

[0181] That is, the neural network model according to one embodiment of the present invention is trained to internalize the multidimensional contextual relationship between the lesion and the medical device into the attention weights between the object queries of the multi-head attention layer during the training phase. Accordingly, during the inference phase, the clinical situation itself, such as the medical device entering the screen and proximity to the lesion, is immediately reflected as a natural change in attention weights without separate distance calculation or rule-based judgment, and the detection confidence, which is adaptively down-adjusted based on these changes in attention weights, is compared with a display threshold so that the suppression of the visual identifier output can be determined internally within a single neural network model pipeline. Accordingly, in the lesion detection method of the present invention, a single integrated processing structure is implemented in which the contextual relationship between objects learned during the detection phase is seamlessly linked to the suppression of the visual identifier output during the display phase. This structurally resolves the limitations of conventional separate post-processing methods, such as processing delay, non-adaptive behavior regarding clinical situations, and flickering of the visual identifier, while consistently providing the operator with a visual protection effect that is consistent with the clinical context.

[0182] FIG. 11 illustrates two representative examples of endoscopic images that are processed by the computing system (1000) according to one embodiment of the present invention, wherein FIG. 11 (a) corresponds to a refined polyp image with excellent visibility that is generally included in a conventional publicly available training dataset, and FIG. 11 (b) corresponds to an image of a microscopic polyp that is actually encountered by the operator during a clinical examination.

[0183] More specifically, the image shown in FIG. 11(a) corresponds to a close-up image taken by an endoscopy specialist close to the lesion to ensure diagnostic clarity, and it can be seen that a relatively large polyp is exposed with a clear outline in the center of the screen. For example, the polyp shown in FIG. 11(a) is a large lesion occupying an area of ​​approximately 30% or more of the total screen area in the image, and it may be a typical form of refined images that make up a majority of conventional public training datasets (e.g., Kvasir, CVC, etc.) or private training datasets of conventional technology. Such large lesions have excellent visibility and correspond to targets that can be detected with relatively high accuracy even with conventional convolutional neural network-based models.

[0184] On the other hand, the image shown in Fig. 11(b) corresponds to a distant image captured with a wide-angle field of view while the operator advances the endoscope, and it can be seen that an extremely small polyp is located adjacent to the mucosal folds in the upper right corner of the image. The polyp shown in Fig. 11(b) corresponds to an ultramicroscopic lesion occupying approximately less than 1% of the total screen area within the image, and its area is only one-tenth of that of the polyp shown in Fig. 11(a). Such an image not only has a very low ratio of the lesion to the screen area, but also has relatively low visual contrast between the lesion and the surrounding normal mucosa, and features atypical noise such as mucus, bubbles, and light reflections distributed in various parts of the screen. Therefore, it is a target that is clinically easy for the operator to miss and corresponds to the core target for which the present invention aims to improve detection performance.

[0185] That is, the images shown in FIG. 11 (a) and FIG. 11 (b) are all training data in which lesions are labeled within the images and correspond to the form of training data included in the first training data group described above, respectively exemplifying the form of training data having contrasting characteristics at both extremes in terms of the area ratio of lesions within the images. The neural network model according to one embodiment of the present invention can be trained to enable stable lesion detection not only for refined images with excellent visibility such as FIG. 11 (a), but also for images of ultra-fine lesions such as FIG. 11 (b), where conventional technology has virtually faced limitations in detection, by microtraining a pre-trained transformer-based model with the clinical domain-specific training dataset of the present invention.

[0186] That is, the computing system (1000) according to one embodiment of the present invention can provide rich diagnostic assistance information to the operator by outputting a precise reliability distribution across multiple classes for a single lesion object as illustrated in FIG. 9, and can improve visual consistency in a clinical environment and visual comfort for the operator during the procedure through a detection reliability that is adaptively adjusted according to the intervention situation of a medical device as illustrated in FIG. 10 and a corresponding variable visual identifier, and can provide the effect of stably exhibiting clinically meaningful detection performance even in micro or ultra-micro lesion environments that were difficult to handle with conventional technology as illustrated in FIG. 11.

[0187] FIG. 12 illustrates, in an exemplary manner, the internal configuration of a computing device according to one embodiment of the present invention.

[0188] The computing system (1000) mentioned in the description of FIG. 2 may include components of the computing device (11000) illustrated in FIG. 12, which will be described later.

[0189] As illustrated in FIG. 12, the computing device (11000) may include at least one processor (11100), memory (11200), peripheral interface (11300), input / output subsystem (I / O subsystem) (11400), power circuit (11500), and communication circuit (11600).

[0190] Specifically, the memory (11200) may include, for example, high-speed random access memory, magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. The memory (11200) may include software modules, instruction sets, or various other data required for the operation of the computing device (11000).

[0191] At this time, access to the memory (11200) from other components, such as the processor (11100) or the peripheral device interface (11300), can be controlled by the processor (11100). The processor (11100) may be composed of a single or multiple units and may include processors in the form of GPUs and TPUs to improve computational processing speed.

[0192] The above peripheral device interface (11300) can connect input and / or output peripheral devices of the computing device (11000) to the processor (11100) and the memory (11200). The processor (11100) can perform various functions for the computing device (11000) and process data by executing a software module or instruction set stored in the memory (11200).

[0193] The input / output subsystem (11400) may connect various input / output peripheral devices to the peripheral device interface (11300). For example, the input / output subsystem (11400) may include a controller for connecting peripheral devices such as a monitor, keyboard, mouse, printer, or, if necessary, a touchscreen or sensor to the peripheral device interface (11300). According to another aspect, the input / output peripheral devices may be connected to the peripheral device interface (11300) without passing through the input / output subsystem (11400).

[0194] The power circuit (11500) may supply power to all or part of the components of the terminal. For example, the power circuit (11500) may include one or more power sources such as a power management system, a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for power generation, management, and distribution.

[0195] The communication circuit (11600) may enable communication with another computing device using at least one external port. Alternatively, as described above, the communication circuit (11600) may enable communication with another computing device by including an RF circuit and transmitting and receiving an RF signal, also known as an electromagnetic signal, as needed.

[0196] The embodiment of FIG. 12 is merely an example of the computing device (11000), and the computing device (11000) may have some components shown in FIG. 12 omitted, additional components not shown in FIG. 12 added, or a configuration or arrangement that combines two or more components. For example, the computing device (11000) for a communication terminal in a mobile environment may include a touchscreen or sensors in addition to the components shown in FIG. 12, and the communication circuit (11600) may include a circuit for RF communication of various communication methods (Wi-Fi, 3G, LTE, 5G, 6G, Bluetooth, NFC, Zigbee, etc.). The components that can be included in the computing device (11000) may be implemented as hardware, software, or a combination of both hardware and software, including one or more integrated circuits specialized for signal processing or applications.

[0197] Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded on a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. An application to which the present invention is applied may be installed on a user terminal through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file upon a request from the user terminal.

[0198] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0199] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave in order to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be standardized and stored or executed in a standardized manner on a networked computing device. Software and data may be stored on one or more computer-readable recording media.

[0200] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0201] According to one embodiment of the present invention, the endoscopic image-based lesion detection method and computing system can improve the reliability of lesion detection and reduce the rate of lesion oversight by the operator even in clinical environments where interference by medical devices is frequent, by integrally recognizing the global context of the image and the contextual relationship between objects through an attention mechanism-based neural network model.

[0202] According to one embodiment of the present invention, the computing system adopts a structure in which an image receiving unit, a lesion detection unit, a tracking update unit, and an overlay display unit are organically combined on a single processing pipeline, thereby configuring the system so that no separate post-processing procedure intervenes between the detection stage and the display stage, so as to stably achieve real-time performance of 30fps or more.

[0203] According to one embodiment of the present invention, the neural network model can effectively resolve the problem of real-time performance impairment caused by anchor box dependency and non-maximum inhibition post-processing of conventional convolutional neural network-based methods by directly predicting the location and class of lesion objects based on learnable object queries through a hybrid architecture combining a convolutional neural network-based backbone and a transformer encoder-decoder structure or a vision transformer-based full-transformer architecture.

[0204] According to one embodiment of the present invention, the neural network model calculates attention weights between a lesion object query and a medical device object query in a multi-head attention layer and adaptively adjusts the detection reliability of the lesion object, thereby enabling the effect of flexibly responding to the intervention situation of a medical device without relying on a rule-based post-processing method.

[0205] According to one embodiment of the present invention, the training dataset adopts a structure in which a first training data group labeled with lesions and medical devices and a second training data group not labeled with lesions and medical devices are combined at a predetermined ratio, thereby enhancing the ability of a neural network model to identify atypical areas such as normal mucosa, light reflection, mucus, or bubbles as no-object states and effectively reducing the frequency of false positives in a clinical environment.

[0206] According to one embodiment of the present invention, the neural network model is based on the inherent data efficiency of a transformer structure that integrally calculates the mutual associations between all locations within an image through attention operations. Unlike conventional convolutional neural network-based models that required the acquisition of large-scale training data, this model can achieve the effect of stably securing clinically meaningful detection sensitivity by inferring the semantic identity of an object from the global context of the entire image using only a relatively small amount of training data.

[0207] According to one embodiment of the present invention, the tracking update unit adopts a multi-stage association tracking structure that sequentially performs first-stage matching and second-stage matching for high-reliability objects and low-reliability objects classified according to reliability scores, thereby stably maintaining the track ID of the same lesion object even in environments of temporary interference by medical devices or partial concealment by mucosal folds, and thus can exert the effect of suppressing the flickering phenomenon of the visual identifier.

[0208] According to one embodiment of the present invention, the tracking update unit integrates a camera motion compensation function, an external shape embedding-based matching function, and a width-height separation state estimation function, thereby producing a tracking result that is robust even to displacement across the entire screen and changes in the shape of atypical polyps caused by the operator's endoscope manipulation, and thus can minimize the track ID switching phenomenon.

[0209] According to one embodiment of the present invention, the overlay display unit applies a speed-sensitive adaptive filter in which the smoothing coefficient is dynamically varied based on the movement speed of the lesion object, thereby suppressing jittering for a stationary lesion object and reducing tracking lag for a moving lesion object, so as to simultaneously achieve adaptive visual stabilization within a single algorithm structure.

[0210] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents. Therefore, other implementations, other embodiments, and equivalents to the claims below are also within the scope of the claims. Explanation of the symbols

[0211] 1000: Computing System 1100: Image receiver 1200: Lesion detection unit 1300: Tracking Update Section 1400: Overlay Display Section S100: Image display stage S200: Lesion detection stage S300: Tracking Update Stage S400: Overlay Display Stage S410: Raw velocity calculation stage S420: Velocity smoothing stage S430: Coordinate correction step S440: Final display step

Claims

Claim 1 A lesion detection method based on endoscopic images performed in a computing system comprising one or more processors and one or more memories, comprising: an image reception step for receiving a real-time image stream from an endoscopic device; a lesion detection step for deriving a detection result including the location and class of a lesion object by detecting a lesion object using a neural network model having an attention mechanism-based encoder-decoder structure for a current frame of the image stream received in the image reception step; a tracking update step for updating tracking information for the lesion object based on the detection result of the lesion detection step; and an overlay display step for overlaying information about the lesion object tracked in the tracking update step onto the image stream and outputting it to a display unit; wherein no separate post-processing step is performed between the lesion detection step and the tracking update step, and the neural network model is a model in which the contextual relationship between the lesion and one or more other objects other than the lesion is learned based on a training dataset in which the lesion and one or more other objects other than the lesion are labeled. Claim 2 A lesion detection method according to claim 1, wherein in the lesion detection step, the neural network model comprises: an encoder that calculates global interactions for features derived from the current frame through an attention operation; and a decoder that predicts the location and class of the lesion object based on a learnable object query, wherein the decoder does not utilize an anchor box. Claim 3 A lesion detection method according to claim 2, wherein the neural network model is a Full-Transformer architecture combining a Vision Transformer (ViT)-based backbone and a Transformer decoder. Claim 4 A method for detecting a lesion according to claim 1, wherein other objects other than the lesion include a medical device, and the training dataset includes training data in which the medical device is labeled, so that the neural network model is trained to distinguish the lesion and the medical device as distinct classes, and when the medical device is identified within the current frame in the lesion detection step, the detection reliability of the lesion object is adaptively adjusted based on the attention weight between the object query corresponding to the lesion and the object query corresponding to the medical device in the multi-head attention layer of the neural network model. Claim 5 A method for detecting a lesion according to claim 1, wherein the training dataset comprises: a first training data group in which at least one of a lesion or a medical device is labeled; and a second training data group in which no labeling is assigned to an object including a lesion and a medical device, and the ratio of the first training data group to the second training data group within the training dataset is in the range of 7:3 to 9.5:0.

5. Claim 6 A lesion detection method according to claim 1, wherein in the tracking update step, tracking information is updated by associating a lesion object with an existing track or creating a new track, and when associating a lesion object with an existing track, a first matching step is performed to first match a first detection object having a preset high reliability standard or higher among the detection results with the existing track; and for an existing track not matched in the first matching step, a second matching step is performed to second match a second detection object having a preset low reliability standard or lower among the detection results. Claim 7 In claim 1, the overlay display step comprises the original coordinates (X) of the current frame for the lesion object tracked in the tracking update step. t ) and the final corrected coordinates (XC) corrected in the previous frame t-1 Using ), the smoothing coefficient (α) according to the movement speed of the lesion object t By applying a velocity-sensitive adaptive filter in which ) is dynamically varied, the final corrected coordinates (XC) of the current frame t A coordinate correction step in which ) is calculated; and a final display step in which a visual identifier including a bounding box indicating the location of the lesion object is generated based on the final corrected coordinates calculated in the coordinate correction step and overlaid on the image stream; are performed, and the velocity-sensitive adaptive filter uses the following [Equation 1] to calculate the final corrected coordinates (XC t A lesion detection method that is a filter producing ). [Equation 1] Claim 8 In claim 7, the overlay display step comprises the original coordinates (X t ) and the final corrected coordinates (XC) of the previous frame mentioned above t-1 Dividing the difference of ) by the inter-frame time interval (Δt) gives the raw velocity (v) of the above-mentioned lesion object. t A raw velocity calculation step for calculating ); the raw velocity (v t The pre-set velocity smoothing coefficient (α) in ) v By applying ) and the operation of [Equation 2] below, the reference movement speed (V t Further performing a detailed step of a velocity smoothing step that calculates ), and the velocity-sensitive adaptive filter, wherein the reference moving speed (V t Smoothing coefficient (α) dynamically calculated based on ) t A lesion detection method applying ) [Equation 2] (here, α v is a preset value between 0 and 1, V t-1 corresponds to the reference movement speed calculated in the previous frame.) Claim 9 In claim 8, the smoothing coefficient is calculated based on a cutoff frequency determined in proportion to the magnitude of the reference moving speed, wherein if the magnitude of the moving speed is smaller than a preset speed reference, the smoothing coefficient (α t Since ) is calculated as a relatively small value, the reflection ratio of the final correction coordinates of the previous frame becomes relatively high, thereby suppressing jittering on the lesion object, and when the magnitude of the movement speed is greater than the speed standard, the smoothing coefficient (α t A lesion detection method in which the tracking lag for the lesion object is reduced by calculating ) as a relatively large value, thereby increasing the reflection ratio of the original coordinates of the current frame relatively high. Claim 10 A computing system comprising one or more processors and one or more memories, and performing a lesion detection method based on endoscopic images, comprising: an image receiving unit that receives a real-time image stream from an endoscopic device; a lesion detection unit that detects a lesion object using a neural network model having an attention mechanism-based encoder-decoder structure for a current frame of the image stream received by the image receiving unit, thereby deriving a detection result including the location and class of the lesion object; a tracking update unit that updates tracking information for the lesion object based on the detection result of the lesion detection unit; and an overlay display unit that overlays information about the lesion object tracked by the tracking update unit onto the image stream and outputs it to a display unit; wherein the detection result of the lesion detection unit is transmitted to the tracking update unit without performing separate post-processing, and the neural network model is a model in which the contextual relationship between the lesion and the other object other than the lesion is learned based on a training dataset in which the lesion and one or more other objects other than the lesion are labeled.

Images