Devices for detecting gastrointestinal lesion(s) and methods for operating the same

Abstract

A device and method for automated analysis of the real time endoscopic feed of an endoscopic procedure to detect gastrointestinal (GI ) lesions in a subject by analyzing endoscopic media in real-time, wherein Artificial Intelligence / Machine Learning (AI / ML) can be used to identify and highlight suspicious lesion ( s ) within the GI tract of the subject. The system can also detect the gastrointestinal (GI ) lesion (s ) in a subject by analyzing endoscopic media in real-time, which can integrate with endoscopic equipment, offering advanced media analysis, and enhancing the diagnostic yield during endoscopic procedures.

Description

Devices for detecting gastrointestinal lesion(s) and methods for operating the sameCROSS REFERENCE TO RELATED APPLICATIONThis application is based on and derives the benefit of Indian Provisional Application IN 202411027822, the contents of which are incorporated herein by reference.TECHNICAL FIELsD

[0001] Embodiments disclosed herein relate to endoscopy systems, and more particularly to detecting gastrointestinal (GI) lesion(s) in a subject by analyzing real-time endoscopic media.BACKGROUND

[0002] Endoscopy is a medical procedure that allows for the direct visual examination of the interior of the gastrointestinal (GI) tract of a subject. Endoscopy can be used for diagnosing, and in many cases, treating conditions affecting the GI system. Endoscopy involves the use of an endoscope, wherein the endoscope can comprise a flexible tube equipped with a light source, water and suction connectors, and a camera at its tip. The endoscope can serve as a diagnostic tool, and as a facilitator for various therapeutic maneuvers and interventions. During an endoscopic procedure, the endoscope is carefully inserted through the mouth or rectum of the subject into the gastrointestinal (GI) tract of the subject, providing real-time media feed for a qualified personnel (such as, but not limited to, endoscopists, gastroenterologists, and so on) to review on a display screen. Through this method, the qualified personnel can visually assess the mucosa to identify potential lesions or abnormalities for further examination or intervention, such as, but not limited to, biopsies for histopathological analysis. Further, the qualified personnel can also use the endoscope for a variety of therapeutic procedures including, but not limited to, resection, dissection, achieving hemostasis, ligation of vessels, cessation of bleeding, the excision of abnormal tissue, and so on.

[0003] The diagnosis and subsequent interventions hinge primarily on the visual assessment of the gastrointestinal mucosa, a process that is highly dependent on the personnel's experience, and real-time observation skills. This is a highly subjective exercise, and is prone to various inter and intra observer variations. However, identifying subtle lesions on themonitor for even a split second can pose a challenge, resulting in a relatively high miss rate of detection of debilitating cancerous and non-cancerous lesions of the GI tract.

[0004] Despite the critical role of GI endoscopy in medical diagnostics, the current methodology is not without its limitations. One significant challenge is the high degree of interobserver variability, attributed to the reliance on the personnel's subjective assessment and expertise. This variability can lead to inconsistencies in diagnosis and treatment outcomes, and can be a contributing factor to the considerable miss rate of critical diagnoses. The primary challenge lies in the inherent limitations of visual diagnosis during endoscopy, which can lead to significant rates of missed or inaccurate diagnoses. Early-stage diseases, which may exhibit subtle changes in the mucosa, are particularly prone to being overlooked. The lack of automated, real-time detection systems amplifies these difficulties, leaving room for human error.

[0005] In an example scenario, gastric cancer, which ranks as the fourth most common cancer globally, exhibits an 11% miss rate during endoscopic procedures. Further, studies suggest that up to 25% of early-stage lesions in early gastric cancer may be overlooked. Early detection and intervention are crucial for halting the disease's progression, underscoring the need for improvements in diagnostic accuracy and reliability. Similarly, studies show that approximately 30% of all lesions are missed during these procedures.

[0006] Hence, there is a need in the art for solutions which will overcome the above mentioned drawback(s), among others.OBJECTS

[0007] The principal object of embodiments herein is to disclose methods and devices for automated analysis of the real time endoscopic feed of an endoscopic procedure, including, but not limited to, detecting gastrointestinal (GI) lesion(s) in a subject by analyzing endoscopic media in real-time, wherein Artificial Intelligence / Machine Learning (AI / ML) can be used to identify and highlight suspicious lesion(s) within the GI tract of the subject, and for use cases, such as, but not limited to, assessing the completeness of the procedure, providing real time depth estimation of the tissue, and so on.

[0008] Another object of embodiments herein is to disclose methods and devices for detecting gastrointestinal (GI) lesion(s) in a subject by analyzing endoscopic media in realtime, which can integrate with endoscopic equipment, offering advanced media analysis, and enhancing the diagnostic yield during endoscopic procedures.

[0009] These and other aspects of the embodiments herein will be better appreciated and understood when considered in conjunction with the following description and the accompanying drawings. It should be understood, however, that the following descriptions, while indicating at least one embodiment and numerous specific details thereof, are given by way of illustration and not of limitation. Many changes and modifications may be made within the scope of the embodiments herein without departing from the spirit thereof, and the embodiments herein include all such modifications.BRIEF DESCRIPTION OF FIGURES

[0010] Embodiments herein are illustrated in the accompanying drawings, throughout which like reference letters indicate corresponding parts in the various figures. The embodiments herein will be better understood from the following description with reference to the following illustratory drawings. Embodiments herein are illustrated by way of examples in the accompanying drawings, and in which:

[0011] FIGs. 1A, IB, and 1C illustrate an endoscopy system, according to embodiments as disclosed herein;

[0012] FIG. 2 depicts the detection device, according to embodiments as disclosed herein;

[0013] FIGs. 3A, and 3B depicts example media streams with one or more detected lesions, according to embodiments as disclosed herein; and

[0014] FIG. 4 illustrates a flow chart of a method for enhancing endoscopic diagnostics using real-time processing and advanced visualization techniques, according to embodiments as disclosed herein.DETAILED DESCRIPTION

[0015] The embodiments herein and the various features and advantageous details thereof are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques are omitted so as to not unnecessarily obscure the embodiments herein. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein may be practiced and to further enable those of skill in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.

[0016] For the purposes of interpreting this specification, the definitions (as defined herein) will apply and whenever appropriate the terms used in singular will also include the plural and vice versa. It is to be understood that the terminology used herein is for the purposes of describing particular embodiments only and is not intended to be limiting. The terms “comprising”, “having” and “including” are to be construed as open-ended terms unless otherwise noted.

[0017] The words / phrases "exemplary", “example”, “illustration”, “in an instance”, “and the like”, “and so on”, “etc ”, “etcetera”, “e.g.,” , “i.e.,” are merely used herein to mean "serving as an example, instance, or illustration." Any embodiment or implementation of the present subject matter described herein using the words / phrases "exemplary", “example”, “illustration”, “in an instance”, “and the like”, “and so on”, “etc.”, “etcetera”, “e.g.,” , “i.e.,” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0018] Embodiments herein may be described and illustrated in terms of blocks which carry out a described function or functions. These blocks, which may be referred to herein as managers, units, modules, hardware components or the like, are physically implemented by analog and / or digital circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits and the like, and may optionally be driven by a firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like. The circuits constituting a block may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware to perform some functions of the block and a processor to perform otherfunctions of the block. Each block of the embodiments may be physically separated into two or more interacting and discrete blocks without departing from the scope of the disclosure. Likewise, the blocks of the embodiments may be physically combined into more complex blocks without departing from the scope of the disclosure.

[0019] It should be noted that elements in the drawings are illustrated for the purposes of this description and ease of understanding and may not have necessarily been drawn to scale. For example, the flowcharts / sequence diagrams illustrate the method in terms of the steps required for understanding of aspects of the embodiments as disclosed herein. Furthermore, in terms of the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the present embodiments so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein. Furthermore, in terms of the system, one or more components / modules which comprise the system may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the present embodiments so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

[0020] The accompanying drawings are used to help easily understand various technical features and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. As such, the present disclosure should be construed to extend to any modifications, equivalents, and substitutes in addition to those which are particularly set out in the accompanying drawings and the corresponding description. Usage of words such as first, second, third etc., to describe components / elements / steps is for the purposes of this description and should not be construed as sequential ordering / placement / occurrence unless specified otherwise.

[0021] The embodiments herein achieve methods and devices for detecting gastrointestinal (GI) lesion(s) in a subject by analyzing endoscopic media in real-time. Referring now to the drawings, and more particularly to FIGS. 1 through 4, where similar reference characters denote corresponding features consistently throughout the figures, there are shown embodiments.

[0022] Embodiments herein use the terms; subject, patient, and so on, to refer to the subject on whom the endoscopy procedure is being performed. Examples of the subject can be, but not limited to, humans, animals, birds, and any other living organism on which an endoscopy procedure can be performed.

[0023] Embodiments herein use the term ‘personnel’ to refer to a person who is performing the endoscopy procedure and / or viewing the media as captured by the endoscope. Embodiments also the term ‘observer’ interchangeably herein to refer to the personnel. Examples of the personnel can be, but not limited to, an endoscopist, a gastroenterologist, a doctor, a nurse, a nurse practitioner, and so on.

[0024] Media as referred to herein can refer to one of images, video, and so on, as received from a camera in the endoscope.

[0025] Embodiments herein can receive input in the form of endoscopic media in realtime. Embodiments herein can process the received input into frames, and run various preprocessing steps including, but not limited to, pixel normalization on the processed frames, etc.. Embodiments herein can run the processed and normalized frames through an Artificial Intelligence (Al) model to generate model predictions, generate bounding boxes around possible lesion(s), and send an output back to an endoscopy workflow for display on a display, without any perceivable time lag.

[0026] Embodiments herein disclose a gastrointestinal feed analysis system, for use in endoscopic systems (such as, but not limited to, upper gastrointestinal endoscopes, colonoscopes, sigmoidoscopes, and so on) to enhance diagnostic capabilities. The device can be integrated with an endoscope. The device can process endoscopy media in real-time to facilitate improved lesion detection, characterization, classification, and reduce observer subjectivity during gastrointestinal endoscopic procedures. Furthermore, embodiments herein can assess the completeness of the procedure, provide analysis on the depth of the tissue, and provide functions related to an automated analysis of the feed.

[0027] FIGs. 1A, IB, and 1C illustrate an endoscopy system. The system 100 comprises an endoscope 101, a gastrointestinal lesion detection device 102, and a display 103. The endoscope 101 can further comprise at least one camera, at least one light source, and one or more channels for instrumentation and suction (not shown). The endoscope 101 can enable one or more personnel to visualize and intervene within a gastrointestinal (GI) tract of the subject.

[0028] The endoscope 101 may be configured to connect to the display 103 which displays the media feed from the endoscope camera, providing the endoscopist with real-time media of the gastrointestinal tract. The system 100 further includes the gastrointestinal lesion detection device 102, alternatively referred to as a detection device 102, integrated with the with the media data pipeline of the endoscope 101, wherein the detection device 102 is connected between the endoscope 101, and the display 103 (as depicted in FIG. 1A). In an embodiment herein, the detection device 102 can interface through one or more integrated media processing units with the media data pipeline. In an embodiment herein, the detection device 102 can interface through one or more Application Programming Interfaces (APIs) with the media data pipeline.

[0029] In an embodiment herein, the detection device 102 can be integrated with the system 100 using a standard hardware interface, such as, but not limited to, Serial digital interface (SDI), High-Definition Multimedia Interface (HDMI), Universal Serial Bus (USB), USB-C, D-SUBminiature connectors, and so on. The detection device 103 can be integrated with existing endoscopic systems using the hardware interface.

[0030] In an embodiment herein, the detection device 102 can be inbuilt into the system 100, wherein the detection device 102 can be present internally to the system 100 (as depicted in FIG. IB).

[0031] In an embodiment herein, the detection device 102 can be implemented on a cloud (as depicted in FIG. 1C), wherein the endoscope 101 can communicate media in real time to the detection device 101 via a wireless communication means (such as, but not limited to, Wi-Fi, 3GPP networks, satellite networks, and so on), and the detection device 102 can communicate the output media to the display 103 (which may be co-located with the endoscope 101 and / or located remotely from the endoscope 101) in real time via a wireless communication means (such as, but not limited to, Wi-Fi, 3GPP networks, satellite networks, and so on).

[0032] In an embodiment herein, the detection device 102 can be deployed as an onpremise edge solution. For instance, the detection device 102 may be hosted on a dedicated GPU module or similar hardware accelerators, ensuring high-performance, real-time inference without relying on external or cloud-based resources. This on-premise setup helps maintain low latency, minimizes data transfer, and provides a secure environment by keeping patient media data locally within the clinical facility. The detection device’s hardware selection can be adapted in the future to accommodate upgrades in GPU or edge-processing modules, providedthey meet the necessary throughput and latency requirements for real-time endoscopic media analysis.

[0033] The detection device 102 inputs the media stream from the endoscope 101 and outputs processed media data to the display 103 in real-time. The display 103 can be a display device, that comprises at least one display, that can display the processed media from the detection device 102.

[0034] The display 103 may display the processed media feed, which includes realtime highlighting of detected lesions. In an embodiment herein, lesions can be marked with bounding boxes and a confidence score that indicates the probability of a lesion's presence; for example, "Polyp: 99%".

[0035] In an embodiment herein, the system 100 can evaluate the performance of the system based on clinical validation and confirm that the detection accuracy may exceed the set benchmarks, endoscopy as a method becomes more objective, and various tasks such as characterization and localization become more efficient solidifying the system’s efficacy in real-time diagnostic enhancement during endoscopic procedures. The system 100 can evaluate the performance of the system by identifying, labelling and capturing at least one anatomical landmark / station point during the endoscopic procedure. The system 100 can further determine whether known blind spots in the gastrointestinal tract were adequately examined during the procedure, and using this data, the system 100 can perform a 3D-reconstruction of the viewed surface in real time, providing objective insights on the blind spots, and various anatomical locations that were adequately examined, or not adequately examined during the procedure.

[0036] FIG. 2 depicts the detection device. The detection device 102 may integrate seamlessly with the endoscope’s media / graphics processor (not shown), thereby enhancing the functionality of the standard endoscopic equipment, and eliminating the need for new endoscopic hardware implementation. The detection device 102 can facilitate autonomous detection of lesions during the endoscopic procedure by processing real-time media data, and enabling the detection of gastrointestinal lesion(s) (if any). The detection device 102 comprises a pre-processing module 102A, an embedded control module 102B (hereinafter referred to as control module 102B), an output module 102C, at least one user interface 102D, at least one communication module 102E, and a memory 102F.

[0037] The pre-processing module 102A can be at least one of a single processor, a plurality of processors, multiple homogeneous or heterogeneous cores, multiple CentralProcessing Units (CPUs) of different kinds, microcontrollers, special media, and other accelerators. The pre-processing module 102A may be an Application Processor (AP), a graphics-only processing unit such as a Graphics Processing Unit (GPU), a Visual Processing Unit (VPU), and / or an Artificial Intelligence (Al) -dedicated processor such as a Neural Processing Unit (NPU).

[0038] The pre-processing module 102A can ensure that there is a continuous process of media feed into individual frames, which can be then optimized by the detection device 102 to ensure a consistent flow and quality of visual data. The pre-processing module 102A can capture real time endoscopy media feed. The pre-processing module 102A can pre-process the media of an endoscopic procedure in a suitable format to be utilized in the inference of a realtime computational model (as implemented by the control module 102B), utilizing pixel normalization and standardization procedures. The format can be dependent on the computational load specifications that the version of the detection device can handle, as well as internet connectivity speed in case of API based processing. The pixel normalization and standardization procedures include the breakdown of the media stream into individual frames, pre-processing of frames into various RGB channel filters, for standardized colour, de- pixelating the media into a smaller size, and sampling a certain number of frames per second for asynchronous / synchronous processing. The real-time computational model can be a computational model suitable for detecting lesions, such as, but not limited to, a Convolutional Neural Network (CNN), a neural network, or any other machine learning (ML) or computational model.

[0039] The pre-processing module 102A can capture the media output from the endoscope’s camera and initiate one or more pre-processing operations for optimizing the media stream for subsequent analytical processing, such as, but not limited to, frame breakdown, rate stabilization, resolution scaling, and color balancing, and so on. The pre- processed media is then provided to the control module 102B.

[0040] The control module 102B can be at least one of a single processor, a plurality of processors, multiple homogeneous or heterogeneous cores, multiple Central Processing Units (CPUs) of different kinds, microcontrollers, special media, and other accelerators. The control module 102B may be an Application Processor (AP), a graphics-only processing unit such as a Graphics Processing Unit (GPU), a Visual Processing Unit (VPU), and / or an Artificial Intelligence (Al) -dedicated processor such as a Neural Processing Unit (NPU).

[0041] The control module 102B can utilize a hardware-accelerated model inference process for real-time detection and analysis of potential lesions, and other inferences about the procedure (such as analysis of certain quality metrics, completeness identification etc.)

[0042] The control module 102B can pass the pre-processed media through each layer of a real-time computational model, which learns features of the distinct pathology. The control module 102B can use initial layers of the real-time computational model to detect one or more morphological features in the media, such as, but not limited to, the boundary and edges of lesion(s) (if any). The control module 102B can use final layers of the real-time computational model to detect finer details that are characteristic of the changes caused by the presence of the lesions(s) (if any), and formulate a definitive diagnosis based on the detected lesion(s).

[0043] The control module 102B can detect various lesions during real time procedures (if any), characterize the detected lesions during real time, classify the detected lesions according to medically accepted classification systems (such as, but not limited to, the Paris Classification, the Forrest Classification, the Vienna Classification, the Japanese Gastric Cancer Association (JGCA) Classification, and so on), perform manipulations (such as, but not limited to, determine the exact anatomical locations of the scope during the tract), estimate the real time location of the detected lesions in the gastrointestinal tract, estimate the completeness of the endoscopic procedure using one or more protocols (examples can be, but not limited to, using a Systematic Screening protocol for the stomach (SSS) protocol for assessment of completeness of upper gastrointestinal endoscopy procedures), measure various quality metrics associated with the procedure (including but not limited to bowel preparation score (for example, using the Boston Bowel Preparation Scale), withdrawal time, which are dependent on the visual clarity of visualization of bowel with respect to faecal contamination of the visual scene and so on), measure adequate visualization of blind spots during the procedure, perform real-time depth estimation of suspicious cancerous lesions during the procedure, perform a 3D reconstruction of the GI tract of the patient, and so on. The control module 102B can estimate one or more blind spots during the procedure by identifying one or more anatomical locations of the gastrointestinal tract, wherein the control module 102B uses at least one model trained on a database of various anatomical locations, and the control module 102B can check off each landmark / station point or known blind spot, on the control module 102B determining that the landmark / station point, or known blind spot is seen during the procedure. The control module 102B can provide a real-time analysis of percentage of completeness of the procedure on the basis of the number of blind spots, anatomical locations seen and other performance metrics.The control module 102B can perform 3-D reconstruction of the gastrointestinal mucosa (as visualized post the procedure) using a multifaceted completeness assessment method, and a depth estimation method. The control module 102 can use the completeness assessment method to assess the completeness of endoscopy (blind spots, visualized anatomical locations). The control module 102 can use the depth estimation method to identify the depth via visual estimation of the gastrointestinal mucosa (via the movements of the endoscope during the endoscopy procedure).

[0044] The control module 102B can utilize computational techniques (such as, but not limited to, advanced attention mechanism techniques, and so on) to prioritize areas in the frame of the media for accurate lesion detection. In addition to convolutional neural networks (CNNs), the control module 102B can also use temporal methods to compute and detect features from the pre-processed media. The control module 102B can identify and analyze spatial and temporal changes across consecutive frames of the media to assess the presence of potential lesions. The control module 102B can perform temporal analysis by synthesizing outputs over short intervals (for example, five frames of the media stream), to address and resolve issues related to overlapping predictions, which can lead to more coherent and consistent presentation of detected lesions over time. For identifying spatial changes, the control module 102B can consider one or more spatial characteristics (such as, but not limited to, texture and color differentiation). Considering both the temporal and spatial characteristics enables the control module 102B to perform accurate lesion highlighting, recognizing the importance of the temporal context in real-time procedures. In an embodiment herein, the control module 102B can optimize the latency of processing of the media feed. In an embodiment herein, the control module 102B can asynchronously process the frames. In an embodiment herein, the control module 102B can measure, and inference the temporal dependencies associated with the real time media feed.

[0045] In an embodiment herein, the control module 102B can use classification methods, which may or may not incorporate convolutional neural networks, transformers, and other varieties of real time computational methods, wherein the classification methods can be trained to perform tasks such as recognition of lesions, identification of normal mucosa, identification and classification of various anatomical landmarks, characterizing the appearance of lesions, classifying lesions, as well as monitoring completeness of procedures (indicated by a quality indicator) by monitoring the appearance of anatomical landmarks and blind spots, and computing the percentage of area or of these spots that were not adequatelyvisualized during the procedure. The control module 102B can further perform 3-D construction of the gastrointestinal surface, and predict the depth of at least one detected lesion in the GI tract. The control module 102B can use spatial techniques, such as, but not limited to, edge detection, detection and concatenation of various features, and various temporal frame processing techniques to determine the final predictions.

[0046] The control module 102B can determine a predictive score for each of the predictions.

[0047] The control module 102B can further generate an output for displaying in realtime on the display 103, along with the predictive score. The control module 102B can determine the predictive score on the basis of the confidence of the model(s), which is on the basis of the data it has previously been trained on. The model(s) have been trained on a wide variety of labelled endoscopic data, which enables the control module 102B to use the models to distinguish between normal anatomy, pathologies, and various classes / degrees of pathologies related to the GI tract. The control module 102B can use the features (as learnt during the training process) to determine the likelihood of prediction, which is computed as a predictive score. The control module 102B can positively determine the presence of a lesion, if the predictive score is beyond a predefined threshold value. The threshold value can be determined based on a calibration process. In an embodiment herein, the threshold can be determined by analyzing predictive scores on a validation dataset to optimize a trade-off between sensitivity and specificity. In an embodiment herein, the threshold may be configurable by the personnel, or any other authorized person. In an embodiment herein, the threshold may be derived from clinical standards or guidelines, to ensure an appropriate balance between minimizing false positives, and maximizing detection accuracy. In an embodiment herein, the threshold may be determined based on the subject, on whom the endoscopy procedure is being performed, wherein the historical data of the subject can be used for determining the threshold dynamically. The output can comprise at least one visual highlight. The at least one visual highlight can serve as visual marker(s) for the endoscopist, drawing attention to areas that require closer examination In an embodiment herein, the at least one visual highlight can be a bounding box (which can be of any shape, such as, but not limited to, square, rectangle, circular, triangular, and so on) for highlighting the predicted lesion in realtime during the endoscopic procedure (as depicted in FIGs. 3A, and 3B). In an embodiment herein, the at least one visual highlight can comprise a segmentation mask for highlighting the predicted lesion in real-time during the endoscopic procedure.

[0048] In an embodiment herein, the control module 102B can continuously monitor latency, and utilization of resources in the detection device 102 to ensure optimal performance during the endoscopic procedure.

[0049] In an embodiment herein, the control module 102B can conduct compatibility and integration checks to confirm that functionality of the detection device 102 with various models and makes of endoscopic equipment.

[0050] In an embodiment herein, the control module 102B can monitor operational status, logs events and errors, and performance parameter sensors for real-time analysis. The control module 102B can monitor and store operational status, logs events and errors, and performance parameter sensors in the memory 102F or any other suitable location (such as, but not limited to, the cloud, a data server, a file server, and so on).

[0051] The control module 102B can ensure the correct operation of software components and data integrity throughout the method (as disclosed herein). The control module 102B can continuously verify the correct execution of one or more components of the device 102, from the initial loading of the models / networks to the final output of processed media data, using error-checking methods, data integrity verification protocols, and real-time performance monitoring. The control module 102B can periodically validate each component’s operational status, ensuring that configurations, model weights, and data streams remain intact and uncorrupted. In an embodiment herein, the control module 102B can implement a dedicated watchdog process to continuously verify the correct execution of the device 102’s components, generating immediate alerts in the event of anomalies, unexpected performance variations, or process interruptions, and can automatically switch off / suspend the other modules (as present in the detection device 102), in case of emergencies.

[0052] In an embodiment herein, the control module 102B can log every event associated with the system 100, such as, but not limited to, system start-ups, shutdowns, activations, error occurrences, user inputs, and so on. The control module 102B can store the logged data in a suitable location, such as, but not limited to, the memory 102F, the cloud, a file server, a data server, and so on. This log can be used for troubleshooting, maintenance, and system optimization, ensuring consistent operational quality.

[0053] In an embodiment herein, the control module 102B can supervise data transmission within the system 100, ensuring that the flow of media frames from the input to the output is uninterrupted and free of errors. The monitoring may include verifying that datapackets are complete, correctly ordered, and not corrupted, safeguarding the quality and consistency of the diagnostic output displayed to the personnel.

[0054] In an embodiment herein, the control module 102A can implement a verification protocol that cross-checks the processes against a predefined set of standards to confirm their integrity. If discrepancies or malfunctions are identified, the control module 102A can halt the compromised processes, initiate a restart, or switch to a backup device / manual process to maintain continuous operation.

[0055] The output module 102C can be at least one of a single processor, a plurality of processors, multiple homogeneous or heterogeneous cores, multiple Central Processing Units (CPUs) of different kinds, microcontrollers, special media, and other accelerators. The output module 102C may be an Application Processor (AP), a graphics-only processing unit such as a Graphics Processing Unit (GPU), a Visual Processing Unit (VPU), and / or an Artificial Intelligence (Al) -dedicated processor such as a Neural Processing Unit (NPU).

[0056] The output module 102C can conduct post-processing on the media (as received from the control module 102B), and can project the post-processed media feed back into the endoscopic workflow. The output module 102C can post-process the media frame, wherein post-processing the media can include, but not limited to, restoring the original pixel quality, color schematics of the frame, and so on. In an embodiment herein, the output module 102C can compile the output (as generated by the control module 102B across a sequence of frames), improving the consistency and accuracy of lesion detection by reducing overlaps in predictions. Similarly, the output module 102C can concatenate an integrated output of the different methods as chosen for analysis by the user and / or by the control module 102B, and relayed to the display 103. The output module 102C can further overlay real-time media streams with bounding boxes to highlight detected lesions. In an embodiment herein, the output module 102C can include a predictive score with each box. In an embodiment herein, the output module 102C can provide additional information regarding the lesion along with the box. Examples of the additional information may include, but not limited to, a description of the morphological characteristics of the lesion, classification of lesion according to various scientifically and medically accepted systems, its location within the gastrointestinal tract, depth / size of the lesion, and so on. In an embodiment herein, the output module 102C can incorporate one or more explainability mechanisms wherein the Al model’s focus or attention maps are made partially viewable to the personnel, enabling them to understand which regions or features that the control module 102B relied on when flagging a particular suspicious area. Theexplainability mechanisms enable the personnel to visualize one or more features associated with the media that led to the prediction, and textual descriptions of such features may also accompany the feature map / heat map and predictive score. This fosters greater trust in the Al output and may facilitate training or educational use for less experienced endoscopists.

[0057] The user interface 102D can comprise one or more interfaces which will enable a personnel to interact with the detection device 102. Examples of the user interface 102D can be, but not limited to, a display, a touchscreen, one or more indicator lights, one or more switches / knobs / dials, a speaker, a microphone, and so on. The user interface 102D can incorporate a controllable interface, which can enable the personnel to switch off one or more functionalities of the detection device 102, change the mode of the detection device 102, and visualize alerts indicative of a detected lesion. The user interface 102D can enable the personnel to adjust various settings related to the detection device 102 and / or the endoscope 101, and review lesions (on being detected). The user interface 102D can be controlled to switch on / off / sleep various modules of the detection device 102. The user interface 102D can enable the personnel to perform tasks such as, but not limited to, gauge the characterization of the lesion, gauge depth estimation and location of lesion, and so on. The user interface 102D can enable the personnel to toggle between a variety of different methods, capable for detecting different lesions and performing additional functions. The user interface 102D can enable the personnel to further pause, record, or capture the process.

[0058] Based on the inputs from the personnel (for example, switch off one or more functionalities of the detection device 102, change the mode of the detection device 102, toggle between a variety of different methods, and visualize alerts indicative of a detected lesion), the control module 102B can perform one or more corresponding actions. Based on adjustments performed by the personnel on various settings related to the detection device 102 and / or the endoscope 101, the control module 102B can perform one or more corresponding adjustments. Based on inputs from the personnel (for example, switch on / off various modules of the detection device 102, pause, record, or capture the process, and so on), the control module 102B can perform the corresponding actions.

[0059] Consider an example scenario, wherein the user interface 102D comprises at least one display (not shown), wherein the display can display mandatory warnings and important instructions. The personnel can use the display to view the media stream, the detected lesions, the bounding boxes, the predictive score, lesion characterization details, anatomical landmarks, and so on. The personnel can use the display to view alerts, warnings, messagesand other information from the device. The display and the control module 102B can enable personnel to navigate through various functionalities, review detected lesions, and confirm or dismiss alerts generated by the system 100 / device 102. The display can provide a graphical representation of the data being processed, and offer a menu-driven system for accessing different features, enhancing the ease of use and flexibility of the device during endoscopic procedures.

[0060] In an embodiment herein, the at least one communication module 102E is configured to enable communication between the detection device 102, and at least one external entity (such as, but not limited to, the endoscope 101, the display 103, and so on) through a network or cloud. The at least one communication module 102E through which the detection device 102 and the at least one external entity communicate may include wired and / or wireless communication medium compatible with one or more different communication protocols. The at least one communication module 102E can use one or more ports, such as, but not limited to, Serial digital interface (SDI), High-Definition Multimedia Interface (HDMI), Universal Serial Bus (USB), USB-C, D-SUBminiature connectors, and so on. The at least one communication module 102E may be configured for communication through a network. The network may comprise, but are not limited to, Global Positioning System (GPS), Global System for Mobile Communications (GSM), Local Area Network (LAN), Wireless Fidelity (Wi-Fi) compatibility, Bluetooth Low Energy (BLE), Near-field Communication (NFC), and so on. The wireless communication may further comprise one or more of Bluetooth, Zonal Intercommunication Global Standard (ZigBee), short-range wireless communication such as Ultra-wideband (UWB), medium-range wireless communication (such as, but not limited to, Wi-Fi, or long-range wireless communication such as Third Generation (3G), Fourth Generation (4G), Fifth Generation (5G), Sixth Generation (6G), or Worldwide Interoperability for Microwave Access (WiMAX)), according to the usage environment.

[0061] In the embodiment shown herein, the at least one memory 102F may comprise one or more volatile and non-volatile memory components that are capable of storing data and instructions to be executed. Examples of the at least one memory 102F can be, but are not limited to, NAND, embedded Multimedia Card (eMMC), Secure Digital (SD) cards, Universal Serial Bus (USB), Serial Advanced Technology Attachment (SATA), solid-state drive (SSD), and so on. The at least one memory 102F may also include one or more computer-readable storage media. Examples of non-volatile storage elements may include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories(EPROM) or electrically erasable and programmable (EEPROM) memories. In addition, the at least one memory 102F may, in some examples, be considered a non-transitory storage medium. The term “non-transitory” may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. However, the term “non-transitory” should not be interpreted to mean that at least one memory 102F is non-movable. In certain examples, a non- transitory storage medium may store data that can, over time, change (for example, in Random Access Memory (RAM) or cache).

[0062] In an embodiment herein, the detection device 102 may include additional sensors and diagnostic tools (not shown) that analyze one or more performance parameters in real time. For instance, the detection device 102 may track the frame rate of media processing, the responsiveness of the system to user inputs, and the accuracy of the lesion detection. If a deviation from the expected performance benchmarks is detected, the detection device 102 can alert the user or initiate corrective measures to address the issue.

[0063] FIG. 4 illustrates a flow chart of a method for enhancing endoscopic diagnostics using real-time processing and advanced visualization techniques. At step 401, the detection device 102 initiates capturing a real-time media stream from an endoscope, when an endoscopic procedure is being performed. In step 402, the detection device 102 pre-processes the captured media stream, wherein pre-processing includes, but is not limited to, frame rate stabilization, color balancing, resolution scaling to condition the media data for analysis, and so on. In an embodiment herein, the detection device 102 pre-processes the media stream in real-time by integrating model outputs back into the original media stream within a minimal latency (for example, 30 milliseconds, 50 milliseconds or less), ensuring no perceivable disruption in the quality of the endoscopy procedure.

[0064] In step 403, the detection device 102 analyses spatial and temporal features in each frame of the media stream to detect one or more lesions in the GI tract in real time using convolutional neural networks, and one or more temporal algorithms and other techniques (optional), with optimized latency and computation requirements. In an embodiment herein, the detection device 102 may analyze the frames of the media stream using at least one method, as selected by a user, wherein the analysis may comprise generating a morphological description of lesions, classifying and characterizing lesions, identifying normal anatomy, determining a quality indicator of the endoscopy procedure, identifying completeness of endoscopy, estimating depth of lesions, performing a 3-dimensional reconstruction of the gastrointestinal tract, and so on. The detection device 102 can perform the analysis using oneor more models, which have been trained on endoscopy data for multi-class classification, delineation, motion and depth estimation tasks. The detection device 102 can generate the morphological description of the lesions using a real-time computational model, such as, but not limited to, a Convolutional Neural Network (CNN), a neural network, or any other machine learning (ML) or real-time computational model. The detection device 102 can classify and characterize the lesions using a real-time computational model, such as, but not limited to, a Convolutional Neural Network (CNN), a neural network, or any other machine learning (ML) or real-time computational model. The detection device 102 can one estimate the depth of one or more blind spots during the procedure using at least one real-time computational model, wherein the detection device 102 can identify the anatomical locations of the gastrointestinal tract. The model has been trained on a database of various anatomical locations, and the detection device 102 can check off each landmark, or known blind spot, on detecting the landmark, or known blind spot in the media (on seeing the landmark, or known blind spot in the media), during the procedure. The detection device 102 can provide a real time analysis of percentage of completeness of the procedure on the basis of the number of blind spots, anatomical locations seen in the media captured during the endoscopy procedure. The detection device 102 can perform 3-D reconstruction of the gastrointestinal mucosa using the multifaceted completeness assessment method, and the depth estimation method. The detection device 102 can use the multifaceted completeness assessment method to assess the completeness of the endoscopy procedure (blind spots, visualized anatomical locations). The detection device 102 can determine the quality indicator of the endoscopy procedure using a real-time computational model, such as, but not limited to, a Convolutional Neural Network (CNN), a neural network, or any other machine learning (ML) or real-time computational model. The detection device 102 can use the depth estimation method to identify the depth via visual estimation of the gastrointestinal mucosa (via the motion of the endoscope during the endoscopy procedure). These detection device 102 can concatenate the outputs of the completeness assessment method, and the depth estimation method to generate a 3-D map of the GI tract (as visualized post the endoscopy procedure).

[0065] At step 404, the detection device 102 consolidates the prediction of detections (based on the concatenated output of the completeness assessment method, and the depth estimation method), as selected by the user, and / or as selected and performed by the detection device 102. The detection device 102 can include one or more spatial models, and one or more temporal models. The detection device 102 can use the one or more spatial models foranalyzing individual frames for lesion morphology. The detection device 102 can use or more temporal models to assess the evolution of these features over sequential frames, operating in parallel. At step 405, the detection device 102 generates bounding boxes for each of the detected lesions, and determines a predictive score for each of the detected lesions.

[0066] In step 406, the detection device 102 post-processes the processed media frames to ensure that there is no pixel quality loss after consolidation, and so that the original media quality, and other relevant specifications of the media stream (such as, but not limited to, color quality, and so on) match the input media stream from the endoscope. The detection device102 can perform the matching by sequentially reversing the pre-processing techniques, to ensure that the original RGB values, original pixel quality, and original sequence of the frames is processed again. The final output can be dependent on the camera specifications of the endoscope. In step 407, the detection device 102 relays the post-processed frames to a display103 in real time, wherein the display 103 displays the post-processed frames in real time, and the post processed frames comprises of the processed media stream, with bounding boxes overlaid on the detected lesions. The various actions in method 400 may be performed in the order presented, in a different order or simultaneously. Further, in some embodiments, some actions listed in FIG. 4 may be omitted.

[0067] Embodiments herein perform real-time media stream processing, wherein embodiments herein can interface with a real-time media stream from the endoscope. Embodiments herein can process the media stream to optimize frame rate and quality, and integrates the model outputs back into the media stream with an exceptionally low latency of less than 1 second; i.e., there is no discernible delay that could potentially affect the efficacy of endoscopic procedures. The immediate processing ensures there is no perceivable alteration to the real-time nature of the endoscopic procedures. The real-time media stream processing may be integrated into existing endoscopy workflows via either an on-premise hardware accelerated on-premise device, with the methods embedded into it or by hosting the detection tool on a cloud-based platform for wireless integration.

[0068] Embodiments herein are compatible with existing endoscope devices. Embodiments herein can connect with any endoscopic system irrespective of the make or age through a variety of hardware connectivity options, ensuring seamless integration with existing medical workflows.

[0069] Embodiments herein can maintain the original media quality at the pixel level, even after model predictions and overlays. Thus, embodiments herein can ensure that the personnel’s view remains unaltered, mitigating any additional risks during the procedure. Embodiments herein can enhance the detection capabilities without modifying the existing endoscopy workflow, or purchasing of additional bulky hardware

[0070] Embodiments herein can be integrated into endoscopic procedures to deliver real-time insights. The detection device, as disclosed herein, has lightweight modeling specifications equipped with substantial memory and GPU power necessary for hosting the diagnostic methods without compromising real time capabilities.

[0071] Embodiments herein provide enhanced diagnostic capabilities of personnel by incorporating real-time detection mechanisms, automating workflow processes, and providing comprehensive coverage of the endoscopic media to minimize the risk of missed diagnoses. Embodiments herein can be integrated with existing endoscopic devices and corresponding hardware and software components to eliminate the need for implementation of new endoscopic apparatus. Embodiments herein can improve accuracy, efficiency, and ultimately, patient care outcomes.

[0072] The embodiments disclosed herein can be implemented through at least one software program running on at least one hardware device and performing network management functions to control the network elements. The elements include blocks which can be at least one of a hardware device, or a combination of hardware device and software module.

[0073] The embodiments disclosed herein describe methods and devices for detecting gastrointestinal (GI) lesion(s) in a subject by analyzing endoscopic media in real-time. Therefore, it is understood that the scope of the protection is extended to such a program and in addition to a computer readable means having a message therein, such computer readable storage means contain program code means for implementation of one or more steps of the method, when the program runs on a server or mobile device or any suitable programmable device. The method is implemented in at least one embodiment through or together with a software program written in e.g., Very high speed integrated circuit Hardware Description Language (VHDL) another programming language, or implemented by one or more VHDL or several software modules being executed on at least one hardware device. The hardware device can be any kind of portable device that can be programmed. The device may also include meanswhich could be e.g., hardware means like e.g., an ASIC, or a combination of hardware and software means, e.g., an ASIC and an FPGA, or at least one microprocessor and at least one memory with software modules located therein. The method embodiments described herein could be implemented partly in hardware and partly in software. Alternatively, the invention may be implemented on different hardware devices.

[0074] The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and / or adapt for various applications such specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the embodiments herein have been described in terms of embodiments and examples, those skilled in the art will recognize that the embodiments and examples disclosed herein can be practiced with modification within the scope of the embodiments as described herein.

Claims

STATEMENT OF CLAIMSWe claim:

1. A method (400) for performing an endoscopy, the method comprising: capturing (401), by a detection device (102), a real-time media stream of a gastrointestinal (GI) tract from an endoscope (101), wherein a personnel is performing an endoscopy procedure using the endoscope; pre-processing (402), by the detection device (102), the captured media stream in real-time; analyzing (403), by the detection device (102), at least one of at least one spatial feature, or at least one temporal feature, in each frame of the pre-processed media; detecting (403), by the detection device (102), at least one lesion in the GI tract, using at least one real-time computational model; generating (405), by the detection device (102), at least one visual highlight for each of the at least one detected lesion; determining (405), by the detection device (102), a predictive score for each of the at least one detected lesion; post-processing (406), by the detection device (102), the media frame, the generated at least one visual highlight, and the determined predictive score for each of the at least one detected lesion to generate a concatenated media stream, while maintaining quality as the captured media stream, and at least one relevant specification of the captured media stream; and displaying (407), by the detection device (102), the post-processed concatenated media stream on a display (103), wherein the displayed media stream comprises the media stream, the at least one visual highlight, and the determined predictive score for each of the at least one detected lesion.

2. The method, as claimed in claim 1, wherein pre-processing (402), by the detection device (102), the media stream comprises pre-processing the media in real-time in a format to be utilized by the real-time computational model, utilizing pixel normalization and standardization procedures.

3. The method, as claimed in claim 1, wherein the method comprises prioritizing, by the detection device (102), at least one area in each frame of the pre-processed media using at least one real-time computational model.

4. The method, as claimed in claim 1, wherein analyzing (403), by the detection device (102), the at least one of at least one spatial feature, or at least one temporal feature, in each frame of the pre-processed media comprises identifying and analyzing, by the detection device (102), the at least one of at least one spatial feature, or at least one temporal feature across consecutive frames of the pre-processed media to assess the presence of potential lesions using edge detection, and detection and concatenation of at least one detected feature.

5. The method, as claimed in claim 1, wherein the method further comprises generating, by the detection module (102), a morphological description of lesions using at least one realtime computational model.

6. The method, as claimed in claim 1, wherein the method comprises classifying and characterizing, by the detection module (102), at least one detected lesion using at least one real-time computational model.

7. The method, as claimed in claim 1, wherein the method comprises determining, by the detection module (102), a quality indicator of the endoscopy procedure using at least one real-time computational model, wherein the quality indicator is based on appearance of anatomical landmarks and blind spots in the pre-processed media, and a percentage of area or of the blind spots that were not adequately visualized during the endoscopy.

8. The method, as claimed in claim 1, wherein the method comprises estimating, by the detection module (102), depth of the at least one detected lesion using at least one real-time computational model.

9. The method, as claimed in claim 1, wherein the method comprises performing, by the detection device (102), a 3-dimensional (3D) reconstruction of the gastrointestinal tract.

10. The method, as claimed in claim 1, wherein the method comprises: monitoring, by the detected device (102), latency, and utilization of resources in the detection device 102; conducting, by the detected device (102), compatibility and integration checks to confirm that functionality of the detection device 102 with at least one model, and different types of endoscopic equipment; monitoring, by the detected device (102), operational status, logs events and errors, and performance parameter sensors for real-time analysis; ensuring, by the detected device (102), data integrity; andimplementing, by the detected device (102), at least one verification protocol.

11. A detection device (102) configured to: capture a real-time media stream of a gastrointestinal (GI) tract from an endoscope (101), wherein a personnel is performing an endoscopy procedure using the endoscope; pre-process the captured media stream in real-time; analyze at least one of at least one spatial feature, or at least one temporal feature in each frame of the pre-processed media; detect at least one lesion in the GI tract, using at least real-time computational model; generate at least one visual highlight for each of the at least one detected lesion; determine a predictive score for each of the at least one detected lesion; post-process the media frame, the generated at least one visual highlight, and the determined predictive score for each of the at least one detected lesion to generate a concatenated media stream, while maintaining quality as the captured media stream, and at least one relevant specification of the captured media stream; and display the post-processed concatenated media stream on a display (103), wherein the displayed media stream comprises the media stream, the at least one visual highlight, and the determined predictive score for each of the at least one detected lesion.

12. The detection device, as claimed in claim 11, wherein the detection device (102) is configured to pre-process the media stream in real-time in a format to be utilized by the real-time computational model, utilizing pixel normalization and standardization procedures.

13. The detection device, as claimed in claim 11, wherein the detection device (102) is configured to prioritize at least one area in each frame of the pre-processed media using an advanced attention mechanism technique.

14. The detection device, as claimed in claim 11, wherein the detection device (102) is configured to analyze the at least one of at least one spatial feature, or at least one temporal feature in each frame of the pre-processed media by identifying and analyzing the at least one of at least one spatial feature, or at least one temporal feature across consecutive frames of the pre-processed media to assess the presence of potential lesions using edge detection, and detection and concatenation of at least one detected feature.

15. The detection device, as claimed in claim 11, wherein the detection device (102) is configured to generate a morphological description of lesions using at least one real-time computational model.

16. The detection device, as claimed in claim 11, wherein the detection device (102) is configured to classify and characterize at least one detected lesion using at least one realtime computational model.

17. The detection device, as claimed in claim 11, wherein the detection device (102) is configured to determine a quality indicator of the endoscopy procedure using at least one real-time computational model, wherein the quality indicator is based on appearance of anatomical landmarks and blind spots in the pre-processed media, and a percentage of area or of the blind spots that were not adequately visualized during the endoscopy.

18. The detection device, as claimed in claim 11, wherein the detection device (102) is configured to estimate the depth of the at least one detected lesion using at least one realtime computational model.

19. The detection device, as claimed in claim 11, wherein the detection device (102) is configured to perform a 3-dimensional (3D) reconstruction of the gastrointestinal tract.

20. The detection device, as claimed in claim 11, wherein the detection device (102) is configured to: monitor latency, and utilization of resources in the detection device 102; conduct compatibility and integration checks to confirm that functionality of the detection device 102 with at least one model, and different types of endoscopic equipment; monitor operational status, logs events and errors, and performance parameter sensors for real-time analysis; ensure data integrity; and implement at least one verification protocol.

Images