Systems and methods for improving the quality of endoscopic images

By detecting and processing contaminated areas in endoscopic images in real time, and using machine learning and processor technology to generate clear images, the problem of visual field obstruction in minimally invasive surgery is solved, thereby improving image quality and surgical efficiency.

CN123124922APending Publication Date: 2026-11-17WOPE MEDICAL CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202580026743.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-02-16
Filing Date
2025-02-13
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In minimally invasive surgery, endoscopic images can be obscured by contaminants, leading to unclear vision, increased surgical error rate and time. Existing cleaning systems lack compatibility and efficiency, and cannot effectively solve this problem.

Method used

By using machine learning algorithms and processor technology, contaminated segments in endoscopic images are detected and processed in real time, resolution and frame rate are adjusted, replacement segments are generated to restore clear images, and image quality is optimized by combining cache memory management and visual cues.

Benefits of technology

It improves the clarity of endoscopic images in minimally invasive surgery, reduces visual obstruction, lowers surgical error rates and time, and improves surgical efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123124922A_ABST
    Figure CN123124922A_ABST
Patent Text Reader

Abstract

A method includes accessing a first image from an image stream; determining that a contaminated segment exists in the first image; determining at least one of a contaminant type or a size associated with the contaminated segment; adjusting at least one of the resolution of the image or the frame rate associated with the image stream, the adjustment being based on at least one of the contaminant type or the size of the contaminated segment, the adjustment resulting in at least one of a reduced-resolution image or a reduced image stream; inputting at least one reduced-resolution image or at least one extracted image obtained from the reduced image stream to a first machine learning algorithm (MLA) to generate a replacement segment of the contaminated segment; and generating a restored image based on the replacement segment.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-references This application claims priority to U.S. Provisional Patent Application No. 63 / 554,430, filed February 16, 2024, entitled "Systems and methods for improving image quality through an endoscopy", the contents of which are incorporated herein by reference in their entirety. Technical Field

[0002] This document describes various embodiments of systems and methods generally relating to improving the quality of endoscopic images. More specifically, the present invention relates to the real-time recovery of endoscopic video signals. Background Technology

[0003] The following paragraphs are provided as background information for this disclosure. However, this does not imply that anything discussed herein is part of the prior art or knowledge of someone skilled in the art.

[0004] Obscured vision during surgery is a problem encountered by surgeons in minimally invasive surgery (MIS), which, in contrast to open surgery, involves the introduction of instruments into the patient's body. One of these instruments is the endoscopic camera, which can be obscured by various types of contaminants such as blood, fog, blur, and fluids.

[0005] It is necessary to reduce, alleviate, or eliminate obstructed vision in surgical procedures (MIS). For example, surgical errors increase by 26% under stress. Fifteen million MIS procedures are performed worldwide each year. Furthermore, in 37% of cases, the surgical site is obstructed due to contamination.

[0006] This invention addresses a major problem in minimally invasive surgery, a type of surgery involving the introduction of cameras and instruments into the body through small incisions, as opposed to open surgery. During minimally invasive surgery, the field of vision is obstructed by contaminants such as blood, tissue, condensate in the form of fog, and smoke 37% of the time. According to Arora, Sonal, et al. “The Impact of Stress on Surgical Performance: A Systematic Review of the Literature.” Surgery, vol. 147, no. 3, 2010, doi:10.1016 / j.surg.2009.10.007, impaired vision is the leading stress factor during surgery, leading to increased error rates and longer surgical times. With over 200 million endoscopic procedures performed annually, this makes it one of the most pressing problems in surgery that needs to be addressed.

[0007] This is a well-known problem in minimally invasive surgery. In the field of cleaning systems, we have found various in-vivo cleaning methods such as mechanical systems, flushing devices, or energy-based devices, but none have been able to penetrate the market due to a variety of problems, such as lack of compatibility, lack of performance, and, in most cases, being very specific to one type of contaminant. The gold standard remains the removal of the scope from the body and manual cleaning, which makes each cleaning event highly interruptible during surgery.

[0008] Clearly, improved systems and methods are needed to address the aforementioned challenges or shortcomings. Summary of the Invention

[0009] Based on the teachings of this document, various implementations of systems and methods for improving the quality of endoscopic images, as well as computer products used therewith, are provided.

[0010] In a first broad aspect of this technology, a method for near real-time image enhancement of endoscopic images is provided, the method being executable by at least one processor, the method comprising: accessing a first image from an image stream; determining that a contaminated segment exists in the first image; determining at least one of (i) a type of contaminant associated with the contaminated segment or (ii) the size of the contaminated segment; adjusting at least one of (iii) an image resolution from the image stream or (iv) a frame rate associated with the image stream, the adjustment being based on at least one of (v) the type of contaminant or (vi) the size of the contaminated segment, the adjustment resulting in at least one of (vii) generating a reduced-resolution image from the image stream or (viii) a reduced-resolution image stream; inputting (ix) at least one reduced-resolution image from the reduced-resolution image or (x) at least one extracted image from the reduced-resolution image stream to a first machine learning algorithm (MLA) to generate a replacement segment of the contaminated segment; and generating a restored image based on the replacement segment.

[0011] In some embodiments of the method, the contamination type indicates the presence of at least one of smoke, fog, blood, fat, fumes, or surgical instruments.

[0012] In some embodiments of the method, the method further includes: accessing a second image from a cache memory storing cached images, wherein accessing the second image from the cache memory includes: determining at least one of the following: temporal relevance scores of a plurality of cached images, the temporal relevance scores indicating the time elapsed between capturing an individual cached image and a first image; or contextual relevance scores of a plurality of cached images, the contextual relevance scores indicating the degree of structural similarity between an individual cached image and the first image; and selecting a second image from the plurality of cached images based on the temporal relevance scores satisfying a temporal relevance threshold or the contextual relevance scores satisfying a contextual relevance threshold.

[0013] In some embodiments of the method, the method further includes: determining whether a first image is suitable for caching, the suitability being determined based on at least one image quality metric; and in response to determining that the first image is suitable for caching, storing the first image as one of a plurality of cached images in a cache memory.

[0014] In some embodiments of the method, at least one image quality metric includes one or more of the following: a sharpness metric determined based on at least one of edge gradient values ​​and the Laplacian variance; or a haze level metric determined based on at least one of dark channel prior (DCP) values ​​and fog perceived density evaluation (FADE) values; or an information richness metric determined based on entropy analysis.

[0015] In some embodiments of the method, an individual cached image is removed from the cache memory in response to at least one of the following: the temporal relevance score of the individual cached image does not exceed a temporal relevance threshold; and the contextual relevance score of the individual cached image does not exceed a contextual relevance threshold.

[0016] In some embodiments of the method, the method further includes selectively upscaling to restore the image.

[0017] In some embodiments of the method, the method further includes selectively increasing the frame rate associated with the image stream.

[0018] In some embodiments of the method, the method further includes selectively adjusting the size of the region of interest (ROI) used to determine where the contaminated segment exists within the first image.

[0019] In some embodiments of the method, the method further includes: generating a visual cue on the restored image, the visual cue indicating at least one of the following: a region corresponding to the replacement segment; and a reliability score of the replacement segment within the synthesized image frame.

[0020] In some embodiments of the method, determining that a contaminated segment exists within a first image includes: performing a second machine learning algorithm (MLA) by inputting the first image into the second MLA and having the second MLA output at least one of the following: (i) the type of contaminant associated with the contaminated segment or (ii) the size of the contaminated segment.

[0021] In some embodiments of the method, before determining that a contaminated segment exists within the first image, the method performs grouping of the first image with other images from the image stream.

[0022] In some implementations of the method, the determination of the presence of contaminated segments is performed on some images of the image stream, rather than all images.

[0023] In some embodiments of the method, adjusting (iii) the image resolution from the image stream or (iv) the frame rate associated with the image stream is based on the available computing resources of the device on which the method is performed.

[0024] In some embodiments of the method, before inputting (ix) at least one reduced-resolution image from a reduced-resolution image or (x) at least one extracted image from a reduced-resolution image stream to the first MLA, the method performs the following: selecting the first MLA from a plurality of MLAs based on at least one of the following: the type of contaminant associated with the contaminated segment; or the available computing resources of the device on which the method is performed.

[0025] In some embodiments of the method, the at least one extracted image obtained from the reduced image stream is further processed to reduce its resolution before being used to generate the replacement segment of the contaminated segment.

[0026] In some embodiments of the method, the method further includes displaying the restored image.

[0027] In other broad aspects of this technology, a method for near real-time image enhancement of endoscopic images is provided, the method being executable by at least one processor, the method comprising: accessing a first image from an image stream; determining that a contaminated segment exists in the first image; determining at least one of (i) a type of contaminant associated with the contaminated segment or (ii) the size of the contaminated segment; adjusting at least one of (iii) the resolution of the first image or (iv) the frame rate associated with the image stream, the adjustment being based on at least one of (v) the type of contaminant or (vi) the size of the contaminated segment, the adjustment resulting in at least one of (vii) generating a reduced-resolution image from the first image or (viii) a reduced image stream; inputting (ix) the reduced-resolution image or (x) at least one extracted image obtained from the reduced image stream into a machine learning algorithm (MLA) to generate a replacement segment of the contaminated segment; and generating a restored image based on the replacement segment.

[0028] In some embodiments of the method, the method further includes: accessing a second image from a cache memory storing cached images, wherein accessing the second image from the cache memory includes: determining at least one of the following: temporal relevance scores of a plurality of cached images, the temporal relevance scores indicating the time elapsed between capturing an individual cached image and a first image; or contextual relevance scores of a plurality of cached images, the contextual relevance scores indicating the degree of structural similarity between an individual cached image and the first image; and selecting a second image from the plurality of cached images based on the temporal relevance scores satisfying a temporal relevance threshold or the contextual relevance scores satisfying a contextual relevance threshold.

[0029] In some embodiments of the method, the method further includes displaying the restored image.

[0030] In other broad aspects of this technology, a method for near real-time image enhancement of endoscopic images is provided, the method being executable by at least one processor, the method comprising: accessing a first image from an image stream; determining that a contaminated segment exists in the first image; determining at least one of (i) a type of contaminant associated with the contaminated segment or (ii) the size of the contaminated segment; adjusting the resolution of the first image, the adjustment being based on at least one of (v) the type of contaminant or (vi) the size of the contaminated segment, the adjustment resulting in the generation of a reduced-resolution image from the first image; inputting (ix) the reduced-resolution image into a machine learning algorithm (MLA) to generate a replacement segment of the contaminated segment; and generating a restored image based on the replacement segment.

[0031] In some embodiments of the method, the method further includes: accessing a second image from a cache memory storing cached images, wherein accessing the second image from the cache memory includes: determining at least one of the following: temporal relevance scores of a plurality of cached images, the temporal relevance scores indicating the time elapsed between capturing an individual cached image and a first image; or contextual relevance scores of a plurality of cached images, the contextual relevance scores indicating the degree of structural similarity between an individual cached image and the first image; and selecting a second image from the plurality of cached images based on the temporal relevance scores satisfying a temporal relevance threshold or the contextual relevance scores satisfying a contextual relevance threshold.

[0032] In some embodiments of the method, the method further includes displaying the restored image.

[0033] In other broad aspects of this technology, a method for near real-time image enhancement of endoscopic images is provided, the method being executable by at least one processor, the method comprising: accessing a first image from an image stream; determining that a contaminated segment exists in the first image; determining at least one of (i) a type of contaminant associated with the contaminated segment or (ii) the size of the contaminated segment; adjusting the frame rate associated with the image stream, the adjustment being based on at least one of (v) the type of contaminant or (vi) the size of the contaminated segment, the adjustment resulting in a reduced image stream; inputting at least one extracted image from the reduced image stream into a machine learning algorithm (MLA) to generate a replacement segment of the contaminated segment; and generating a restored image based on the replacement segment.

[0034] In some embodiments of the method, the method further includes: accessing a second image from a cache memory storing cached images, wherein accessing the second image from the cache memory includes: determining at least one of the following: temporal relevance scores of a plurality of cached images, the temporal relevance scores indicating the time elapsed between capturing an individual cached image and a first image; or contextual relevance scores of a plurality of cached images, the contextual relevance scores indicating the degree of structural similarity between an individual cached image and the first image; and selecting a second image from the plurality of cached images based on the temporal relevance scores satisfying a temporal relevance threshold or the contextual relevance scores satisfying a contextual relevance threshold.

[0035] In some embodiments of the method, the method further includes displaying the restored image.

[0036] In other broad aspects of this technology, a method for near real-time image enhancement of endoscopic images is provided, the method being executable by at least one processor, the method comprising: accessing a first image from an image stream; determining that a contaminated segment exists in a region of interest (ROI) of the first image; determining at least one of (i) a type of contaminant associated with the contaminated segment or (ii) the size of the contaminated segment; estimating the duration required to perform image processing operations on the contaminated segment in response to the type of contaminant or the size of the contaminated segment being below a first threshold; selectively reducing at least one of the resolution of the first image, the rate associated with the image stream, and the size of the ROI based on the estimated duration; accessing a second image from a cache memory storing cached images; generating a prediction of a replacement segment for replacing the contaminated segment based on the second image; composite the replacement segment onto the contaminated segment to generate a composite image; and selectively outputting at least one of the first image and the composite image to a display.

[0037] In other broad aspects of this technology, a method for managing a cache memory storing multiple cached images is provided, the method comprising: accessing a first image from an image stream; accessing the cache memory storing the multiple cached images; determining at least one of: a temporal correlation score of the multiple cached images, the temporal correlation score indicating the time elapsed between capturing an individual cached image and the first image; or a contextual correlation score of the multiple cached images, the contextual correlation score indicating the degree of structural similarity between an individual cached image and the first image; and selecting a second image from the multiple cached images based on the temporal correlation score satisfying a temporal correlation threshold or the contextual correlation score satisfying a contextual correlation threshold; and returning the second image.

[0038] In some embodiments of the method, the method further includes: determining whether the first image is suitable for caching, the suitability being determined based on at least one image quality metric; and in response to determining that the first image is suitable for caching, storing the first image as one of the plurality of cached images in the cache memory.

[0039] In some embodiments of the method, at least one image quality metric includes one or more of the following: a sharpness metric determined based on at least one of edge gradient values ​​and the Laplacian variance; or a haze level metric determined based on at least one of dark channel prior (DCP) values ​​and fog perceived density evaluation (FADE) values; or an information richness metric determined based on entropy analysis.

[0040] In some embodiments of the method, an individual cached image is removed from the cache memory in response to at least one of the following: the temporal relevance score of the individual cached image does not exceed a temporal relevance threshold; and the contextual relevance score of the individual cached image does not exceed a contextual relevance threshold.

[0041] In some embodiments of the method, the method further includes selectively inputting a second image into a machine learning algorithm (MLA).

[0042] In another first broad aspect of this technology, a system for near real-time image enhancement of endoscopic images is provided, the system comprising at least one processor configured to: access a first image from an image stream; determine that a contaminated segment exists in the first image; determine at least one of (i) a type of contaminant associated with the contaminated segment or (ii) the size of the contaminated segment; adjust at least one of (iii) an image resolution from the image stream or (iv) a frame rate associated with the image stream, the adjustment being based on at least one of (v) the type of the contaminant or (vi) the size of the contaminated segment, the adjustment resulting in at least one of (vii) generating a reduced-resolution image from the image stream or (viii) a reduced-resolution image stream; input (ix) at least one reduced-resolution image from the reduced-resolution image or (x) at least one extracted image from the reduced-resolution image stream to a first machine learning algorithm (MLA) to generate a replacement segment of the contaminated segment; and generate a restored image based on the replacement segment.

[0043] In some embodiments of the system, the contamination type indicates the presence of at least one of smoke, fog, blood, fat, fumes, or surgical tools.

[0044] In some embodiments of the system, at least one processor is further configured to: access a second image from a cache memory storing cached images, the access of the second image from the cache memory comprising: determining at least one of the following: temporal correlation scores of a plurality of cached images, the temporal correlation scores indicating the time elapsed between capturing an individual cached image and a first image; or contextual correlation scores of a plurality of cached images, the contextual correlation scores indicating the degree of structural similarity between an individual cached image and the first image; and selecting a second image from the plurality of cached images based on the temporal correlation scores satisfying a temporal correlation threshold or the contextual correlation scores satisfying a contextual correlation threshold.

[0045] In some embodiments of the system, the at least one processor is further configured to: determine whether a first image is suitable for caching, the suitability being determined based on at least one image quality metric; and, in response to determining that the first image is suitable for caching, store the first image as one of a plurality of cached images in the cache memory.

[0046] In some embodiments of the system, at least one image quality metric includes one or more of the following: a sharpness metric determined based on at least one of edge gradient values ​​and Laplacian variance; or a haze level metric determined based on at least one of dark channel prior (DCP) values ​​or fog perceived density assessment (FADE) values; and an information richness metric determined based on entropy analysis.

[0047] In some implementations of the system, an individual cached image is removed from the cache memory in response to at least one of the following: the temporal relevance score of the individual cached image does not exceed a temporal relevance threshold; and the contextual relevance score of the individual cached image does not exceed a contextual relevance threshold.

[0048] In some embodiments of the system, at least one processor is further configured to selectively oversample and recover the image.

[0049] In some implementations of the system, at least one processor is further configured to selectively increase the frame rate associated with the image stream.

[0050] In some implementations of the system, at least one processor is further configured to selectively adjust the size of the region of interest (ROI) used to determine where the contaminated segment exists within the first image.

[0051] In some embodiments of the system, at least one processor is further configured to generate a visual cue on the restored image, the visual cue indicating at least one of the following: a region corresponding to the replacement segment; and a reliability score of the replacement segment within the synthesized image frame.

[0052] In some embodiments of the system, determining that a contaminated segment exists within a first image includes: executing a second machine learning algorithm (MLA) by inputting the first image into the second MLA and having the second MLA output at least one of the following: (i) the type of contaminant associated with the contaminated segment or (ii) the size of the contaminated segment.

[0053] In some embodiments of the system, the system performs grouping of the first image with other images from the image stream before determining that a contaminated segment exists within the first image.

[0054] In some implementations of the system, the presence of contaminated segments is determined for some images, rather than all images, of the image stream.

[0055] In some implementations of the system, adjusting at least one of (iii) the image resolution from the image stream or (iv) the frame rate associated with the image stream is based on the available computing resources of the system.

[0056] In some embodiments of the system, before inputting (ix) at least one reduced-resolution image from a reduced-resolution image or (x) at least one extracted image from a reduced-resolution image stream to the first MLA, the system performs the following: selecting the first MLA from a plurality of MLAs based on at least one of the following: the type of pollutant associated with the contaminated segment; or the available computing resources of the system.

[0057] In some embodiments of the system, the at least one extracted image obtained from the reduced image stream is further processed to reduce its resolution before being used to generate the replacement segment for the contaminated segment.

[0058] In some embodiments of the system, the system is configured to display a restored image.

[0059] In other broad aspects of this technology, a near real-time image enhancement system for endoscopic images is provided, the system being executable by at least one processor, the system comprising: accessing a first image from an image stream; determining that a contaminated segment exists in the first image; determining at least one of (i) a type of contaminant associated with the contaminated segment or (ii) the size of the contaminated segment; adjusting at least one of (iii) the resolution of the first image or (iv) the frame rate associated with the image stream, the adjustment being based on at least one of (v) the type of contaminant or (vi) the size of the contaminated segment, the adjustment resulting in at least one of (vii) generating a reduced-resolution image from the first image or (viii) a reduced image stream; inputting (ix) the reduced-resolution image or (x) at least one extracted image from the reduced image stream into a machine learning algorithm (MLA) to generate a replacement segment of the contaminated segment; and generating a restored image based on the replacement segment.

[0060] In some embodiments of the system, at least one processor is further configured to: access a second image from a cache memory storing cached images, the access of the second image from the cache memory comprising: determining at least one of the following: temporal correlation scores of a plurality of cached images, the temporal correlation scores indicating the time elapsed between capturing an individual cached image and a first image; or contextual correlation scores of a plurality of cached images, the contextual correlation scores indicating the degree of structural similarity between an individual cached image and the first image; and selecting a second image from the plurality of cached images based on the temporal correlation scores satisfying a temporal correlation threshold or the contextual correlation scores satisfying a contextual correlation threshold.

[0061] In some embodiments of the system, the system is configured to display a restored image.

[0062] In other broad aspects of this technology, a system for near real-time image enhancement of endoscopic images is provided, the system comprising at least one processor configured to: access a first image from an image stream; determine the presence of a contaminated segment in the first image; determine at least one of (i) a type of contaminant associated with the contaminated segment or (ii) the size of the contaminated segment; adjust the resolution of the first image based on at least one of (v) the type of the contaminant or (vi) the size of the contaminated segment, the adjustment resulting in the generation of a reduced-resolution image from the first image; input (ix) the reduced-resolution image to a machine learning algorithm (MLA) to generate a replacement segment of the contaminated segment; and generate a restored image based on the replacement segment.

[0063] In some embodiments of the system, at least one processor is further configured to: access a second image from a cache memory storing cached images, the access of the second image from the cache memory comprising: determining at least one of the following: temporal correlation scores of a plurality of cached images, the temporal correlation scores indicating the time elapsed between capturing an individual cached image and a first image; or contextual correlation scores of a plurality of cached images, the contextual correlation scores indicating the degree of structural similarity between an individual cached image and the first image; and selecting a second image from the plurality of cached images based on the temporal correlation scores satisfying a temporal correlation threshold or the contextual correlation scores satisfying a contextual correlation threshold.

[0064] In some embodiments of the system, the system is configured to display a restored image.

[0065] In other broad aspects of this technology, a system for near real-time image enhancement of endoscopic images is provided, the system comprising at least one processor configured to: access a first image from an image stream; determine the presence of a contaminated segment in the first image; determine at least one of (i) a type of contaminant associated with the contaminated segment or (ii) the size of the contaminated segment; adjust the frame rate associated with the image stream based on at least one of (v) the type of contaminant or (vi) the size of the contaminated segment, the adjustment resulting in a reduced image stream; input at least one extracted image from the reduced image stream to a machine learning algorithm (MLA) to generate a replacement segment of the contaminated segment; and generate a restored image based on the replacement segment.

[0066] In some embodiments of the system, at least one processor is further configured to: access a second image from a cache memory storing cached images, the access of the second image from the cache memory comprising: determining at least one of the following: temporal correlation scores of a plurality of cached images, the temporal correlation scores indicating the time elapsed between capturing an individual cached image and a first image; or contextual correlation scores of a plurality of cached images, the contextual correlation scores indicating the degree of structural similarity between an individual cached image and the first image; and selecting a second image from the plurality of cached images based on the temporal correlation scores satisfying a temporal correlation threshold or the contextual correlation scores satisfying a contextual correlation threshold.

[0067] In some embodiments of the system, the system is configured to display a restored image.

[0068] In other broad aspects of this technology, a near real-time image enhancement system for endoscopic images is provided, the system comprising at least one processor configured to: Access a first image from an image stream; determine that a contaminated segment exists in a region of interest (ROI) of the first image; determine at least one of (i) the type of contaminant associated with the contaminated segment or (ii) the size of the contaminated segment; in response to the type of contaminant or the size of the contaminated segment being below a first threshold, estimate the duration required to perform image processing operations on the contaminated segment; based on the estimated duration, selectively reduce at least one of the resolution of the first image, the frame rate associated with the image stream, and the size of the ROI; access a second image from a cache memory storing cached images; generate a prediction of a replacement segment for replacing the contaminated segment based on the second image; composite the replacement segment onto the contaminated segment to generate a composite image; selectively output at least one of the first image and the composite image to a display.

[0069] In other broad aspects of this technology, a system is provided for managing a cache memory storing multiple cached images, the system including at least one processor configured to: access a first image from an image stream; access the cache memory storing the multiple cached images; determine at least one of the following: a temporal correlation score of the multiple cached images, the temporal correlation score indicating the time elapsed between capturing an individual cached image and the first image; or a contextual correlation score of the multiple cached images, the contextual correlation score indicating the degree of structural similarity between an individual cached image and the first image; and select a second image from the multiple cached images based on the temporal correlation score satisfying a temporal correlation threshold or the contextual correlation score satisfying a contextual correlation threshold; and return the second image.

[0070] In some embodiments of the system, the at least one processor is further configured to: determine whether a first image is suitable for caching, the suitability being determined based on at least one image quality metric; and, in response to determining that the first image is suitable for caching, store the first image as one of a plurality of cached images in the cache memory.

[0071] In some embodiments of the system, at least one image quality metric includes one or more of the following: a sharpness metric determined based on at least one of edge gradient values ​​and Laplacian variance; or a haze level metric determined based on at least one of dark channel prior (DCP) values ​​and fog perceived density assessment (FADE) values; or an information richness metric determined based on entropy analysis.

[0072] In some implementations of the system, at least one processor is further configured to remove an individual cached image from the cache memory in response to at least one of the following: the temporal relevance score of the individual cached image does not exceed a temporal relevance threshold; and the contextual relevance score of the individual cached image does not exceed a contextual relevance threshold.

[0073] In some implementations of the system, at least one processor is further configured to selectively input a second image into a machine learning algorithm (MLA).

[0074] In other broad aspects of this technology, a non-transitory computer-readable medium including instructions is provided, which, when executed by at least one processor, configures at least one processor to perform the various methods described herein.

[0075] According to another aspect of the present invention, a method for improving the quality of endoscopic images is disclosed, the method comprising: receiving image data including image frames; caching a first image frame in the image frames to generate a cached image frame; capturing a second image frame in the image frames to obtain a captured image frame; determining that the captured image frame is contaminated; identifying contaminated image segments from the contaminated captured image frame; generating a prediction of a replacement image segment for replacing the contaminated image segment; and synthesizing the replacement image segment into the cached image frame to generate a synthesized image.

[0076] In at least one embodiment, the method further includes replacing the contaminated segmented image frames with a synthetic image.

[0077] In at least one embodiment, the method further includes classifying the cached image frames as last clear frames using a first AI model for segmentation.

[0078] In at least one implementation, determining that the captured image frame is contaminated is based at least in part on transmitting the captured image frame to a first AI model for segmentation.

[0079] In at least one embodiment, the method further includes segmenting contaminated image segments from occluded captured image frames.

[0080] In at least one embodiment, the method further includes inputting contaminated image segments into a second AI model for generation to preprocess the contaminated image segments for recovery.

[0081] In at least one implementation, the prediction of generating replacement image segments to replace contaminated image segments is based at least in part on the use of a second AI model for generation.

[0082] In at least one implementation, the replacement image segment is synthesized onto a cached image frame based at least in part on the use of a second AI model for generation, to generate a synthesized image using the last clear frame.

[0083] According to another aspect of the present invention, a method for improving the quality of endoscopic images is disclosed, the method comprising: receiving image data including image frames; capturing a first image frame from the image frames to obtain a captured image frame; determining that the captured image frame is contaminated; segmenting the contaminated portion of the captured image frame; and generating a restored image frame based at least in part on a last clear frame and the captured image frame.

[0084] In at least one embodiment, the method further includes replacing the contaminated capture frame with a restored image frame.

[0085] In at least one implementation, determining that the captured frame is contaminated is based at least in part on a classification by a first AI model used for segmentation, which classifies the captured frame as contaminated.

[0086] In at least one implementation, a first AI model for segmentation is used to perform segmentation of the contaminated portion of the captured frame.

[0087] In at least one embodiment, the method further includes inputting contaminated image segments into a second AI model for generation to preprocess the contaminated image segments for recovery.

[0088] In at least one embodiment, the method further includes generating a prediction of a replacement image segment to replace the contaminated image segment for generating a restored image frame.

[0089] According to another aspect of the present invention, a method for improving the quality of endoscopic images is disclosed, the method comprising: receiving image data including image frames; capturing a first image frame from the image frames to obtain a first captured image frame; determining that the first captured image frame is not contaminated; storing the first captured image frame as a last clear frame; capturing a second image frame from the image frames to obtain a second captured image frame; determining that the second captured image frame is contaminated; and generating a restored image frame based at least in part on the last clear frame and the second captured image frame.

[0090] In at least one embodiment, the method further includes replacing the contaminated second capture frame with a restored image frame.

[0091] In at least one implementation, a first AI model for segmentation is used to determine that the first captured frame is not contaminated.

[0092] In at least one embodiment, the method further includes segmenting the contaminated portion of the second captured image frame.

[0093] In at least one embodiment, the method further includes inputting contaminated image segments into a second AI model for generation to preprocess the contaminated image segments for recovery.

[0094] In at least one implementation, the generation of the restored image frame is also based at least in part on the processing of the last clear frame by a second AI model used for generation.

[0095] Other features and advantages of this application will become apparent from the following detailed description taken in conjunction with the accompanying drawings: However, it should be understood that the detailed description and specific examples, while indicating preferred embodiments of this application, are given by way of example only, as various variations and modifications within the spirit and scope of this application will become apparent to those skilled in the art based on the detailed description. Attached Figure Description

[0096] To better understand the various embodiments described herein and to more clearly show how these various embodiments can be implemented, reference will be made to the accompanying drawings by way of example, which illustrate at least one exemplary embodiment, which will now be described. The drawings are not intended to limit the scope of the teachings described herein in any way. In the drawings: Figure 1 A block diagram of an exemplary embodiment of a system for improving the quality of endoscopic images is shown; Figures 2A-2B An exemplary embodiment of the operating environment for a system for improving the quality of endoscopic images is shown; Figure 3 An exemplary embodiment of an operating room setup for minimally invasive surgery is shown; Figure 4 A flowchart illustrating an exemplary implementation of system integration in an operating room (OR) setup is provided. Figure 5 A block diagram of a data flow is shown for an exemplary implementation of signal processing in a system for improving the quality of endoscopic images; Figure 6 A block diagram illustrating the data flow between individual parts of an exemplary embodiment of a system for improving endoscopic images is shown; Figure 7A A flowchart illustrating an exemplary implementation of the decision-making process of a system for improving endoscopic images is shown; Figures 7B-7G It shows in Figure 7A An instance of an image at the processing stage during the decision-making process; Figure 8A flowchart illustrating an exemplary implementation of an image restoration process for a system used to improve endoscopic images is provided. Figure 9A A flowchart illustrating an exemplary implementation of a decision-making process for improving endoscopic images with a cache memory is shown. Figures 9B-9H It shows in Figure 9A An instance of an image at the processing stage during the decision-making process; Figure 10 A flowchart illustrating an exemplary implementation of a method for improving image quality is shown; Figures 11A-11B The first example of the actual surgical field of view and the improved field of view after image quality improvement is shown; Figures 12A-12B A second example of the actual surgical field of view and the improved field of view after image quality improvement is shown; Figure 13 An example of an architecture for image segmentation and restoration is shown; and Figure 14 A flowchart illustrating an exemplary implementation of an image enhancement process 2000 for improving visual clarity via endoscopy is provided.

[0097] Other aspects and features of the exemplary embodiments described herein will become apparent from the following description taken in conjunction with the accompanying drawings. Detailed Implementation

[0098] The following describes examples of various embodiments based on the teachings herein, providing at least one embodiment of the claimed subject matter. None of the embodiments described herein are limiting of any claimed subject matter. The claimed subject matter is not limited to an apparatus, system, or method having all the features of any of the devices, systems, or methods described below, or limited to features common to multiple or all of the devices, systems, or methods described herein. There may be apparatus, systems, or methods that are embodiments of any claimed subject matter described herein. Any subject matter described herein but not claimed in this document may be the subject matter of another protective document, for example, a continuing patent application, and the applicant, inventor, or owner does not intend to waive, deny, or publicly disclose any such subject matter through this disclosure.

[0099] It should be understood that, for the sake of simplicity and clarity, reference numerals may be repeated between figures where deemed appropriate to indicate corresponding or similar elements. Furthermore, numerous specific details are set forth to provide a thorough understanding of the embodiments described herein. However, those skilled in the art will understand that the embodiments described herein can be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail to avoid obscuring the embodiments described herein. Moreover, this specification should not be construed as limiting the scope of the embodiments described herein.

[0100] It should also be noted that the terms “coupled” or “coupling” as used herein can have several different meanings depending on the context in which they are used. For example, the terms “coupled” or “coupling” can have mechanical or electrical meanings. For instance, as used herein, the terms “coupled” or “coupling” can indicate that two elements or devices are directly connected to each other or connected to each other through one or more intermediate elements or devices, depending on the specific context.

[0101] It should be noted that, as used herein, the term “and / or” is intended to mean inclusive-or. That is, for example, “X and / or Y” is intended to mean X or Y or both. As another example, “X, Y and / or Z” is intended to mean X or Y or Z or any combination thereof.

[0102] It should also be noted that, as used herein, the terms “at least one of X or Y” and “at least one of X and Y” are intended to mean either X or Y or both X and Y.

[0103] It should be noted that degree terms such as “basically,” “about,” and “approximately” are used in this document to indicate a reasonable amount of deviation from the modified term such that the final result is not significantly altered. These degree terms may also be interpreted as including deviations from the modified term, such as deviations of 1%, 2%, 5%, or 10%, if such deviations do not negate the meaning of the term they modify.

[0104] Furthermore, the numerical ranges expressed as endpoints in this document include all numbers and fractions within that range (e.g., 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.90, 4, and 5). It should also be understood that all numbers and fractions are considered to be modified by the term “about,” which means a maximum variation of a certain amount in the referenced number, provided that the final result does not change significantly, such as, for example, 1%, 2%, 5%, or 10%.

[0105] It should also be noted that the term "window," used in conjunction with the operations describing any system or method described herein, is intended to be understood as describing a user interface used for initialization, configuration, or other user operations. A window may, for example, display an image exported from an imaging device such as a camera or medical imaging device, or even a picture-in-picture.

[0106] Exemplary implementations of the devices, systems, or methods described in accordance with the teachings herein can be implemented as a combination of hardware and software. For example, the implementations described herein can be implemented, at least in part, using one or more computer programs executing on one or more programmable devices including at least one processing element and at least one storage element (i.e., at least one volatile storage element and at least one non-volatile storage element). The hardware may include input devices, including at least one of a touchscreen, keyboard, mouse, button, key, slider, etc.; and one or more of a display, printer, etc., depending on the hardware implementation.

[0107] It should also be noted that some elements may exist for implementing at least a portion of the embodiments described herein, and these elements can be implemented via software written in a high-level procedural language such as object-oriented programming. The program code can be in C... ++ It may be written in C#, JavaScript, Python, or any other suitable programming language, and may include modules or classes, as known to those skilled in object-oriented programming. Alternatively or in addition, some of these elements, implemented via software, may be written in assembly language, machine language, or firmware as needed. In any case, the language may be a compiled language or an interpreted language.

[0108] At least some of these software programs can be stored on a computer-readable medium, such as, but not limited to, read-only memory (ROM), a hard disk, an optical disk, a universal serial bus (USB) flash drive, etc., which can be read by a device having at least one processor, an operating system, and associated hardware and software necessary to implement the functionality of at least one of the embodiments described herein. When read by the device, the software program code configures the device to operate in a new, specific, and predefined manner (e.g., as a purpose-built computer) to perform at least one of the methods described herein.

[0109] At least some of the programs associated with the devices, systems, and methods of the embodiments described herein can be distributed in a computer program product comprising a computer-readable medium carrying computer-usable instructions, such as program code, for one or more processing units. The medium can be provided in various forms, including non-transitory forms such as, but not limited to, one or more floppy disks, optical disks, magnetic tapes, chips, and magnetic and electronic storage. In alternative embodiments, the medium can be inherently transient, such as, but not limited to, wired transmission, satellite transmission, Internet transmission (e.g., download), media, digital and analog signals, etc. The computer-usable instructions can also be in various formats, including compiled and uncompiled code.

[0110] In at least some of the embodiments described herein, artificial intelligence (AI) is used to recover endoscopic video signals in real time, while integrating with equipment already available in hospitals.

[0111] At least some of the implementation methods described herein have advantages over conventional methods because they improve the quality of minimally invasive surgery by providing AI-based approaches to address the challenges of contaminated camera lenses. To this end, AI-driven software can improve vision in the presence of camera lens contamination, thereby ensuring enhanced visibility for the surgeon during the procedure.

[0112] AI-driven software is advantageous because it addresses the unmet need for clear imaging in minimally invasive surgery. The lack of automation in traditional cleaning procedures leads to frustration, increased errors, and worsened patient outcomes. AI-driven software can improve the cleaning process by reducing the number of cleaning events through image enhancement.

[0113] Using AI-based image processing, images can be proactively enhanced in real time. Cleaning events can be delayed or eliminated, thereby minimizing the number of cleaning events and the time lost in cleaning.

[0114] Image enhancement can allow for fewer interruptions during surgery.

[0115] At least some of the embodiments described herein have the benefit of monitoring and improving the quality of current endoscopic images. In one respect, monitoring image quality allows surgeons to better determine when to clean the endoscope. In another respect, improved image quality can reduce the frequency of endoscope cleaning while providing adequate image quality throughout the procedure.

[0116] At least some of the embodiments described herein automatically enhance the visualization of any camera display in real time during surgery to reduce damage caused by contaminants, thereby reducing the number of cleaning events during the procedure. At least some of these embodiments are systems or methods residing on an edge platform connected between the endoscope processor and the display. These systems or methods determine when a cleaning event is needed, for example, by providing an API for integration with third-party cleaning systems. These systems or methods also use computer vision algorithms to reconstruct images of the contaminated camera field of view in real time.

[0117] At least some of the implementations described herein provide technical benefits, such as being non-disruptive, versatile, or general-purpose.

[0118] These implementations are non-disruptive because the processing is done automatically and in real time, eliminating the need for intervention from the surgical team.

[0119] These implementations are versatile because they address any type of lens contamination, such as fog, smoke, blood, fluids, etc.

[0120] These implementations are general because the systems and methods operate in any endoscopic procedure. Alternatively or additionally, the systems and methods are platform-independent, making them compatible with any similar medical imaging platform.

[0121] In at least some of the embodiments described herein, the system or method does not handle completely occluded camera fields of view; however, this is not its purpose. In this case, the goal is to reduce the number of necessary cleaning events by enhancing image quality in real time.

[0122] In at least some of the embodiments described herein, the systems or methods provide a systematic approach to improving visual conditions during surgery. They offer a technical approach to address all types of contamination. They can mitigate unstable reconstructions, also known as “illusions” (where the model output would not exist in real life), for example, by restoring the contaminated portions of an image while keeping the rest of the image unchanged.

[0123] Based on the teachings of this article, various implementations of systems for improving the quality of endoscopic images are provided, as well as methods and computer products for use therewith.

[0124] First refer to Figure 1 The diagram illustrates a block diagram of an exemplary embodiment of a system 100 for improving the quality of endoscopic images. System 100 includes at least one server 120. Server 120 may communicate, for example wirelessly or via the Internet, with one or more user devices (not shown). When used in this way, system 100 may also be referred to as a machine learning system.

[0125] User equipment can be a computing device operated by a user. User equipment can be, for example, a smartphone, smartwatch, tablet computer, laptop computer, virtual reality (VR) device, or augmented reality (AR) device. User equipment can also be, for example, a combination of computing devices operating together, such as a smartphone and sensors. User equipment can also be, for example, a device operated by a user in other ways, such as a drone, robot, or remotely controlled device; in this case, the user equipment can be operated by the user, for example, through a personal computing device (such as a smartphone). User equipment can be configured to run applications (e.g., mobile applications) that communicate with other parts of system 100 (such as server 120).

[0126] Server 120 may operate on a single computer, which includes at least one processor unit 124, a display 126, a user interface 128, an interface unit 130, input / output (I / O) hardware 132, a network unit 134, a power unit 136, and a memory unit (also referred to as a "data storage device") 138. In other embodiments, server 120 may have more or fewer components, but generally functions in a similar manner. For example, server 120 may be implemented using more than one computing device.

[0127] At least one processor unit 124 may include a standard processor, such as, for example, an Intel Xeon processor. Alternatively, multiple processors may be used by at least one processor unit 124, and these processors may operate in parallel and perform certain functions. Display 126 may be, but is not limited to, a computer monitor or a liquid crystal display (LCD) such as for a tablet device. User interface 128 may be an application programming interface (API) or a web-based application accessible via network unit 134. Network unit 134 may be a standard network adapter such as an Ethernet or 802.11x adapter.

[0128] At least one processor unit 124 can execute a prediction engine 152, which provides predictions using a machine learning model 146 stored in a memory unit 138. The prediction engine 152 can construct prediction algorithms using machine learning. Training data may include, for example, image data, video data, audio data, and text.

[0129] At least one processor unit 124 may also execute a GUI engine 154 for generating various graphical user interfaces (GUIs). The GUI engine 154 provides data according to the specific layout of each user interface and also receives data input or control input from the user. The GUI then uses the input from the user to change the data displayed on the current user interface or to change the operation of the server 120, which may include displaying different user interfaces.

[0130] Memory unit 138 may store program instructions for operating system 140, program code 142 for other applications, input module 144, multiple machine learning models 146, output module 148, and database 150. Machine learning models 146 may include, but are not limited to, image recognition and classification algorithms based on deep learning models and other methods. Database 150 may be, for example, a local database, an external database, a cloud database, multiple databases, or a combination thereof.

[0131] In at least one implementation, the machine learning model 146 includes a combination of a convolutional neural network (CNN) and a recurrent neural network (RNN). A CNN can be designed to recognize images or patterns. A CNN can perform convolution operations, which can be used, for example, to classify regions of an image and view the edges of objects identified within those regions. A recurrent neural network (RNN) can be used to recognize sequences such as text, speech, and temporal evolution; therefore, an RNN can be applied to a data sequence to predict what will happen next. Thus, a CNN can be used to read what is happening on a given image at a given time, while an RNN can be used to provide informational messages.

[0132] Program 142 includes program code that, when executed, configures at least one processor unit 124 to operate in a specific manner to implement various functions and tools of system 100.

[0133] Figure 2A A schematic diagram of an exemplary embodiment of a first operating environment 200 for a system 210 for improving image quality is shown. Figure 2BA schematic diagram of an exemplary implementation of a second operating environment 250 for system 210 is shown. Some or all of system 100 may be used to implement some or all of system 210. Some or all of endoscope processor 220 may be used to implement some or all of system 100. System 210 includes a video processor connected between endoscope processor 220 and display 230 using, for example, a standard connection. System 210 embeds a software module for real-time image quality enhancement. The software module may use AI trained on synthetic data to enhance image sharpness and remove contaminants when the field of view is partially obstructed. The software module may also use AI to determine when a cleaning event is desired, for example, when the lens is completely obstructed. The software module may provide an API for integration with third-party cleaning systems.

[0134] Alternatively, system 210 may be embedded in endoscope processor 220. In this case, the functionality of system 210 is implemented by endoscope processor 220, wherein some or all of system 100 is embedded in endoscope processor to provide endoscope processor 220 with the hardware and / or software required to perform the operations, steps and / or actions described herein for improving images.

[0135] In some implementations, system 210 may include Internet Protocol Audio / Video Transport (AVoIP) capabilities. System 210 may integrate with existing AVoIP platforms. System 210 may be configured to communicate with (or work with) an AVoIP platform. For example, system 210 may provide image enhancement for endoscopy, digital operating room (OR) applications, and / or catheterization labs. Also, for example, system 210 may support various types of medical procedures, which may include technologies such as 4K video routing with IP-based fiber optic video switching.

[0136] Advantageously, system 210 can provide a real-time AI-enhanced surgical field of view with seamless endoscopic system integration. Advantageously, system 210 can reduce endoscope removal, optionally by up to 40%. The inventors observed a 40% reduction in endoscope removal after performing internal analysis of retrospective videos of sinus surgeries, wherein, based on an assessment of image blur, it is estimated that 40% of the endoscope removal could have been avoided using the methods described herein for improving image quality.

[0137] When an endoscope is inserted into the gastrointestinal tract (or other body part), the endoscope processor 220 can receive input from a camera built into the endoscope. The endoscope processor 220 can receive image signals from the endoscope, which can be processed for display or otherwise output. The endoscope processor 220 can control the processing of image signals from the endoscope and thus functions as an image processor.

[0138] Endoscope processor 220 can receive input from the endoscope, such as buttons that can be pressed by a user to send input signals to control the endoscope or a camera built into the endoscope. These buttons can be user-programmable buttons to send input signals to endoscope processor 220 to control the endoscope video stream or image stream.

[0139] The endoscope processor 220 can be a microcontroller, microprocessor, or microcomputer, and can employ a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC). The endoscope processor 220 can be, for example, an NVIDIA... TM Jetson TM Microcomputers, which may include software accelerators (e.g., Raspberry Pi) TM TensorFlow TM wait).

[0140] Now for reference Figure 2B System 210 and third-party cleaning system 260 provide AI-enhanced visualization for minor damage (e.g., to prevent 40% of the scope from being removed). System 210 and third-party cleaning system 260 provide a contamination detection API for third-party integration and automatically clean severe damage, which can advantageously allow for non-disruptive workflows.

[0141] System 210 can advantageously enhance endoscopic video signals in real time. Some of the challenges addressed by system 210 and its functionality (e.g., with reference to the data flow and methods described herein) include: (a) processing with very low latency; and (b) constructing AI models that fit embedded constraints.

[0142] Figure 3 An exemplary embodiment of an operating room setup 300 for minimally invasive surgery is illustrated. The operating room setup 300 includes an operating table 310, surgical instruments 320, and an endoscope tower 330. The endoscope tower 330 has shelves on which various hardware components, such as an endoscope processor 220, a monitor 230, and additional consoles if needed, are placed.

[0143] Now for reference Figure 4Endoscope 410, endoscope processor 420, and display 440 are connected via common video interface standards such as Digital Vision Interface (DVI), Serial Digital Interface (SDI), or (High Definition Multimedia Interface) HDMI. Supported video formats can range from, for example, 1080i or 1080p to 4K resolution. VOPE 430 (also referred to as System 430) connects between endoscope processor 420 and display 440, processing the signal in real time to restore the video feed in the presence of contamination. Some or all of System 430 can be implemented using some or all of System 210 and / or some or all of System 100. System 430 can be considered a “middleware” or “bridge,” as referred to in the video industry.

[0144] Endoscope 410 may be an endoscope adapted for insertion into a patient's body. In alternative embodiments, endoscope 410 may be replaced by another imaging device and / or sensor for other medical applications and / or imaging modalities. These imaging modalities may be, for example, arthroscopy (e.g., for orthopedics), bronchoscopy (e.g., for pulmonology), colposcopy (e.g., for obstetrics and gynecology), cystography (e.g., for urology), CT scans (e.g., for cardiology, ENT, obstetrics and gynecology, pulmonology), cystoscopy (e.g., for urology), laparoscopy (e.g., for general surgery, obstetrics and gynecology), laryngoscopy (e.g., for ENT), MRI (e.g., for ENT, obstetrics and gynecology, pulmonology), and X-rays (e.g., for obstetrics and gynecology, pulmonology).

[0145] Endoscope processor 420 may be the same as endoscope processor 220 described herein. In alternative embodiments, when endoscope 410 is used for other medical applications and / or as a replacement for other imaging devices and / or sensors in other imaging modalities, endoscope processor 420 may be replaced by a medical imaging processor suitable for the corresponding imaging modality. In this case, endoscope processor 420 may be more generally referred to as medical imaging processor 420.

[0146] System 430 can be a medical-grade console, which includes a capture card configured to acquire and forward video signals and a powerful graphics processing unit (GPU) for running AI models.

[0147] System 430 can operate as an off-the-shelf AI platform. For example, System 430 can serve as an AI solution comprising pre-built software applications designed for specific tasks of image quality improvement, either independently or integrated into other general or broad-ranging tasks. As an off-the-shelf AI platform, it can offer the benefit of rapid deployment. As an example, System 430 can be implemented using a mini-tower or a standalone medical computer such as the USM-500. Some benefits may include its autonomous development, suitability for OR, and / or compliance with IEC 60601. For example, software development kits (SDKs) such as the NVIDIA Holoscan Clara can facilitate the integration of System 430 into off-the-shelf platforms such as the Advantech USM-500. Such SDKs are designed to facilitate the real-time integration of AI applications into sources such as endoscopic video signals.

[0148] System 430 can operate as an original equipment manufacturer (OEM) AI platform. For example, System 430 can be used as an AI solution that includes software applications, which can be implemented by the hardware and / or software manufacturer for the purpose of bundling with existing products. As an example, System 430 can be implemented using an AI-based endoscope module such as Medtronic GIGenius. One advantage may include its scalability, requiring no additional technology customization.

[0149] Figure 5 A block diagram illustrating an exemplary implementation of a data flow 500 for signal processing in a system for image improvement is provided. The data flow 500 shown is between a console 510 and input signals 550 and 560. The console 510 includes a capture card 520 that interfaces with console software 530, which in turn utilizes at least one processor 540. The input signal 550 is first acquired and decoded by the capture card 520 and sent to the console software 530, which then sends data representing the input signal 550 to the at least one processor 540 for processing by an AI model. A prediction is then sent back to the console software 530, which synthesizes the prediction with the initial result and sends the result back to the capture card 520 to be encoded as a video stream sent to a monitor as the output signal 560. Some or all of the data flow 500 can be implemented using some or all of the systems 430, 210, or 100.

[0150] Capture card 520 can be a video capture card that supports one or more video modes, such as HDMI, DisplayPort, single-link or dual-link DVI, 3G-SDI, HD-SDI, RGB, component YPbPr, composite video, and / or S-Video. For example, capture card 520 may include DisplayPort 1.2, SDI, and HDMI inputs, with 4K capture up to 60 frames per second, 10-bit color transmission, and HDCP compliance.

[0151] The console software 530 may include image analysis algorithms, such as object detection. Object detection may be based on YOLOv4, which uses convolutional neural networks (CNNs) to perform certain functions. The YOLOv4 object detection algorithm may be advantageous because it allows for image analysis at a faster rate. The YOLOv4 object detection algorithm can be implemented, for example, by a microcomputer with a software accelerator. For example, U-Net can be used for object segmentation. Object detection can identify objects and their locations using bounding boxes, while segmentation can depict object boundaries at the pixel level, distinguish different instances (instance segmentation) or assign classification labels to pixels (semantic segmentation).

[0152] At least one processor 540 may be a GPU optimized for machine learning, such as an NVIDIA GPU (e.g., with NVIDIA CUDA cores) or on the cloud using AWS GPUs. At least one processor 540 may be combined with or configured to operate with a suitable CPU (e.g., NVIDIA Camel ARM). In this case, at least one processor 540 may operate in combination with a CPU to run one or more of program 142, machine learning model 146, and prediction engine 152.

[0153] DataStream 500 advantageously addresses the latency issue in signal processing, enabling its real-time use. A typical goal is to maintain latency within 100 to 300 milliseconds at the "glass-to-glass" level, depending on the task—that is, from the lens's glass to the display's glass. Above these thresholds, users may begin to notice a delay between their actions and what they see on the display. This reduction in latency can be achieved by building and / or using AI models that conform to embedded constraints.

[0154] Figure 6A block diagram of a data flow 600 between individual components of an exemplary implementation of a system for improving endoscopic images is shown. Data flow 600 can be implemented using some or all of system 430, some or all of system 210, and / or some or all of system 100. Data flow 600 includes a first processor 610 (which may be, for example, a CPU) communicating with a PCI express 620 (peripheral component interconnect). PCI express 620 communicates with a third-party device 630 and a second processor 640 (which may be, for example, a GPU). The third-party device 630 and the second processor 640 communicate using Remote Direct Memory Access (RDMA) 650 (e.g., GPUDirect RDMA). To minimize latency, data flow 600 benefits from RDMA, which allows multiple hardware components on the same PCI bus to share the same memory space to prevent data from being transferred by the first processor 610 (also known as a PCI bottleneck), such as... Figure 6 As shown. In this way, the data remains in the same space and is not subject to the latency introduced by moving data to different hardware components. Low AI model latency is also advantageous in the timely recovery of contaminated images. Considering the latency of the endoscope system is approximately 50 milliseconds, and also taking into account I / O latency, an AI model latency of approximately 30 milliseconds maintains a real-time experience. Data stream 600 can be considered to provide “zero-copy communication” enabled by RDMA, where data does not move within the hardware to minimize latency.

[0155] Figure 7A A flowchart illustrating an exemplary implementation of a decision process 700 for improving endoscopic images is provided. Process 700 may be implemented using some or all of system 430, some or all of system 210, and / or some or all of system 100. Process 700 corrects (or enhances) the endoscopic video signal when it is obstructed by any kind of contaminant such as blood, fog, blur, fluid, etc.

[0156] In process 700, the system decodes the video signal (which may, for example, come from a DVI / SDI / HDMI interface) and converts it into a separate frame at the same rate (e.g., 30 FPS) to be processed by the system at capture frame 710 (see [link to capture frame 710]). Figure 7B (Image 712 in the image). As an example, surgical instrument 320 is used to visualize minimally invasive cholecystectomy, but the field of vision is obscured by blurring caused by the temperature difference between the operating room and the patient (e.g., fog, condensation, sinus fluid, residue from contacted internal organ structures).

[0157] The captured frames are converted to RGB 8-bit as instances and are first sent to AI model #1 for segmentation 720 (see...). Figure 7CImage 722 in the image determines whether the video feed is contaminated or not. 730 (see image 722) Figure 7D Image 732 in the image also segments the occluded parts of the frame.

[0158] As an example, AI model #1 segmentation 720 is implemented using U-Net or any convolutional neural network (CNN) capable of segmenting images and providing a classification of whether an image is contaminated. In the case of a contaminated cholecystectomy frame, the model only segments the contaminated (e.g., blurred) regions of the image. If AI model #1 segmentation 720 does not detect any contamination, the frame is forwarded to the display monitor as is via forwarding frame 760. If contamination is detected, the segmented regions are sent to AI model #2 generation 740, which is responsible for recovering the contaminated portions (see [link to relevant documentation]). Figure 7E Image 742 in the image (in this case). In other words, the system can input the contaminated image segment into AI model #2 to generate 740, so as to preprocess the contaminated image segment for recovery.

[0159] As an example, AI model #2 generates 740 using a Generative Adversarial Network (GAN) or any model in the generative model family (e.g., designed to restore, deblur, or denoise images). This model is trained by feeding it pairs of images: an uncontaminated image and an image of the same image after it has been filtered or artificially contaminated by a generative model trained to contaminate images. In the instance of restoring a blurred image, a Gaussian blur filter can be used to train the model on a clear image and its synthetic contaminated equivalent. The image restoration model can include tens to hundreds of convolutional layers for feature extraction, followed by pooling layers for spatial downsampling, then convolutional layers for feature refinement, possibly enhanced with skip connections to preserve finer details, and finally upsampling layers to reconstruct the image at its original resolution, often integrated with techniques such as residual learning or attention mechanisms for performance enhancement and fine-grained restoration.

[0160] The predictions generated by AI model #2 at output 740 are synthesized into the initial frame 750 before being forwarded to the display monitor and becoming forwarded frame 760 (see [link]). Figure 7F On image 750 (see image 750) Figure 7G (Image 760 in the image). An example of synthesis is achieved by replacing the pixel values ​​of contaminated portions identified by AI model #1 segmentation 720 of the original image with predictions generated by AI model #2 740. In the case of a contaminated cholecystectomy frame, one or more blurred region pixel values ​​are replaced with clear pixel values ​​generated by AI model #2 740.

[0161] This same compositing can also be achieved using alpha blending, which uses different levels of transparency to indicate the areas being corrected. Alpha blending is a technique used in computer graphics and image processing to combine two images or layers by blending pixel values ​​based on a specified transparency value called the alpha channel. The alpha channel represents the transparency or opacity of each pixel in an image. In the context of alpha blending, each pixel has four components: red, green, blue, and alpha (RGBA). The alpha channel determines how much the pixel's color contributes to the final blended result. Alpha values ​​typically range from 0 (completely transparent) to 1 (completely opaque).

[0162] In other implementations, additional or alternative markers may be implemented on the restored area or elsewhere on the display to achieve the same purpose. Process 700 advantageously enables the image restoration mechanism only when necessary to mitigate inconsistencies such as artifacts when the image is actually sharp.

[0163] Image segmentation can be achieved using algorithms that divide an image into regions. One example of such an algorithm involves dividing the image into a grid of rectangular regions. For instance, an image with a resolution of 640x480 can be divided into 100 regions of size 64x48 or 3072 regions of size 10x10.

[0164] Another example of segmentation algorithms involves dividing an image into a set of segments (or contours) where each pixel in a region is similar with respect to some feature or computed property, such as color, intensity, or texture.

[0165] Alternatively or additionally, image segmentation can be performed using AI trained for segmentation. For example, a training set of labeled images can be fed into a neural network that learns how to segment a particular image from the labeled images. This supervised learning can then replace or even improve existing algorithms or human-based (expert, such as surgeons) medical image segmentation.

[0166] After an image is segmented (e.g., according to any one or more techniques described herein), contamination within the image can be determined using algorithms for determining whether an image is contaminated or whether one or more regions of the image are contaminated. One example of such an algorithm involves determining to assign a blur score to each region and calculating a contamination score using a formula based on the blur score. This formula can be, for example, a count, sum, average, or weighted average of all or some blur scores. The blur score can be calculated, for example, using rule-based methods, such as deviations from normal values ​​assigned to the segment category for each region. The blur score can also be calculated, for example, using the Laplacian variance of the image. When using Laplacian, the algorithm can handle blur levels of 50 calculated from the Laplacian variance over contaminated regions of the image.

[0167] Alternatively or additionally, the determination of contamination within an image can be achieved using AI trained to classify an image or a region within an image as contaminated. For example, a training set of labeled images can be fed into a neural network that learns from the labeled images how to classify a particular image or a region within an image as contaminated. This supervised learning can then replace or even improve existing algorithms or human-based (expert, such as surgeons) medical image contamination determination.

[0168] Figure 8 A flowchart illustrating an exemplary implementation of an image restoration process 800 for improving endoscopic images is provided. Process 800 can be implemented using some or all of system 430, some or all of system 210, and / or some or all of system 100. To mitigate inconsistencies, the process reconstructs contaminated portions of the image. In this process, an initial contaminated frame is processed by AI model #1 segmentation 810 to identify and segment the contaminated portions, which becomes the output of AI model #1 segmentation 810. This segmented region is then sent to AI model #2 generation 820, which is responsible for restoring the image region; the initial frame is the input to provide more context to the model, thereby improving restoration accuracy. The output of the second AI model is the restored region, which is then superimposed on the initial image, step 830. Thus, process 800 leaves the clear portions of the image unprocessed to prevent unnecessary restoration.

[0169] Image analysis training algorithms can be used to train AI model #1 for segmentation (810) and / or AI model #2 for generation (820). This algorithm can employ, for example, an encoder that compresses the input into feature vectors (e.g., using a CNN). The algorithm then feeds these feature vectors into a decoder that reconstructs a high-resolution image from the low-resolution feature vectors. The algorithm can then map the feature vectors to a distribution for object classification. This algorithm can employ supervised, semi-supervised, or unsupervised training.

[0170] AI model #1 segmentation 810 and / or AI model #2 generation 820 can use a CNN with convolutional blocks, deconvolutional blocks, and classifier blocks. Convolutional blocks generate features and can consist of convolutional layers, activation layers, and pooling layers. Deconvolutional blocks add information to the features to allow for image reconstruction and can consist of convolutional layers, transposed convolutional layers, and activation layers. Classifier blocks generate the categories of objects in the analyzed image and can consist of convolutional layers, activation layers, and fully connected layers.

[0171] Figure 9A A flowchart illustrating an exemplary implementation of a decision process 900 for improving endoscopic images with a cached memory is provided. Process 900 can be implemented using some or all of system 430, some or all of system 210, and / or some or all of system 100. While process 900 can minimize inconsistencies and increase spatial awareness of the model, a system for improving images can also, for example, store previous frames in memory before the video is contaminated to increase temporal awareness of the output.

[0172] When the endoscopic video signal is obstructed by any kind of contaminant such as blood, fog, blur, fluid, etc., process 900 recovers the endoscopic video signal. In process 900, the system decodes the video signal (which may come from, for example, a DVI / SDI / HDMI interface) and converts it into individual frames at the same rate (e.g., 30 FPS) for processing by the system at capture frame 910 (see [link to capture frame 910]). Figure 9B Image 912 in the image). The captured frame is first sent to AI model #1 segmentation 920 (see image 912). Figure 9C Image 922 in the image determines whether the video feed is contaminated (see image 930). Figure 9D Image 932 in the image is processed, and contaminated (e.g., occluded) portions of the frame are also segmented. If AI model #1 segmentation 920 does not detect any contamination, the frame is stored in memory as the last clear frame in memory 960 (see image 932). Figure 9G (See image 962). An example of this implementation is random access in any memory storage large enough to store the frame. The frame is then forwarded to the display monitor via frame forwarding 970 (see image 962). Figure 9H Image 972 in the image. If contamination is detected, the segmented region is sent to AI model #2 responsible for recovering the contaminated part to generate 940 (see image 972). Figure 9E Image 942 in the image (see image 942). In other words, the system can input contaminated image segments into AI model #2 generation 940 to preprocess the contaminated image segments for recovery. The predictions output by the same model are synthesized into the initial frame 950 before being forwarded to the display monitor and becoming forwarded frame 970 (see image 942). Figure 9F Image 952 in the reference procedure 700). To reuse the cholecystectomy example described in the reference procedure 700, when a frame is classified as clear by AI model #1 segmentation 920, it is stored in memory 960 as the "last clear frame". When a frame is contaminated (e.g., by blurring), the "last clear frame" is retrieved and provided to AI model #2 generation 940 for more faithful reconstruction. Memory 960 can also store multiple clear frames in a first-in-first-out (FIFO) data structure and retrieve the best match based on structural similarity (SSIM) or any method that calculates the similarity between images. After a timeout, the earliest stored frame in memory 960 is released for space.

[0173] While process 700 uses a simplified architecture describing how frames are processed to capture the data stream, process 900 adds an additional step: storing frames that are not detected as clear frames in memory 960. Then, when the video feed is occluded, process 900 retrieves the nearest and similar frames from the buffer to feed the recovery model. This allows the model to provide a more realistic recovery based on what the system has previously seen.

[0174] In at least some embodiments of the system for improving image quality described herein, the system is integrated with existing equipment in an operating room. The system also describes mitigation strategies to reduce unstable reconstruction by: (1) recovering frames only when they are detected as “contaminated”; (2) recovering only the contaminated portions of frames; and (3) caching previously clear images to enable reconstruction based on what the system has previously seen. These mitigation strategies are advantageous because they can improve the accuracy of the video signal and / or enhance the accuracy of the recovered image.

[0175] In at least one embodiment of process 900, process 900 is performed using the following steps: (a) receiving image data including image frames; (b) capturing a first image frame from the image frames, resulting in a first captured image frame; (c) determining that the first captured frame is not contaminated; (d) storing the first captured image frame as a last clear frame; (e) capturing a second image frame from the image frames, resulting in a second captured image frame; (f) determining that the second captured image frame is contaminated; and (g) generating a recovered image frame based at least in part on the last clear frame and the second captured image frame. It should be understood that process 900 may be performed in different orders as described above, with fewer steps (e.g., omitting one of the steps), or with more steps (e.g., having additional or interventional steps).

[0176] Figure 10A flowchart illustrating an exemplary implementation of a method 1000 for improving image quality is provided. Method 1000 can be implemented using some or all of system 430, some or all of system 210, and / or some or all of system 100. While method 1000 can minimize inconsistencies and increase spatial awareness of the model, a system for improving image quality can also, for example, store previous frames in memory before the video is contaminated to increase temporal awareness of the output.

[0177] At 1010, the system receives image data including image frames. Image frames can be, for example, a sequence of image frames from a video feed. Image frames can also be the result of a video signal being converted into individual frames.

[0178] At position 1020, the system caches the first image frame in the image frame set, thus generating a cached image frame. The system can cache multiple image frames. The system can determine whether to cache multiple image frames, for example, based on periodicity or on differences from previously cached image frames (e.g., predetermined differences or differences based on anatomical changes). The system can use AI model #1 (for segmentation) to classify the cached image frames as "last clear frames".

[0179] At 1030, the system captures the second image frame in the image frame sequence, thus obtaining the captured image frame. The system can determine whether to capture all image frames that form part of a video feed, in which case the action of capturing an image frame represents a specific image capture in a series of image captures.

[0180] At position 1040, the system determines that the captured image frame is tainted. This determination can be made by transmitting the captured image frame to AI model #1 (for segmentation). AI model #1 can then segment the occluded portion of the captured image frame.

[0181] At position 1050, the system identifies contaminated image segments from the contaminated captured image frames. These contaminated image segments can be sent to AI model #2 (for generation) responsible for recovering the contaminated portions. In other words, the system can input contaminated image segments into AI model #2 to preprocess them for recovery.

[0182] At position 1060, the system generates a prediction of a replacement image segment for replacing the contaminated image segment. The system can use AI model #2 to generate the prediction.

[0183] At position 1070, the system composites the replaced image segment into a cached image frame, thereby generating a composite image. The cached image frame can be, for example, the "last clear frame". The system can use AI model #2 to create a more faithful restoration using the "last clear frame".

[0184] At position 1080, the system replaces the corrupted captured image frame with a synthetic image.

[0185] Although method 1000 has been described in a specific order with specific steps, it should be understood that method 1000 may be performed in different orders as described above, with fewer steps (e.g., omitting one of the steps), or with more steps (e.g., with additional or interventional steps). For example, method 1000 may end after step 1070, where the system generates a synthetic image (a) without continuing to step 1080 or (b) continues with other post-processing steps.

[0186] Figure 11A-11B This illustrates the actual surgical field of view and the improved field of view after image quality enhancement, in a first example. Figure 11A In the image, actual field of view 1100 shows the actual field of view from the endoscope before image quality improvement. Figure 11B The improved field of view 1150 illustrates the improved field of view after image quality improvement. This example demonstrates how the improved field of view 1150 provides an improvement relative to the actual field of view 1100 as a result of any of the methods described herein (such as method 700, method 900, or method 1000). Advantageously, the improvement can be performed, for example, in real time, to assist during surgery. Furthermore, it is advantageous that the inventors have observed a reduction in scope removal of up to 40% resulting from this image quality improvement.

[0187] Figure 12A-12B A second example is shown, illustrating the actual surgical field of view and the improved field of view after image quality enhancement. Figure 12A In the image, actual field of view 1200 shows the actual field of view from the endoscope before image quality improvement. Figure 12B In the illustration, improved field of view 1250 shows the improved field of view after image quality improvement. This example illustrates how improved field of view 1250 can provide an improvement relative to the actual field of view 1200 as a result of any of the methods described herein (such as method 700, method 900, or method 1000). Advantageously, the improvement can be performed, for example, in real time, to assist during surgery.

[0188] Figure 13An example of architecture 1300 for image segmentation and restoration is shown. In architecture 1300, input 1310 (e.g., an RGB image) is fed into a convolutional encoder-decoder. The convolutional encoder-decoder includes: convolution and batch normalization plus ReLU 1302; pooling 1304; upsampling 1306; and softmax 1308. The convolutional encoder-decoder then provides output 1320 with the desired segmentation. This architecture 1300 can be a SegNet architecture. Therefore, there are no fully connected layers, and it ends up being just convolution. In particular, the decoder can upsample the input using the pooling index passed from its encoder to produce one or more sparse feature maps. These feature maps can then be convolved with a trainable filter bank to densify them. The final decoder output feature map can then be fed into a softmax classifier for pixel-wise classification.

[0189] Figure 14 A flowchart illustrating an exemplary implementation of an image enhancement process 2000 for improving visual clarity via endoscopy is provided. Process 2000 may be implemented using some or all of system 430, some or all of system 210, or some or all of system 100.

[0190] Figure 14 A flowchart illustrating an exemplary implementation of an image enhancement process 2000 for improving visual clarity via endoscopy is provided. Process 2000 may be implemented using some or all of system 430, some or all of system 210, or some or all of system 100.

[0191] Process 2000 begins by capturing image frames from the endoscope video feed. In process 2000, the system decodes the video signal (which may come from, for example, a DVI / SDI / HDMI interface) and converts it into individual frames at the same rate (e.g., 30 FPS) for processing by the system at step 2006.

[0192] After capturing the frame, the system performs contamination detection and segmentation in step 2001. Here, the system assesses the presence of any contaminants in the frame, such as fog, smoke, blood, fat, or blur. These contaminants can originate from a variety of sources. For example, during cauterization during a surgical procedure, smoke may be generated due to the thermal interaction between the cauterizing instrument and biological tissue (such as blood vessels or muscle). This smoke may condense on the lens of the endoscope camera or in the operating room, creating fog that obstructs the camera's field of view. Furthermore, bleeding from incisions or ruptured blood vessels may cause blood to splatter onto the lens of the endoscope camera. Additionally, manipulating fatty tissue during a surgical procedure may cause fatty deposits to smear onto the lens. All of these contaminants can obstruct or blur the camera's field of view. Furthermore, sudden camera movement may cause temporary defocusing, resulting in a blurred image.

[0193] In some implementations, instead of performing contamination detection and segmentation on individual frames, the system can process a set of frames collectively. Extracting information from multiple frames can provide a more comprehensive understanding of the scene and contaminants, improving the accuracy of image processing models or algorithms in detecting and segmenting contaminated areas. In some implementations, based on surgical procedures, available computational resources, or user input, the system can be configured to perform contamination detection and segmentation only on a subset of the frame set.

[0194] During contamination detection and segmentation in step 2001, the system can use various methods to identify the type and potential cause of contamination. Scene understanding (SU) can be used to provide context awareness of the surgical scene and classify contamination types, thereby assisting subsequent processing steps. In some implementations, machine learning models can be used, where the system processes input frames to automatically detect and classify contaminants. For example, the machine learning model can take frames as input and output the type of contaminant, such as smoke or blood. Machine learning models can include convolutional neural networks (CNNs), Transformers (e.g., visual Transformers), etc. In some implementations, to reduce latency, quantization (e.g., INT8 or FP16 precision) can be used to optimize the model to reduce computational load, pruning can be used to eliminate redundant weights, and knowledge distillation can be used to create smaller, faster models while maintaining accuracy. Deploying optimization frameworks such as TensorRT, ONNX Runtime, or OpenVINO can be used to reduce inference latency. In some implementations, techniques such as frame skipping and parallel processing on GPUs or another computing platform can also enhance throughput and reduce latency.

[0195] In some implementations, machine learning models, such as CNNs with architectures like the faster region-based CNN (R-CNN), can be used to localize contaminants such as fog, smoke, and blood by identifying and predicting bounding boxes around affected regions in an image. These models can be trained on labeled datasets where contaminants are labeled with bounding box coordinates, allowing the network to learn spatial patterns and object boundaries. Transformer-based localization models, such as the Detection Transformer (DETR), can be used to enhance the process by capturing both global context and local features, potentially improving the accuracy and robustness of the localization task. Training can involve optimizing a loss function that combines a classification loss (e.g., cross-entropy) with a localization loss (e.g., smoothed L1 loss), thereby allowing for accurate bounding box predictions.

[0196] In some implementations, machine learning models, such as CNNs with architectures like U-Net, can be used to segment contaminants. These models can be trained on labeled datasets, where each pixel is labeled according to the contaminant category, allowing the network to learn complex spatial patterns. Transformer-based segmentation models using architectures such as Vision Transformers (ViTs) can be used to extend this capability by capturing global context in addition to local features. Training may involve optimizing loss functions such as the Dice coefficient or cross-entropy loss, thereby allowing for accurate boundary segmentation.

[0197] In some implementations, conventional computer vision algorithms can be used. For example, the system can analyze image histograms or other pixel-level properties to detect contaminants based on different patterns, such as darker areas associated with smoke or bright reflective spots caused by blood splatter.

[0198] In some implementations, edge detection methods, such as those using Sobel or Canny filters, can be used to highlight the boundaries of a blurred image of contaminants, while thresholding techniques can be used to segment regions based on intensity or color values. For example, hazy regions can be segmented by applying a brightness threshold, and blood regions can be detected using color space analysis in the HSV or YUV domains.

[0199] In some implementations, a hybrid approach can be used, combining machine learning and traditional algorithms to improve accuracy. By merging machine learning models with traditional methods, the system can also cross-validate pollutant detection results or refine detection accuracy.

[0200] Hybrid approaches leverage the strengths of both machine learning models and traditional algorithms to achieve efficient and accurate image analysis. Traditional methods, such as edge detection, thresholding, or histogram-based preprocessing, can quickly identify coarse regions of interest (ROIs) in an image. Based on this preprocessing, the system can dynamically decide in real-time whether to apply a localization or segmentation model. For example, if the preprocessing suggests a well-defined region, a localization model like Faster R-CNN can be used to place bounding boxes around the contaminant for efficient processing. In contrast, if the contaminant boundary is diffuse or complex, a segmentation model like U-Net can be applied to achieve pixel-level accuracy. This dynamic selection balances computational efficiency and analytical precision, providing an appropriate level of detail while optimizing resource usage to reduce latency.

[0201] If no contamination is detected in the frame, in step 2002, the system continues to output the frame to the monitor, allowing a clear view of the surgical area without additional processing. Furthermore, in step 2003, the system uses predefined criteria to determine if the frame is suitable for storage in the cache. If the frame meets the necessary conditions, it can be stored as a reference for future recovery procedures. Suitability assessment can involve calculating the edge gradients across the entire frame, where higher gradient values ​​indicate sharper, well-defined edges. Additionally, texture consistency can be analyzed by measuring local variations in pixel intensity, ensuring the preservation of fine anatomical details such as tissue texture and vascular patterns. Suitability assessment can also involve parameters calculated based on Dark Channel Prior (DCP), Fog Aware Density Evaluation (FADE), etc. These parameters can be used to quantify fog levels and occlusion degrees. Similarly, entropy can be analyzed to measure the richness of image information within the frame. If the frame maintains a satisfactory level of visibility, it can be considered sufficient for storage as a reference, even if it is slightly contaminated by artifacts such as haze.

[0202] In some implementations, techniques such as measuring the Laplacian variance of image sharpness can be used to identify frames impaired by motion blur. Frames with significant blur can be discarded. In some implementations, a frame can be discarded if it is detected that the frame is outside the patient's body. Furthermore, a frame can be discarded if a surgical instrument is detected to be too close to a lens (as determined by the size of the image region), as it may obstruct too much of the scene, thus providing insufficient information for further processing.

[0203] In step 2004, frames that meet one or more predefined thresholds of these metrics can be stored in a buffer by the system as reference frames. If a frame fails to meet one or more of these predefined thresholds, it is discarded, as indicated in step 2070. In some implementations, the thresholds can be user-defined, while in others, the system can be configured to automatically determine these thresholds based on predefined settings or real-time factors such as program requirements or specific surgical requirements.

[0204] although Figure 14 The flowchart in the example illustrates caching only uncontaminated, clear frames (i.e., those frames that do not require recovery). However, in some implementations, the system can also cache frames that have already undergone partial recovery, where only the unaffected, unprocessed regions are retained as reference frames. This approach allows the use of partially processed frames as references without introducing AI-modified data into the recovery process, thereby maintaining fidelity in future recovery processes.

[0205] In the context of this technology, a cache is used as a repository of selected frames as reference images to help recover potentially contaminated future frames. By maintaining a set of recent, clear, and context-sensitive frames, the cache enables the system to apply accurate image information to contaminated areas, thereby enhancing the quality of the video output.

[0206] In some implementations, the cache can be implemented using high-speed random access memory (RAM) to allow fast access to the reference frame. In some implementations, for broader storage requirements, a dedicated solid-state drive (SSD) can be used, enabling fast access and higher capacity for storing multiple frames.

[0207] The system can manage the cache by invalidating outdated or no longer relevant frames, as indicated in step 2005. Cache management helps maintain a set of reference frames that are temporally or contextually related to the current frame.

[0208] Time-dependent frames are those captured within the most recent time frame, reflecting the surgical environment and camera position from a recent period. In some implementations, the time frame within which frames are considered time-dependent can be user-defined by the system. In other implementations, the time frame within which frames are considered time-dependent can be automatically determined by the system based on various factors, such as camera movement speed, frequency of frame contamination, specific procedural requirements, or specific surgical requirements.

[0209] Context-relevant frames are those that exhibit at least a certain degree of structural similarity to the current frame and closely match its image content and scene dynamics. Such frames are suitable for guiding recovery by providing reference data consistent with the ongoing procedure and the current field of view. In some implementations, the system user can define the minimum degree of structural similarity that frames considered context-relevant must satisfy. In some implementations, the system can automatically determine the minimum degree of structural similarity that frames considered context-relevant must satisfy based on various factors such as camera movement speed, the type of contaminant detected, the complexity of the anatomical structure, the degree of contamination affecting the area, etc.

[0210] For cache management, the system determines which frames should be invalidated and removed from the cache based on their temporal or contextual relevance to the current frame.

[0211] In some implementations, the system can invalidate frames based on elapsed time, because frames that have exceeded a certain time range may no longer accurately represent the current surgical field of view.

[0212] In some implementations, the system can invalidate frames based on their contextual relevance. In this approach, structural similarity is evaluated between each buffered frame and the current frame. Lower structural similarity between past frames and the current frame indicates that the camera is currently viewing a different region, meaning that past frames will not be used as valid references for the current frame.

[0213] The system can utilize cached frames in various ways. In some implementations, both the current frame and the cached reference frame can be used during recovery, combining information from all regions of both frames for a more comprehensive recovery process. In some implementations, the system can selectively input only the contaminated region (rather than the entire frame) from the cached frame as a reference. This approach minimizes the processing load on the system but may reduce the available image context.

[0214] In some implementations, the system can implement a mask-based method where the current frame, a cached reference frame, and a specified mask can be used during recovery. The mask acts as an indicator, specifying which areas need to be recovered. Uncontaminated image data from the cached reference frame can be used by the system to recover only the contaminated areas specified by the mask, ensuring that affected areas are recovered while leaving uncontaminated areas unchanged. In some implementations, the mask can be specified by the user, while in others, the mask can be automatically generated by the system based on detected contamination patterns or predefined settings.

[0215] Furthermore, for the recovery of the current frame, the system can select an appropriate reference frame from multiple buffered frames in several ways. In some implementations, the system can select the immediately preceding frame as the reference frame. For example, if the surgical scene remains stable and there is no significant camera movement between frames, the immediately preceding frame can provide a highly relevant reference for recovery based on temporal and contextual similarity.

[0216] In some implementations, the system can select a non-immediately adjacent previous frame as the reference frame. For example, in a scene where the camera moves from position 1 at time t1 to position 2 at time t2, and then returns to position 1 at a later time t3, the system can retrieve an uncontaminated, sharp frame cached at time t1 as a reference for reconstructing the contaminated frame captured at time t3, instead of using an uncontaminated, sharp frame cached at time t2. This approach avoids using frames cached at position 2 and time t2, which, while being the immediate previous frame, may not accurately match the current field of view. This selective retrieval allows the system to use a reference frame that better matches the current position and field of view, thereby enhancing reconstruction accuracy.

[0217] If contamination is detected in step 2001, the system proceeds to step 2050 to use the segmentation model, together with the SU, to determine whether the contamination affects at least one region of interest (ROI) in the frame. In some implementations, at least one ROI may be defined by a user using gestures such as zooming in on the region or using a pointing device to select the region. In some implementations, the system may be configured to automatically determine at least one ROI in the frame based on its knowledge of the current surgical procedure.

[0218] If contamination does not affect any ROI, then in step 2002, the system can bypass any recovery and output the frame directly to the monitor. Furthermore, if the frame meets the buffer criteria, the system can store the frame in the buffer, as indicated in steps 2003 and 2004. In this case, although the frame is contaminated, it can still serve as a reference for recovering the contaminated ROI in future frames because the contamination does not affect any ROI.

[0219] However, if the contamination affects the ROI, the system proceeds to step 2060 to assess the size of the contaminated area.

[0220] In step 2060, the system considers factors such as the type and source of contamination to evaluate whether the size of the contaminated area falls within a predefined range. For example, if the contamination is dense fog covering the entire field of view of the camera, the size of the contaminated area may exceed the upper limit, indicating that recovery may be infeasible. In contrast, if the contamination is minimal, such as tiny, localized bloodstains, the size of the contaminated area can be reduced to below the lower limit, in which case the contamination can be ignored and recovery is not required. However, if the size of the contaminated area is neither negligible nor too large to make recovery infeasible, the system may proceed to step 2061 to determine whether downsampling 2013, frame rate reduction 2062, ROI size reduction 2062, or a combination of these adjustments are needed to maintain near real-time operation. Depending on the specific implementation, the system may trigger one or more of these adjustments.

[0221] In the context of this technology, near real-time operation can be defined as minimizing perceived latency to a level where any delay between frame capture and display remains imperceptible to the human eye. End-to-end latency, including capture, processing, and display, can be kept below 200 milliseconds to ensure near real-time operation. This ensures uninterrupted surgical workflows, providing surgeons with continuous and responsive video feeds. The system achieves this by dynamically adjusting processing parameters such as resolution, frame rate, and ROI size to balance computational efficiency and image sharpness while ensuring the displayed recovered frames are shown with minimal latency.

[0222] In some implementations, instead of downsampling the entire frame, only the ROI or contaminated areas within the ROI may be downsampled. This reduces processing load while preserving important details in unaffected areas. In some implementations, this decision can be made by the user based on program requirements, while in others, the system can automatically determine whether to downsample the entire frame or only the ROI or areas within the ROI, based on factors such as the degree and location of contamination.

[0223] Downsampling can be helpful when dealing with large contaminated areas because it reduces the image resolution, thus decreasing the amount of data the system needs to process. This reduction lowers computational load and latency, allowing the restoration model to run more efficiently in a shorter time, which is crucial in real-time applications such as endoscopy. By processing a lower-resolution version of the image, the system can apply the restoration model faster, minimizing latency without compromising essential image information. For example, a 4K resolution (3840×2160) frame might require 50 milliseconds for restoration, while downsampling it to 1080p (1920×1080) can reduce the required processing time to 20 milliseconds, ensuring restoration is completed within the available time before capturing the next frame, thus allowing for near real-time operation.

[0224] In some implementations, ROI size reduction can be used as an alternative to downsampling and frame rate reduction, or in combination with them. The system can dynamically define the region of interest (ROI) within a frame, focusing processing effort only on the most relevant areas, such as those containing surgical instruments or key anatomical structures. The system can track these areas in real time, ensuring image clarification is applied where it is most needed. By limiting restoration to a smaller, dynamically adjusted window instead of the entire frame, the system can reduce processing load, allowing for faster restoration and ensuring real-time operation. For example, processing an entire 1080p frame might take 20 milliseconds, while limiting processing to a 50% cropped ROI (960×540) can reduce the time to 10 milliseconds. Since restoring a smaller ROI requires less computation than processing the entire frame, this adjustment can further reduce latency and allow for near real-time operation.

[0225] In some implementations, the system can dynamically adjust the frame rate of the video feed based on the computing resources available to the system in terms of both software and hardware. In scenarios where recovering each frame would require more time to compute, the system can reduce the frame rate to allow sufficient processing time to ensure that each recovered frame is displayed before the next frame is captured.

[0226] For example, if the system is processing a 30 FPS video feed, each frame must be recovered within 33.3 milliseconds to maintain real-time operation. If the system estimates that the recovery operation will take longer than this time range, it can reduce the frame rate to 20 FPS or lower, effectively increasing the available processing time per frame while maintaining smooth visualization. This adaptive frame rate adjustment prevents processing latency and reduces the delay between captured and displayed frames, thus allowing near real-time operation.

[0227] Following downsampling in step 2013, or if it is determined in step 2061 that adjustments (downsampling, frame rate reduction, ROI size reduction) are not necessary, the image undergoes scene-understanding preprocessing in step 2093. In this step, the system applies adjustments to the frame based on the detected contaminant type, location, and the context of the surgical scene, thereby optimizing the frame for restoration in subsequent processing steps. For example, if the contaminant is identified as fog, the system can apply contrast enhancement to increase the distinction between different anatomical structures and improve overall visibility. If the contaminant is identified as smoke, color correction can be implemented to counteract any shading effects caused by deep smoke particles, thereby restoring color balance in the frame. For blood contamination, the system can employ brightness adjustment to reduce glare and prevent blood from obscuring other details in the frame. Scene understanding can further inform these preprocessing adjustments by considering specific ROIs within the frame and the stage of the procedure. This customized preprocessing step prepares the frame for the next step of model inference, enhancing restoration accuracy by ensuring that the most relevant image information is optimized.

[0228] Following the preprocessing in step 2093, the system proceeds to step 2014, where a reference frame is selected from the cache. In this step, the system can retrieve one or more cached reference frames based on their temporal relevance and contextual similarity to the current frame. The selection process can prioritize reference frames that exhibit high structural similarity to the contaminated frame, while maintaining close temporal proximity to ensure consistency with the ongoing surgical scene. If multiple cached frames are available, the system can use predefined selection criteria, such as the highest sharpness score to prioritize the reference frame with the best clarity, the closest temporal match to select the most recently captured frame, and anatomical consistency to ensure that the reference frame represents the same anatomical structure as the current contaminated frame.

[0229] Next, in step 2015, the system performs model inference, using cached reference frames and scene understanding to guide the recovery. Here, the system can use a Mixture of Experts (MoE) approach, where a specialized model is selected based on the type and characteristics of the detected contaminants. For example, if fog has been identified, a defogging model can be activated to remove the fog and enhance contrast. In the case of motion blur, a deblurring model can be applied to sharpen details lost due to camera movement, while a despeculiarization model can be used to remove particulate noise caused by smoke. If blood or fat deposits are present, models designed to detect and mask these occlusions can be applied to reduce their visibility. In some implementations, if multiple contaminant types are present, a combination of multiple models can be used, such as applying both deblurring and despeculiarization models to address simultaneous motion blur and smoke contamination, ensuring recovery across multiple forms of contamination.

[0230] During model inference in step 2015, Scene Understanding (SU) can assist in identifying key anatomical structures within a frame, thereby helping to prioritize cached frames that best fit the current scene context. For example, in a laparoscopic procedure targeting the gallbladder, if the current frame has a blurred or occluded view of important anatomical features (such as the cystic duct), SU can help the system retrieve a clear, uncontaminated reference frame from the cache that clearly shows the cystic duct. This reference frame provides detailed image information that helps reconstruct partially occluded areas in the current frame, enhancing clarity. And continuity. By utilizing the latest, uncontaminated view of the cystic duct and surrounding structures guided by the SU, the system can generate clear, visualized images of anatomical features, helping users avoid accidental injury during gallbladder dissection or separation.

[0231] In some implementations, various machine learning architectures can be used for inference, including CNNs with solutions such as U-Net or Transformers (such as Visual Transformers (ViTs)) to capture global and local context. Generative Adversarial Networks (GANs), such as Pix2Pix, may be effective for image-to-image translation, converting tainted images into clear ones. Image inpainting models (such as LaMa) can be used to restore missing or damaged regions in images. These architectures rely on large labeled datasets containing different taint scenarios to ensure robust performance in surgical settings.

[0232] In some implementations, data augmentation techniques (such as adding synthetic smoke, fog, or blur) can be used to further improve model generalization. Custom loss functions can be employed to optimize training, including mean squared error for deblurring or perceptual loss for GANs. Evaluation metrics such as peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and mean absolute error can be used to assess fidelity and similarity to the ground truth.

[0233] Next, in step 2009, the system determines whether to apply model inference to local or global restoration based on the degree and distribution of contamination. Local restoration can be performed when contamination affects only a portion of the frame, particularly if the contamination is confined to one or more regions of interest (ROIs), such as the area containing surgical instruments or key anatomical structures. In this case, the system can restore only the contaminated areas while preserving the unaffected parts of the frame without altering the uncontaminated areas. In contrast, global restoration can be applied when contamination is widely distributed across the entire frame, making local restoration insufficient to maintain image sharpness. For example, if fog or smoke diffuses uniformly across the scene, or if motion blur affects the entire frame due to sudden camera movement, the system can choose to perform full-frame restoration to correct the entire frame.

[0234] In some implementations, the system can automatically select between local and global restoration based on the contamination segmentation results. In other implementations, the system can allow the user to manually select the restoration method based on procedural requirements, such as prioritizing local restoration to maintain the clarity of unaffected surgical areas, or selecting global restoration when comprehensive image enhancement is required.

[0235] Once the restoration process is complete, the system can evaluate whether the frame was downsampled earlier in step 2013. If the frame was downsampled, the system proceeds to step 2222, where the restored frame undergoes upsampling to restore its original resolution. In some implementations, upsampling can be performed using interpolation-based techniques such as bilinear or bicubic interpolation. In some implementations, a class of machine learning models known as super-resolution models can be used to upsample the image and restore details from the original frame.

[0236] In some implementations, the required level of resolution may depend on the specific anatomical structures in the field of view, as some surgical tasks may require higher resolution to provide a detailed view of smaller or more complex anatomical structures, such as in pediatric cases or when observing small anatomical areas, such as capillaries or delicate nerve pathways.

[0237] In some implementations, users may be provided with options to select whether upsampling is needed for a specific frame or a specific segment of a frame, using gestures such as zooming in on the area of ​​interest or pointing at the device to select a region. In other implementations, the system may be configured to automatically determine when upsampling is needed based on factors such as the type of anatomical structure detected, specific surgical requirements, the type of contaminant detected, and the degree of contamination in the area.

[0238] Additionally, in step 2081, the system can determine whether contamination was detected in step 2001, or whether the frame rate or ROI size was previously reduced in step 2062. If contamination is detected, or the ROI frame size was reduced, the system can proceed to step 2036, where the original frame rate or ROI size is restored.

[0239] Once a frame has been recovered through model inference and, if necessary, upsampled, it can undergo further refinement in post-processing. During post-processing, the system can apply various visual cues or labels to inform the user that the AI-based recovered area has been accessed. These visual cues or labels can provide information about the specific area modified by the AI, the type of contaminant detected, the recovery model used, and the associated confidence level for each recovery.

[0240] For example, the system can overlay subtle, semi-transparent color codes to highlight areas recovered by different models, such as a blue hue for areas processed with a dehazing model, a green hue for areas processed with deblurring, or a red hue for areas recovered from cached frames. Short, inconspicuous labels can appear near the recovered areas, displaying details such as "AI: Fog Removal" or "AI: Applied Deblurring," and confidence percentages such as "Confidence: 95%" to reassure users of the reliability of the recovered frames.

[0241] In some implementations, the system may additionally provide an interactive feature that allows the operator to temporarily turn the AI-restored area on or off. This comparison tool enables the operator to view the original contaminated view side-by-side with the restored version, thereby fostering increased trust in the AI-generated enhancements.

[0242] Furthermore, in some implementations, the system can use small icon indicators, such as glowing dots or dashed outlines around the processing area, to alert the user in real time whenever AI-based restoration has been activated in a specific area. These visual markers can be fine-tuned for transparency, color, and size, making them easily noticeable without obstructing the user's observation of basic anatomical structures.

[0243] It is conceivable that these visual cues and labels could be valuable during delicate surgical procedures, where real-time information about the recovery process enables users to make informed decisions. By incorporating these detailed indicators, some implementations of this technology provide options for verifying the reliability of frames used in AI-assisted recovery and greater user control.

[0244] It is anticipated that in various embodiments of this technology, machine learning models, conventional computer vision algorithms (e.g., histogram analysis, edge detection, etc.) or a combination of both can be used to perform one or more image processing operations in process 2000, including contamination detection, segmentation, restoration, and enhancement.

[0245] Although the applicant's teachings set forth herein are illustrated with various embodiments for ease of explanation, the applicant's teachings are not limited to these embodiments, which are merely examples. In contrast, the applicant's teachings described and exemplified herein encompass a variety of alternatives, modifications, and equivalents without departing from the embodiments described herein, the overall scope of which is defined in the appended claims.

Claims

1. A method for near real-time image enhancement of endoscopic images, the method being executable by at least one processor, the method comprising: Access the first image from the image stream; It was determined that a contaminated section existed in the first image; Determine at least one of (i) the type of pollutant associated with the contaminated section or (ii) the size of the contaminated section; Adjusting at least one of (iii) the image resolution from the image stream or (iv) the frame rate associated with the image stream, the adjustment being based on at least one of (v) the type of contaminant or (vi) the size of the contaminated segment, the adjustment resulting in at least one of (vii) generating a reduced-resolution image from the image stream or (viii) a reduced image stream; Input (ix) at least one down-resolution image from the down-resolution image or (x) at least one extracted image obtained from the down-resolution image stream to a first machine learning algorithm (MLA) to generate a replacement segment for the contaminated segment; as well as A restored image is generated based on the replaced segment.

2. The method according to claim 1, wherein, The pollution type indicates the presence of at least one of the following: smoke, fog, blood, fat, fumes, or surgical instruments.

3. The method according to any one of claims 1 to 2, further comprising: Accessing the second image from a cache memory that stores cached images, the access to the second image from the cache memory includes: Determine at least one of the following: Temporal correlation scores of multiple cached images, the temporal correlation scores indicating the time elapsed between capturing an individual cached image and the first image; or The context relevance scores of the multiple cached images, wherein the context relevance scores indicate the degree of structural similarity between individual cached images and the first image; as well as The second image is selected from the plurality of cached images based on whether the time relevance score meets the time relevance threshold or the context relevance score meets the context relevance threshold.

4. The method according to any one of claims 1 to 3, further comprising: Whether the first image is suitable for caching is determined based on at least one image quality metric; as well as In response to determining that the first image is suitable for caching, the first image is stored as one of the plurality of cached images in the cache memory.

5. The method according to claim 4, wherein, The at least one image quality indicator includes one or more of the following: The sharpness metric is determined based on at least one of the edge gradient value and the Laplacian variance; or The haze level index is determined based on at least one of the Dark Channel Prior (DCP) value and the Fog Sensing Density Evaluation (FADE) value; or The information richness index is determined based on entropy analysis.

6. The method according to claim 5, wherein, Individual cached images are removed from the cache memory in response to at least one of the following: The temporal relevance score of the individual cached image did not exceed the temporal relevance threshold; and The context relevance score of the individual cached image did not exceed the context relevance threshold.

7. The method according to any one of claims 1 to 6, further comprising selectively upsampling the restored image.

8. The method according to any one of claims 1 to 7, further comprising selectively increasing the frame rate associated with the image stream.

9. The method according to any one of claims 1 to 8, further comprising selectively adjusting the size of the region of interest (ROI) used to determine that the contaminated segment exists within the first image.

10. The method according to any one of claims 1 to 9, further comprising: Generate a visual cue on the restored image, the visual cue indicating at least one of the following: The area corresponding to the replacement segment; and The reliability score of the replacement segment within the synthesized image frame.

11. The method according to any one of claims 1 to 10, wherein, Determining that the contaminated segment exists within the first image includes: performing a second machine learning algorithm (MLA) by inputting the first image into the second MLA and having the second MLA output at least one of the following: (i) the type of contaminant associated with the contaminated segment or (ii) the size of the contaminated segment.

12. The method according to any one of claims 1 to 11, wherein, Before determining that the contaminated segment exists within the first image, the method performs grouping of the first image with another image from the image stream.

13. The method according to any one of claims 1 to 12, wherein, The presence of the contaminated segment is determined for some, but not all, images in the image stream.

14. The method according to claims 1 to 13, wherein, Adjusting (iii) the image resolution from the image stream or (iv) the frame rate associated with the image stream is based on the available computing resources of the device on which the method is performed.

15. The method according to claims 1 to 14, wherein, Before inputting (ix) the at least one down-resolution image from the down-resolution image or (x) the at least one extracted image from the down-resolution image stream to the first MLA, the method performs: The first MLA is selected from a plurality of MLAs based on at least one of the following: The type of pollutant associated with the contaminated section; or Available computing resources of the device on which the method is performed.

16. The method according to any one of claims 1 to 15, wherein, Before being used to generate the replacement segment of the contaminated segment, the at least one extracted image obtained from the reduced image stream is further processed to reduce its resolution.

17. The method according to any one of claims 1 to 16, further comprising displaying the restored image.

18. A method for near real-time image enhancement of endoscopic images, the method being executable by at least one processor, the method comprising: Access the first image from the image stream; It was determined that a contaminated section existed in the first image; Determine at least one of (i) the type of pollutant associated with the contaminated section or (ii) the size of the contaminated section; Adjusting at least one of (iii) the resolution of the first image or (iv) the frame rate associated with the image stream, the adjustment being based on at least one of (v) the type of the contaminant or (vi) the size of the contaminated segment, the adjustment resulting in at least one of (vii) generating a reduced-resolution image from the first image or (viii) a reduced image stream; Input (ix) the reduced-resolution image or (x) at least one extracted image obtained from the reduced image stream into a machine learning algorithm (MLA) to generate a replacement segment for the contaminated segment; and A restored image is generated based on the replaced segment.

19. The method of claim 18, further comprising: Accessing the second image from a cache memory that stores cached images, the access to the second image from the cache memory includes: Determine at least one of the following: Temporal correlation scores of multiple cached images, the temporal correlation scores indicating the time elapsed between capturing an individual cached image and the first image; or The context relevance scores of the multiple cached images, wherein the context relevance scores indicate the degree of structural similarity between individual cached images and the first image; and The second image is selected from the plurality of cached images based on whether the time relevance score meets the time relevance threshold or the context relevance score meets the context relevance threshold.

20. The method according to any one of claims 18 to 19, further comprising displaying the restored image.

21. A method for near real-time image enhancement of endoscopic images, the method being executable by at least one processor, the method comprising: Access the first image from the image stream; It was determined that a contaminated section existed in the first image; Determine at least one of (i) the type of pollutant associated with the contaminated section or (ii) the size of the contaminated section; The resolution of the first image is adjusted based on at least one of (v) the type of the contaminant or (vi) the size of the contaminated segment, resulting in the generation of a reduced-resolution image from the first image; The reduced-resolution image is input (ix) into a machine learning algorithm (MLA) to generate a replacement segment for the contaminated segment; and A restored image is generated based on the replaced segment.

22. The method of claim 21, further comprising: Accessing the second image from a cache memory that stores cached images, the access to the second image from the cache memory includes: Determine at least one of the following: Temporal correlation scores of multiple cached images, the temporal correlation scores indicating the time elapsed between capturing an individual cached image and the first image; or The context relevance scores of the multiple cached images, wherein the context relevance scores indicate the degree of structural similarity between individual cached images and the first image; and The second image is selected from the plurality of cached images based on whether the time relevance score meets the time relevance threshold or the context relevance score meets the context relevance threshold.

23. The method according to any one of claims 21 to 22, further comprising displaying the restored image.

24. A method for near real-time image enhancement of endoscopic images, the method being executable by at least one processor, the method comprising: Access the first image from the image stream; It was determined that a contaminated section existed in the first image; Determine at least one of (i) the type of pollutant associated with the contaminated section or (ii) the size of the contaminated section; Adjusting the frame rate associated with the image stream, the adjustment being based on at least one of (v) the type of contaminant or (vi) the size of the contaminated segment, the adjustment resulting in a reduced image stream; Input at least one extracted image from the reduced image stream into a machine learning algorithm (MLA) to generate a replacement segment for the contaminated segment; as well as A restored image is generated based on the replaced segment.

25. The method of claim 24, further comprising: Accessing the second image from a cache memory that stores cached images, the access to the second image from the cache memory includes: Determine at least one of the following: Temporal correlation scores of multiple cached images, the temporal correlation scores indicating the time elapsed between capturing an individual cached image and the first image; or The context relevance scores of the multiple cached images, wherein the context relevance scores indicate the degree of structural similarity between individual cached images and the first image; as well as The second image is selected from the plurality of cached images based on whether the time relevance score meets the time relevance threshold or the context relevance score meets the context relevance threshold.

26. The method according to any one of claims 24 to 25, further comprising displaying the restored image.

27. A method for near real-time image enhancement of endoscopic images, the method being executable by at least one processor, the method comprising: Access the first image from the image stream; It was determined that a contaminated segment existed within the region of interest (ROI) of the first image; Determine at least one of (i) the type of pollutant associated with the contaminated section or (ii) the size of the contaminated section; In response to the type of pollutant or the size of the contaminated segment being below a first threshold, Estimate the duration required to perform image processing operations on the contaminated section; Based on the estimated duration, selectively reduce at least one of the following: The resolution of the first image, The rate associated with the image stream, and The dimensions of the ROI; Access the second image from the cache memory that stores the cached images; Based on the second image, a prediction of a replacement segment for replacing the contaminated segment is generated; The replacement segment is composited onto the contaminated segment to generate a composite image; At least one of the first image and the synthesized image may be selectively output to a display.

28. A method for managing a cache memory storing multiple cached images, the method comprising: Access the first image from the image stream; Access the cache memory that stores multiple cached images; Determine at least one of the following: The temporal correlation scores of the plurality of cached images, wherein the temporal correlation scores indicate the time elapsed between capturing an individual cached image and the first image; or The context relevance scores of the multiple cached images, wherein the context relevance scores indicate the degree of structural similarity between individual cached images and the first image; as well as The second image is selected from the plurality of cached images based on whether the time relevance score meets the time relevance threshold or the context relevance score meets the context relevance threshold. as well as Return to the second image.

29. The method of claim 28, further comprising: Whether the first image is suitable for caching is determined based on at least one image quality metric; as well as In response to determining that the first image is suitable for caching, the first image is stored as one of the plurality of cached images in the cache memory.

30. The method according to claim 29, wherein, The at least one image quality indicator includes one or more of the following: The sharpness metric is determined based on at least one of the edge gradient value and the Laplacian variance; or The haze level index is determined based on at least one of the Dark Channel Prior (DCP) value and the Fog Sensing Density Evaluation (FADE) value; or The information richness index is determined based on entropy analysis.

31. The method according to any one of claims 28 to 30, wherein, The individual cached image is removed from the cache memory in response to at least one of the following: The temporal correlation score of the individual cached image does not exceed the temporal correlation threshold; and The context relevance score of the individual cached image did not exceed the context relevance threshold.

32. The method according to any one of claims 28 to 31, further comprising selectively inputting the second image into a machine learning algorithm (MLA).

33. A system for near real-time image enhancement of endoscopic images, the system comprising at least one processor configured to: Access the first image from the image stream; It was determined that a contaminated section existed in the first image; Determine at least one of (i) the type of pollutant associated with the contaminated section or (ii) the size of the contaminated section; Adjusting at least one of (iii) the image resolution from the image stream or (iv) the frame rate associated with the image stream, the adjustment being based on at least one of (v) the type of contaminant or (vi) the size of the contaminated segment, the adjustment resulting in at least one of (vii) generating a reduced-resolution image from the image stream or (viii) a reduced image stream; Inputting (ix) at least one down-resolution image from the down-resolution image or (x) at least one extracted image from the down-resolution image stream into a first machine learning algorithm (MLA) to generate a replacement segment for the contaminated segment; and A restored image is generated based on the replaced segment.

34. The system according to claim 33, wherein, The pollution type indicates the presence of at least one of the following: smoke, fog, blood, fat, fumes, or surgical instruments.

35. The system according to any one of claims 33 to 34, wherein, The at least one processor is further configured to: Accessing the second image from a cache memory that stores cached images, the access to the second image from the cache memory includes: Determine at least one of the following: Temporal correlation scores of multiple cached images, the temporal correlation scores indicating the time elapsed between capturing an individual cached image and the first image; or The context relevance scores of the multiple cached images, wherein the context relevance scores indicate the degree of structural similarity between individual cached images and the first image; as well as The second image is selected from the plurality of cached images based on whether the time relevance score meets the time relevance threshold or the context relevance score meets the context relevance threshold.

36. The system according to any one of claims 33 to 35, wherein, The at least one processor is further configured to: Whether the first image is suitable for caching is determined based on at least one image quality metric; and In response to determining that the first image is suitable for caching, the first image is stored as one of the plurality of cached images in the cache memory.

37. The system according to claim 36, wherein, The at least one image quality indicator includes one or more of the following: The sharpness metric is determined based on at least one of the edge gradient value and the Laplacian variance; or The haze level index is determined based on at least one of the Dark Channel Prior (DCP) value and the Fog Sensing Density Evaluation (FADE) value; or The information richness index is determined based on entropy analysis.

38. The system according to claim 37, wherein, Individual cached images are removed from the cache memory in response to at least one of the following: The temporal relevance score of the individual cached image did not exceed the temporal relevance threshold; and The context relevance score of the individual cached image did not exceed the context relevance threshold.

39. The system according to any one of claims 33 to 38, wherein, The at least one processor is further configured to selectively oversample the restored image.

40. The system according to any one of claims 33 to 39, wherein, The at least one processor is further configured to selectively increase the frame rate associated with the image stream.

41. The system according to any one of claims 33 to 40, wherein, The at least one processor is further configured to selectively adjust the size of the region of interest (ROI) used to determine whether the contaminated segment exists within the first image.

42. The system according to any one of claims 33 to 40, wherein, The at least one processor is further configured to: Generate a visual cue on the restored image, the visual cue indicating at least one of the following: The area corresponding to the replacement segment; and The reliability score of the replacement segment within the synthesized image frame.

43. The system according to any one of claims 33 to 42, wherein, Determining that the contaminated segment exists within the first image includes: performing a second machine learning algorithm (MLA) by inputting the first image into the second MLA and having the second MLA output at least one of the following: (i) the type of contaminant associated with the contaminated segment or (ii) the size of the contaminated segment.

44. The system according to any one of claims 33 to 43, wherein, Before determining that the contaminated segment exists within the first image, the system performs grouping of the first image with another image from the image stream.

45. The system according to any one of claims 33 to 44, wherein, The presence of the contaminated segment is determined for some, but not all, images in the image stream.

46. ​​The system according to any one of claims 33 to 45, wherein, Adjusting at least one of (iii) the image resolution from the image stream or (iv) the frame rate associated with the image stream is based on the available computing resources of the system.

47. The system according to any one of claims 33 to 46, wherein, Before inputting (ix) the at least one down-resolution image from the down-resolution image or (x) the at least one extracted image from the down-resolution image stream to the first MLA, the system performs: The first MLA is selected from a plurality of MLAs based on at least one of the following: The type of pollutant associated with the contaminated section; or The system has available computing resources.

48. The system according to any one of claims 33 to 47, wherein, Before being used to generate the replacement segment of the contaminated segment, the at least one extracted image obtained from the reduced image stream is further processed to reduce its resolution.

49. The system according to any one of claims 33 to 48, further configured to display the restored image.

50. A system for near real-time image enhancement of endoscopic images, the system being executable by at least one processor, the system comprising: Access the first image from the image stream; It was determined that a contaminated section existed in the first image; Determine at least one of (i) the type of pollutant associated with the contaminated section or (ii) the size of the contaminated section; Adjusting at least one of (iii) the resolution of the first image or (iv) the frame rate associated with the image stream, the adjustment being based on at least one of (v) the type of the contaminant or (vi) the size of the contaminated segment, the adjustment resulting in at least one of (vii) generating a reduced-resolution image from the first image or (viii) a reduced image stream; Input (ix) the reduced-resolution image or (x) at least one extracted image obtained from the reduced image stream into a machine learning algorithm (MLA) to generate a replacement segment for the contaminated segment; as well as A restored image is generated based on the replaced segment.

51. The system of claim 50, further comprising: Accessing the second image from a cache memory that stores cached images, the access to the second image from the cache memory includes: Determine at least one of the following: Temporal correlation scores of multiple cached images, the temporal correlation scores indicating the time elapsed between capturing an individual cached image and the first image; or The context relevance scores of the multiple cached images, wherein the context relevance scores indicate the degree of structural similarity between individual cached images and the first image; as well as The second image is selected from the plurality of cached images based on whether the time relevance score meets the time relevance threshold or the context relevance score meets the context relevance threshold.

52. The system according to any one of claims 50 to 51, further configured to display the restored image.

53. A system for near real-time image enhancement of endoscopic images, the system comprising at least one processor configured to: Access the first image from the image stream; It was determined that a contaminated section existed in the first image; Determine at least one of (i) the type of pollutant associated with the contaminated section or (ii) the size of the contaminated section; The resolution of the first image is adjusted based on at least one of (v) the type of the contaminant or (vi) the size of the contaminated segment, resulting in the generation of a reduced-resolution image from the first image; The reduced-resolution image is input (ix) into a machine learning algorithm (MLA) to generate a replacement segment for the contaminated segment; as well as A restored image is generated based on the replaced segment.

54. The system according to claim 53, wherein, The at least one processor is further configured to: Accessing the second image from a cache memory that stores cached images, the access to the second image from the cache memory includes: Determine at least one of the following: Temporal correlation scores of multiple cached images, the temporal correlation scores indicating the time elapsed between capturing an individual cached image and the first image; or The context relevance scores of the multiple cached images, wherein the context relevance scores indicate the degree of structural similarity between individual cached images and the first image; as well as The second image is selected from the plurality of cached images based on whether the time relevance score meets the time relevance threshold or the context relevance score meets the context relevance threshold.

55. The system according to any one of claims 53 to 54, further configured to display the restored image.

56. A system for near real-time image enhancement of endoscopic images, the system comprising at least one processor configured to: Access the first image from the image stream; It was determined that a contaminated section existed in the first image; Determine at least one of (i) the type of pollutant associated with the contaminated section or (ii) the size of the contaminated section; Adjusting the frame rate associated with the image stream, the adjustment being based on at least one of (v) the type of contaminant or (vi) the size of the contaminated segment, the adjustment resulting in a reduced image stream; Input at least one extracted image from the reduced image stream into a machine learning algorithm (MLA) to generate a replacement segment for the contaminated segment; as well as A restored image is generated based on the replaced segment.

57. The system according to claim 56, wherein, The at least one processor is further configured to: Accessing the second image from a cache memory that stores cached images, the access to the second image from the cache memory includes: Determine at least one of the following: Temporal correlation scores of multiple cached images, the temporal correlation scores indicating the time elapsed between capturing an individual cached image and the first image; or The context relevance scores of the multiple cached images, wherein the context relevance scores indicate the degree of structural similarity between individual cached images and the first image; as well as The second image is selected from the plurality of cached images based on whether the time relevance score meets the time relevance threshold or the context relevance score meets the context relevance threshold.

58. The system according to any one of claims 56 to 57, further configured to display the restored image.

59. A system for near real-time image enhancement of endoscopic images, the system comprising at least one processor configured to: Access the first image from the image stream; It was determined that a contaminated segment existed within the region of interest (ROI) of the first image; Determine at least one of (i) the type of pollutant associated with the contaminated section or (ii) the size of the contaminated section; In response to the type of pollutant or the size of the contaminated segment being below a first threshold, Estimate the duration required to perform image processing operations on the contaminated section; Based on the estimated duration, selectively reduce at least one of the following: The resolution of the first image, The rate associated with the image stream, and The dimensions of the ROI; Access the second image from the cache memory that stores the cached images; Based on the second image, a prediction of a replacement segment for replacing the contaminated segment is generated; The replacement segment is composited onto the contaminated segment to generate a composite image; At least one of the first image and the synthesized image may be selectively output to a display.

60. A system for managing a cache memory storing multiple cached images, the system comprising at least one processor configured to: Access the first image from the image stream; Access the cache memory that stores multiple cached images; Determine at least one of the following: The temporal correlation scores of the plurality of cached images, wherein the temporal correlation scores indicate the time elapsed between capturing an individual cached image and the first image; or The context relevance scores of the multiple cached images, wherein the context relevance scores indicate the degree of structural similarity between individual cached images and the first image; as well as Based on whether the time relevance score meets the time relevance threshold or the context relevance score meets the context relevance threshold, a second image is selected from the plurality of cached images; as well as Return to the second image.

61. The system according to claim 60, wherein, The at least one processor is further configured to: Whether the first image is suitable for caching is determined based on at least one image quality metric; as well as In response to determining that the first image is suitable for caching, the first image is stored as one of the plurality of cached images in the cache memory.

62. The system according to claim 61, wherein, The at least one image quality indicator includes one or more of the following: The sharpness metric is determined based on at least one of the edge gradient value and the Laplacian variance; or The haze level index is determined based on at least one of the Dark Channel Prior (DCP) value and the Fog Sensing Density Evaluation (FADE) value; or The information richness index is determined based on entropy analysis.

63. The system according to any one of claims 60 to 62, wherein, The at least one processor is further configured to remove an individual cached image from the cache memory in response to at least one of the following: The temporal correlation score of the individual cached image does not exceed the temporal correlation threshold; and The context relevance score of the individual cached image did not exceed the context relevance threshold.

64. The system according to any one of claims 60 to 63, wherein, The at least one processor is further configured to selectively input the second image into a machine learning algorithm (MLA).

65. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, configure the at least one processor to perform the method of any one of claims 1 to 32.

66. A method for improving the quality of endoscopic images, the method comprising: Receive image data including image frames; The first image frame in the image frame is cached, thereby generating a cached image frame; Capture the second image frame from the image frame to obtain the captured image frame; It was determined that the captured image frame was contaminated; Identify contaminated image segments from the contaminated captured image frames; Generate predictions of replacement image segments for replacing contaminated image segments; as well as The replaced image segment is composited onto the cached image frame to generate a composite image.

67. The method of claim 66, further comprising: Replace the contaminated segmented image frames with the synthesized image.

68. The method according to any one of claims 1 to 67, further comprising: The cached image frames are classified into the last clear frames using a first AI model for segmentation.

69. The method according to any one of claims 1 to 68, wherein, The determination that the captured image frame is contaminated is based at least in part on transmitting the captured image frame to a first AI model for segmentation.

70. The method according to any one of claims 1 to 69, further comprising: Segmenting contaminated image segments from the occluded captured image frames.

71. The method according to any one of claims 1 to 70, further comprising: The contaminated image segment is input into a second AI model for generation to preprocess the contaminated image segment for recovery.

72. The method according to any one of claims 1 to 71, wherein, The prediction of generating the replacement image segment to replace the contaminated image segment is at least partially based on the use of a second AI model for generation.

73. The method according to any one of claims 1 to 72, wherein, The replacement image segment is synthesized onto the cached image frame, at least in part based on the use of a second AI model for generation to produce the synthesized image using the last clear frame.

74. A method for improving the quality of endoscopic images, the method comprising: Receive image data including image frames; Capture the first image frame in the image frame to obtain the captured image frame; It was determined that the captured image frame was contaminated; Segment the contaminated portion of the captured image frame; The recovered image frames are generated based at least in part on the last clear frame and the captured image frames.

75. The method of claim 74, further comprising: The contaminated captured frame is replaced with the restored image frame.

76. The method according to any one of claims 74 to 75, wherein, The determination that the captured frame is contaminated is based at least in part on the classification by a first AI model used for segmentation, which classifies the captured frame as contaminated.

77. The method according to any one of claims 74 to 76, wherein, The first AI model used for segmentation is used to complete the segmentation of the contaminated portion of the captured frame.

78. The method according to any one of claims 74 to 77, further comprising: The contaminated image segment is input into a second AI model for generation to preprocess the contaminated image segment for recovery.

79. The method according to any one of claims 74 to 78, further comprising: A prediction of a replacement image segment is generated to replace the contaminated image segment, in order to generate the restored image frame.

80. A method for improving the quality of endoscopic images, the method comprising: Receive image data including image frames; Capture the first image frame in the image frame to obtain the first captured image frame; It was determined that the first captured frame was not contaminated; Store the first captured image frame as the last clear frame; Capture the second image frame from the image frame to obtain the second captured image frame; It was determined that the second captured image frame was contaminated; and The recovered image frame is generated based at least in part on the last clear frame and the second captured image frame.

81. The method of claim 80, further comprising: The contaminated second captured frame is replaced with the restored image frame.

82. The method according to any one of claims 80 to 81, wherein, Determining that the first captured frame was not contaminated was accomplished using a first AI model for segmentation.

83. The method according to any one of claims 74 to 82, further comprising: The contaminated portion of the second captured image frame is segmented.

84. The method according to any one of claims 74 to 83, further comprising: The contaminated image segment is input into a second AI model for generation to preprocess the contaminated image segment for recovery.

85. The method according to any one of claims 74 to 84, wherein, The generation of the restored image frame is also based, at least in part, on processing the last clear frame using a second AI model used for generation.