Image processing system and method of using the same

Abstract

To visualize blood vessel and a target therapeutic part of a subject, for example, a tumor or lesion in the gastrointestinal tract of the subject.SOLUTION: The current invention relates to a method for processing an image. The method comprises: receiving an image (or an image frame); removing a plurality of first pixel values from the received image; determining a second plurality of second pixel values at a plurality of first locations and a plurality of third locations on the image; determining a third plurality of second predicted pixel values at the plurality of first locations and the plurality of second locations on the image; and generating a processed image on the basis of the plurality of second pixel values, the plurality of second predicted pixel values, a plurality of third pixel values, and a plurality of third predicted pixel values.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] Various aspects of the present disclosure generally relate to image processing systems, devices, and related methods. Examples of the present disclosure relate to systems, devices, and related methods for digital dye endoscopy and the like.

Background Art

[0002] With technological development, users of medical systems, devices, and methods have been able to perform increasingly complex procedures on subjects. One problem in the technical field of minimally invasive surgery involves visualizing a target treatment site within a subject's body, such as a tumor or lesion within the subject's gastrointestinal tract. Dye endoscopy with dye injection can facilitate the detection of changes on the mucosal surface within the gastrointestinal tract, which is a lumen. However, due to the limited imaging methods and devices for visualizing blood vessels, the procedure can be prolonged, its effectiveness can be limited, and / or harm can be inflicted on the patient.

Summary of the Invention

[0003] Aspects of the present disclosure relate, among other things, to systems, devices, and methods for providing an image processing system and, among other things, to wavelength shift demosaicking logic. Each of the aspects disclosed herein may include one or more of the features described with respect to any of the other disclosed aspects.

[0004] In one example, a medical device includes a shaft and a sensor connected to the distal end of the shaft, which includes a filter array. The sensor is configured to capture a raw image, and the filter array is configured to filter the raw image to obtain a frame of raw pixel values ​​including a plurality of first pixel values, a plurality of second pixel values, and a plurality of third pixel values. The medical device includes a processor and a non-temporary computer-readable medium which, when executed by the processor, stores demosaiking instructions causing the processor to exclude a plurality of first pixel values ​​from a frame of raw pixel values. The processor generates a plurality of second expected pixel values ​​at the locations of the excluded first pixel values ​​and a plurality of third pixel values ​​on the frame. The processor generates a processed image having a partially resolved image frame from the plurality of second pixel values, a plurality of second expected pixel values, a plurality of third pixel values, and a plurality of third expected pixel values.

[0005] Any medical device described herein may include any of the following features: Demosaiking instructions stored in a non-temporary computer-readable medium cause a processor to detect one or more edges in the original image and to perform sharpened enhancement on one or more edges to increase the edge detail of the processed image. Demosaiking instructions stored in a non-temporary computer-readable medium cause a processor to output a sharpened enhancement image created from performing the sharpened enhancement step and to fuse the sharpened enhancement image with the processed image. Demosaiking instructions stored in a non-temporary computer-readable medium cause a processor to perform contrast enhancement on multiple second pixel values ​​and multiple third pixel values ​​by setting the luminance values ​​of each of multiple second pixel values ​​and multiple third pixel values ​​and adjusting the luminance values ​​to increase the contrast of the processed image. Demosaiking instructions stored in a non-temporary computer-readable medium cause the processor to output a contrast-enhanced image created from the execution of a contrast enhancement step and to fuse the contrast-enhanced image with the processed image. Demosaiking instructions stored in a non-temporary computer-readable medium cause the processor to repeat all of the above steps up to a threshold. Demosaiking instructions stored in a non-temporary computer-readable medium cause the processor to receive a wavelength shift input to identify the color pixel values ​​of a plurality of first pixel values. Furthermore, it includes a user interface that is communicatively coupled to the processor and configured to transmit signals indicating the wavelength shift input to the processor. The sensor includes an RGB image sensor and the filter array includes a red-green-blue Bayer color filter array. A plurality of first pixel values ​​include red pixel values, a plurality of second pixel values ​​include blue pixel values, and a plurality of third pixel values ​​include green pixel values. The sensor includes an RGB+Ir image sensor and the filter array includes a red-green-blue-infrared Bayer color filter array. Multiple first pixel values ​​include blue pixel values, multiple second pixel values ​​include red and green pixel values, and multiple third pixel values ​​include infrared pixel values.The sensor includes an RGB image sensor and a monochrome sensor. Each location in the partially resolved image frame includes one captured color pixel value and one reconstructed color pixel value, thereby excluding at least one color pixel value from the frame of original pixel values. Furthermore, it includes a light source connected to the distal end of the shaft, which is an optical fiber, ultraviolet light, or a multicolor LED array. Demosaiking instructions stored in a non-temporary computer-readable medium cause the processor to output the processed image of the partially resolved image frame to a display device.

[0006] According to other examples, an image processing method includes the steps of capturing a source image, and filtering the source image to obtain a frame of source pixel values ​​containing a plurality of first pixel values, a plurality of second pixel values, and a plurality of third pixel values. The method includes the steps of excluding at least a plurality of first pixel values, and generating missing second pixel values ​​at the pixel locations of the plurality of excluded first pixel values ​​and a plurality of third pixel values ​​along the frame. The method includes the steps of generating missing third pixel values ​​at the pixel locations of the plurality of excluded first pixel values ​​and a plurality of second pixel values ​​along the frame, and constructing a digital image partially sampled from the plurality of second pixel values, a plurality of third pixel values, the generated second pixel values, and the generated third pixel values.

[0007] Any method described herein may include any of the following steps: a method comprising the steps of detecting an edge in a frame and increasing the sharpness of the edge to increase edge detail in a partially sampled digital image; a method comprising the steps of changing the luminance values ​​for each of a plurality of second pixel values ​​and a plurality of third pixel values ​​to increase the contrast between a plurality of second pixel values ​​and a plurality of third pixel values ​​in a partially sampled digital image; a method comprising the steps of receiving a wavelength shift input to identify the color pixel values ​​of a plurality of first pixel values; the frame of original pixel values ​​further comprising a plurality of fourth pixel values; and a method comprising the step of excluding at least a plurality of fourth pixel values.

[0008] In another example, the system comprises a processor and a non-temporary computer-readable medium that stores instructions, when executed by the processor, causing the processor to transmit light of multiple wavelengths from an illumination source and to remove at least a small portion of multiple pixel values ​​detected by a digital image sensor communicatively coupled to the processor. The digital image sensor includes a filter array configured to filter multiple wavelengths of light to obtain multiple pixel values. The processor generates multiple predicted pixel values, creates a processed image having multiple pixel values ​​and multiple predicted pixel values ​​in a partially resolved image frame, and removes at least a small portion of the multiple pixel values ​​from the processed image.

[0009] Please understand that both the general description above and the detailed description below are for illustrative and explanatory purposes only and do not limit the present invention as defined by the claims. The accompanying drawings incorporated herein and constituting part thereof illustrate exemplary embodiments of the Disclosure and, together with the explanatory text, serve to illustrate the principles of the Disclosure. [Brief explanation of the drawing]

[0010] [Figure 1] A schematic diagram illustrating an exemplary medical system according to the aspects of this disclosure. [Figure 2A]A partial perspective view of a medical device of the medical system of Figure 1, including a sensor and a light source, according to an aspect of this disclosure. [Figure 2B] A partial perspective view of another medical device of the medical system of Figure 1, including a sensor and multiple light sources, according to an aspect of this disclosure. [Figure 2C] A partial perspective view of another medical device of the medical system of Figure 1, including a pair of sensors and a light source, according to an aspect of this disclosure. [Figure 3] A schematic diagram of an exemplary image sensor of the medical device shown in Figure 2A, according to an aspect of this disclosure. [Figure 4] A schematic diagram of a frame of raw pixel data received from an image captured by the image sensor of Figure 3, according to an aspect of this disclosure. [Figure 5] A block diagram of an exemplary method for imaging a target site using the medical system shown in Figure 1, according to an aspect of this disclosure. [Modes for carrying out the invention]

[0011] Examples of the present disclosure include systems, devices, and methods for improving the quality of images of one or more targeted therapeutic sites within the body of a subject (e.g., a patient) by highlighting one or more features of the target site (e.g., blood vessels, vascular system, etc.) in the processed images. Examples of aspects of the present disclosure are described below in detail, with illustrations provided in the accompanying drawings. Where possible, the same or similar reference numerals throughout the drawings are used to refer to the same or similar parts. The term “distal” refers to the part of the device furthest from the user when it is introduced into the patient’s body. The term “proximal” refers to the part of the device closest to the user when it is placed in the subject’s body. As used herein, the terms “comprises,” “comprising,” or any other variations thereof cover non-exclusive inclusion, and a process, method, article, or device containing the enumerated elements does not necessarily contain only those elements, but may include other elements not expressly listed or specific to such process, method, article, or device. The term “exemplary” is used to mean “example” rather than “ideal.” As used herein, the terms “about,” “substantially,” and “almost” refer to a range of values ​​within + / - 10% of the stated value.

[0012] An example of the present disclosure may be used to identify a target site within a subject's body by generating a processed image having a partial-resolution frame of pixel values ​​that visually highlights one or more features and / or characteristics of the subject's gastrointestinal tract, which is a lumen of the subject. Such features and / or characteristics include, for example, tumors, lesions, blood vessels, changes in the mucosal surface, etc. In some embodiments, the medical device may include an image processing device, which includes a processor and a memory that stores one or more algorithms for generating partial-resolution frames. In embodiments, the memory may include programmable instructions according to demosaicing logic, edge enhancement logic, and / or contrast enhancement logic. Furthermore, the image processing device may include a user interface that can be operated to receive user input, which is, for example, a wavelength shift input for filtering out at least one color pixel value, after which interpolation and enhancement of other color pixel values ​​captured by an image sensor are performed. The processed image generated by the image processing device of the medical device may include a partial-resolution frame of pixel values, which may be output to a display device.

[0013] Examples of the present disclosure may relate to devices and methods for performing various medical procedures and / or treating any other part of the large intestine (colon), small intestine, cecum, esophagus, and gastrointestinal tract and / or any other appropriate part of the patient's internal structure (collectively referred to herein as “target therapeutic sites”). The various examples described herein include single-use or disposable medical devices. The examples of the present disclosure described above and shown in the accompanying drawings are described in detail below. Wherever possible, the same reference numerals are used throughout the drawings to refer to the same or similar parts.

[0014] Figure 1 shows a schematic diagram of an exemplary medical system 100 according to an example of the present disclosure. The medical system 100 may include one or more light sources 130, an image processing device 101, a medical instrument 110, and a medical device 140. The image processing device 101 may be communicably connected to the medical instrument 110, for example, by a wired connection, a wireless connection, etc. In the example, the image processing device 101 is a computer system incorporating a plurality of hardware components that enable the image processing device 101 to receive data (e.g., image sensor data), process information (e.g., wavelength data), and / or generate a processed image for output to the user of the medical system 100. Exemplary hardware components of the image processing device 101 may include at least one processor 102, at least one memory 103, at least one user interface 108, and at least one display 109.

[0015] The processor 102 of the image processing apparatus 101 may include any computing device capable of executing machine-readable instructions, the instructions may be stored in a non-temporary computer-readable medium, such as the memory 103 of the image processing apparatus 101. For example, the processor 102 may include a controller, an integrated circuit, a microchip, a computer, and / or any other computer processing unit capable of operating to perform calculations and logic operations necessary to execute a program. As will be described in more detail herein, the processor 102 is configured to perform one or more operations, such as imaging logic 104, demosaicing logic 105, edge enhancement logic 106, contrast enhancement logic 107, etc., according to instructions stored on the memory 103.

[0016] Continuing to refer to Figure 1, the memory 103 of the image processing device 101 may include a non-temporary computer-readable medium on which machine-readable instructions, such as imaging logic 104, demosaicing logic 105, edge enhancement logic 106, and contrast enhancement logic 107 are stored. The imaging logic 104 may include executable instructions that enable the medical system 100 to capture a primal digital image by activating one or more components of the medical device 110, such as one or more image sensors 150, 150A, 150B (Figures 2A to 2C).

[0017] Furthermore, the demosaiking logic 105 may include executable instructions that enable the medical system 100 to process a digital image (e.g., a mosaic image) by demosaiking the image and reconstructing missing and / or unknown pixel values ​​in the mosaic image. It should be understood that a digital image captured by an image sensor using a color filter sensor array may provide a raw image having various color pixel values ​​arranged in a mosaic pattern. Each pixel array in the pattern contains only one color pixel value, thereby allowing one or more color pixel values ​​to be removed. As will be described in detail herein, a digital image includes a two-dimensional array of pixel values, where each pixel value corresponds to a light intensity (e.g., a color pixel value) in one of several spectral bands at a given pixel location in the image.

[0018] Continuing to refer to Figure 1, the edge enhancement logic 106 may include executable instructions that allow the medical system 100 to process a mosaic image of a target area and enhance the sharpness of one or more edges within the mosaic image. It should be understood that the demosaiking process may have inherent side effects, such as a decrease in the sharpness of one or more edges within the image. For example, the demosaiking process may attenuate high-frequency detail in the image and / or enhance low-frequency detail in the image.

[0019] In this example, color fringing may occur at the edges of sharp contrast boundaries in the image, in which case the edges of sharp contrast boundaries may contain fringing artifacts within the color pixel values ​​of the mosaic image. Furthermore, as will be described later, the edge enhancement logic 106 may include executable instructions that enable the medical system 100 to process a digital image (e.g., a mosaic image) by detecting edges and increasing the detail of the edges to provide sharper detail within the color pixel values ​​of the image.

[0020] Continuing to refer to Figure 1, the contrast enhancement logic 107 may include executable instructions that enable the medical system 100 to process a mosaic image of a target area and enhance the contrast of one or more pixel values ​​within the mosaic image. It should be understood that the demosaicing process may include inherent side effects, such as a decrease in image contrast due to a reduction in the color difference signal between pixel values ​​within the image.

[0021] In this example, the resolution frame of the color pixel values may lack sufficient brightness to differentiate one or more features of the image among the various color pixel values. As will be described further below, the contrast enhancement logic 107 may include executable instructions that enable the medical system 100 to process a digital image (e.g., a mosaic image) by scaling the luminance of specific color pixel values and increasing the brightness of the resolution frame to provide a clearer sharpness of the image within the color pixel values.

[0022] In some embodiments, the imaging logic 104, the demosaicking logic 105, the edge enhancement logic 106, and / or the contrast enhancement logic 107 may include executable instructions that enable the medical system 100 to automatically perform periodic image processing of the target site without requiring user input. In other embodiments, the image processing device 101 may be configured to receive user input, e.g., from the user interface 108 of the image processing device 101, to initiate image processing of the target site. It should be recognized that in some embodiments, the user interface 108 may be an apparatus integral with the image processing device 101, and in other embodiments, the user interface 108 may be a remote apparatus that communicates with the image processing device 101 (e.g., wirelessly, wired, etc.).

[0023] It should be understood that all or part of various programming algorithms and data for assisting the operation of the medical system 100 may be in the memory 103. The memory 103 may include any type of computer-readable medium suitable for storing data and algorithms, which may be, for example, random access memory (RAM), read-only memory (ROM), flash memory, hard drive, and / or any device capable of storing machine-readable instructions. The memory 103 may include one or more data sets, which may include, but are not limited to, image data from one or more components of the medical system 100 (e.g., the medical instrument 110, the medical device 140, etc.).

[0024] Continuing to refer to FIG. 1, the medical instrument 110 may be configured to facilitate positioning of one or more components of the medical system 100, such as the medical device 140, with respect to a subject (e.g., a patient). In an embodiment, the medical instrument 110 may be any type of endoscope, duodenoscope, gastroscope, colonoscope, ureteroscope, bronchoscope, catheter, or other delivery system, and may include a handle 112, an operating mechanism 114, at least one port 116, and a shaft 120. The handle 112 of the medical instrument 110 may have one or more lumens (not shown) that communicate with the lumens of one or more other components of the medical system 100. The handle 112 further includes at least one port 116 that opens into one or more lumens of the handle 112. As will be described in more detail herein, the at least one port 116 is sized and shaped to receive one or more instruments, such as the medical device 140 of the medical system 100, therein.

[0025] The shaft 120 of the medical device 110 may include a sufficiently flexible tube, and the shaft 120 is configured to selectively bend, rotate, and / or twist as it is inserted into and / or through the undulating internal structure of the subject to a target treatment site. The shaft 120 may have one or more lumens (not shown) extending through it, which include, for example, a working lumen for receiving an instrument (e.g., medical device 140). In other examples, the shaft 120 may include additional lumens such as a control wire lumen for receiving one or more control wires for operating one or more distal components / tools (e.g., joints, elevators, etc.), a fluid lumen for delivering fluid, an illumination lumen for receiving at least a portion of an illumination assembly (not shown), and / or an imaging lumen for receiving at least a portion of an imaging assembly (not shown).

[0026] Continuing to refer to Figure 1, the medical device 110 may further include a tip 122 at the distal end of the shaft 120. In some embodiments, the tip 122 may be attached to the distal end of the shaft 120, and in other embodiments, the tip 122 may be integrated with the shaft 120. For example, the tip 122 may include a cap configured to receive the distal end of the shaft 120. The tip 122 may include one or more openings communicating with one or more lumens of the shaft 120. For example, the tip 122 may include a work opening 123 through which the medical device 140 may exit the work lumen of the shaft 120. It should be noted that one or more other openings of the tip 122 on the shaft 120 are not shown. The operating mechanism 114 of the medical device 110 is located on the handle 112 and may include one or more knobs, buttons, levers, switches, and / or appropriate actuators thereof. The operating mechanism 114 is configured to control at least the orientation of the shaft 120 (e.g., through the operation of a control wire).

[0027] A medical device 140 of the medical system 100 may include a catheter having a longitudinal body 142 between a proximal end 141 and a distal end 144 of the medical device 140. The longitudinal body 142 of the medical device 140 may be flexible and thereby configured to bend, rotate, and / or twist when the medical device 140 is inserted into the working lumen of the medical instrument 110. The medical device 140 may include a handle at the proximal end 141 of the longitudinal body 142, which may be configured to move, rotate, and / or bend the longitudinal body 142. Furthermore, the handle at the proximal end 141 of the medical device 140 may define one or more ports (not shown) of a size that can receive one or more tools through the longitudinal body 142 of the medical device 140.

[0028] Continuing to refer to Figure 1, the medical device 110 may be configured to receive the medical device 140 from at least one port 116, through the working lumen, through the shaft 120, to the working port 123 of the tip 122. In this example, the medical device 140 may extend distally from the working port 123 to the surrounding environment of the tip 122, for example, at the target treatment site of the patient, as will be described in more detail later. The distal end 144 of the medical device 140 may extend distally from the tip 122 in response to the movement of the longitudinal body 142 within the working lumen of the shaft 120. The medical device 140 may include one or more end effectors (not shown) at the distal end 144 of the longitudinal body 142 for performing one or more actions on the target treatment site.

[0029] The medical device 110 may further be configured to receive one or more light sources 130 passing through the shaft 120 via at least one of the lumens of the medical device 110 so as to be connected to the optical fiber 146. In this example, the one or more light sources 130 are shown as separate components from the image processing device 101, thereby connecting the light sources 130 to the medical device 110 separately from the image processing device (e.g., via cables). It should be recognized that in other embodiments, the one or more light sources 130 may be included in the image processing device 101, thereby allowing the light sources 130 to be communicably connected to the medical device 110 together with the image processing device 101.

[0030] Next, referring to Figures 2A to 2C, the chip 122 of a medical device 110 according to one or more examples of the present disclosure is shown. First, referring to Figure 2A, in one embodiment, the chip 122 of the medical device 110 may include an optical fiber 146 and an image sensor 150 in the chip 122. In this example, the optical fiber 146 may be coupled to one or more light sources 130 of the medical system 100, so that each of the one or more light sources 130 can transmit light through one optical fiber 146. Not shown, but to be recognized, multiple light sources 130 may be coupled to the optical fiber 146 via a fiber splitter / combiner. The optical fiber 146 of the medical device 110 may be configured and capable of operating in such a way as to deliver light of various amplitudes from one or more light sources 130 distally from the chip 122 on the shaft 120. In some embodiments, the optical fiber 146 may be configured to deliver white light, ultraviolet light, near-infrared (NIR) light, and / or various other wavelengths in or beyond the visible spectrum.

[0031] Continuing to refer to Figure 2A, the image sensor 150 of the medical device 110 may be communicably connected to the image processing device 101 of the medical system 100, for example, via a wired connection and / or a wireless connection. The image sensor 150 of the medical device 110 may be configured and capable of operating to capture a raw image (e.g., a digital image) of the environment surrounding the tip 122 of the shaft 120. In some embodiments, the image sensor 150 may include an image sensor, such as an RGB (i.e., red-green-blue) digital sensor, an RGB-Ir (i.e., red-green-blue-infrared) digital sensor, and / or a monochrome sensor. As will be described in more detail herein, the image sensor 150 may include one or more components for filtering colors from white light, ultraviolet light, near-infrared light, and / or other wavelengths in or beyond the visible spectrum.

[0032] In another embodiment, referring to Figure 2B, the medical device 110 may include a multicolor LED assembly on the chip 122 of the shaft 120. In this example, the multicolor LED assembly may include one or more light-emitting diodes (hereinafter, LEDs) 146A, 146B, 146C, 146D arranged as an annular array around the image sensor 150. Each of the LEDs 146A, 146B, 146C, 146D may be configured and capable of transmitting different wavelengths and / or amplitudes of light (e.g., color) with respect to each other. It should be understood that different illuminating sources may produce different spectra. It should be recognized that the LEDs 146A, 146B, 146C, 146D of the medical device 110 may include more and / or fewer diodes on the chip 122 than those illustrated and described herein, and this also does not depart from the scope of the present disclosure.

[0033] In another embodiment, referring to Figure 2C, the medical device 110 may include a multisensor assembly at the tip 122 of the shaft 120. In this example, the multisensor assembly may include a color image sensor 150A and a monochrome image sensor 150B. As will be described in more detail herein, the color image sensor 150A may be configured and capable of operating to image the portion of the incident light corresponding to the color of the incident light at each of the individual pixel locations of the color image sensor 150A. In some embodiments, the color image sensor 150A may include, for example, an RGB (red-green-blue digital sensor) and an RGB-Ir (red-green-blue-infrared) digital sensor. As will be described in more detail herein, the monochrome image sensor 150B may be configured and capable of operating to completely image the entire incident light at each of the individual pixel locations of the monochrome sensor 150B, regardless of the color of the incident light.

[0034] Referring next to Figure 3, the image sensor 150 of the medical device 110 may include an outer surface 152 on which a plurality of microlenses 154 are arranged. In some examples, the outer surface 152 and / or the plurality of microlenses 154 may be made of glass, plastic, and / or other transparent material. The image sensor 150 may be a color image sensor including a color filter array 156 positioned relatively below the outer surface 152. The color filter array 156 may include an optical fiber device having a plurality of color pixel locations 158A, 158B, 158C arranged in a predetermined pattern. In this example, each of the plurality of microlenses 154 may be positioned in alignment with at least one of the plurality of color pixel locations 158A, 158B, 158C of the color filter array 156 positioned below the outer surface 152.

[0035] In some embodiments, the color filter array 156 may include a plurality of first color pixel locations 158A, a plurality of second color pixel locations 158B, and / or a plurality of third color pixel locations 158C. The plurality of color pixel locations 158A, 158B, and 158C may be arranged along the color filter array 156 in a mosaic pattern, such as a Bayer pattern. In this example, the plurality of first color pixel locations 158A may include red filters, the plurality of second color pixel locations 158B may include green filters, and the plurality of third color pixel locations 158C may include blue filters. In other embodiments, the plurality of color pixel locations 158A, 158B, and 158C may include various appropriate color filters and / or patterns other than those illustrated and described herein. For example, in embodiments in which the image sensor 150 includes an RGB-Ir sensor, the color filter array 156 should be understood to additionally include a plurality of fourth color pixel locations corresponding to infrared color filters.

[0036] Continuing to refer to Figure 3, the color filter array 156 of the image sensor 150 may be configured and capable of selectively transmitting one or more wavelengths 12, 14, 16 (e.g., light intensity, spectral band, color, etc.) of the light beam 10. For example, each of the color pixel locations 158A, 158B, 158C is configured to deliver and / or transmit through it a portion of the light beam 10 (e.g., at least one wavelength 12, 14, 16) corresponding to the color filter of the color pixel locations 158A, 158B, 158C. Thus, at each color pixel location 158A, 158B, 158C, only one color component (e.g., color pixel value) can be measured by the image sensor 150. As will be further described herein, the color pixel value may include the amount of energy in the colored range of the spectrum (e.g., red range, green range, blue range, etc.).

[0037] In this example, each of the multiple color pixel locations 158A, 158B, and 158C may allow only one wavelength 12, 14, and 16 of the light beam 10 to pass through the color filter array 156. The image sensor 150 may further include a photosensor array 160 positioned relatively below the color filter array 156, and the color filter array 156 of the image sensor 150 may be positioned between the outer surface 152 and the photosensor array 160. The photosensor array 160 of the image sensor 150 may include photodiodes (e.g., semiconductor devices) having multiple photosites 162 and a circuit configuration 164 communicably connected to the multiple photosites 162.

[0038] Continuing to refer to Figure 3, the multiple photosites 162 are arranged in an array (e.g., a grid), and each of the multiple photosites 162 is positioned in alignment with at least one of the multiple color pixel locations 158A, 158B, 158C of the color filter array 156 positioned above the photosensor array 160. The photosensor array 160 is configured and may be able to convert the light beam 10 received through the outer surface 152 and the color filter array 156 into an electric current. In this example, an electric current can be generated by the photosensor array 160 when photons from the received light are absorbed by the multiple photosites 162.

[0039] In this example, each of the multiple photosites 162 may be configured to measure the quantity of only one color pixel value (e.g., red, green, blue) in the incident light 10 at the location of the photosite 162 along the surface of the photosensor array 160. Thus, the multiple photosites 162 can image the incident light 10, generate an electrical signal which can be quantified and stored as a numerical value in the resulting processed image file. It should be recognized that the photosensor array 160 may include various appropriate shapes, sizes, and / or configurations other than those illustrated and described herein. In other embodiments, the image sensor 150 may be a monochrome sensor (e.g., monochrome sensor 150B), thereby the illustrated and described color filter array 156 may be completely excluded from between the outer surface 152 and the photosensor array 160. In this example, each photosite 162 along the photosensor array 160 may be operable to receive, image, and absorb all three wavelengths 12, 14, and 16 of the light beam 10.

[0040] Next, referring to Figures 3 and 4 in conjunction with the flowchart in Figure 5, an exemplary method 200 for generating processed images of a target site using a medical system 100 is schematically shown. The figures in Figures 3-5 and the following descriptions thereof are not intended to limit the subject matter described herein to any particular method.

[0041] First, with respect to Figure 1, the medical device 110 of the medical system 100 may be inserted into the body of a subject (not shown) and positioned so that the tip 122 is adjacent to the target site. For example, the shaft 120 may be guided in the digestive tract of a subject (e.g., a patient) by inserting the tip 122 into the subject's nose or mouth (or other suitable natural opening in the body) and guiding it through the gastrointestinal tract (e.g., esophagus, stomach, small intestine, etc.) within the subject's body to the target site. It should be recognized that the length of the shaft 120 may be sufficient so that the proximal end of the medical device 110 (including the handle 112) is outside the subject's body while the tip 122 of the medical device 110 is inside the subject's body. Although this disclosure relates to the use of the medical system 100 in the digestive tract of a subject, it should be understood that a feature of this disclosure is that it can be used in various other locations within the subject's body (e.g., other organs, tissues, etc.).

[0042] Additionally, once the medical device 110 has been received into the patient's body and the tip 122 of the shaft 120 is positioned relatively close to the target site, the medical device 140 may be received into the medical device 110 through at least one port 116. In this example, the longitudinal body 142 of the medical device 140 is moved within the shaft 120 through at least one of the lumens of the shaft 120 (e.g., the working lumen). The distal end 144 of the longitudinal body 142 may extend distally from the tip 122 of the shaft 120 through a work port 123 communicating with the working lumen of the shaft 120. It should be recognized that this step may be optional, and the receiving of the medical device 140 into the medical device 110 may be done in various other steps of Method 200 and / or may be omitted entirely. The tip 122 may be positioned adjacent to and opposite the target therapeutic site.

[0043] Referring to Figure 5, in step 202, in response to the processor 102 of the image processing device 101 executing imaging logic 104 to activate one or more light sources 130, one or more target objects can be illuminated by the medical device 110. In an example of the medical device 110 including an optical fiber 146 (Figure 2A) and / or multicolor LED assemblies 146A, 146B, 146C, 146D (Figure 2B), light from one or more light sources 130 can be emitted from the chip 122 of the medical device 110 to illuminate the target object.

[0044] In step 204, when the target object is illuminated by light from the medical device 110, the image sensor 150 may be activated by the processor 102 executing imaging logic 104 to capture one or more original digital images of the target object. It should be understood that the processor 102 of the image processing device 101 may be communicably connected to the image sensor 150 of the medical device 110 via circuit configuration 164. For example, referring again to Figure 3, light 10 transmitted to the target object by the optical fiber 146 and / or multicolor LED assemblies 146A, 146B, 146C, 146D may be reflected by the target object and received by the image sensor 150. In this example, several wavelengths 12, 14, 16 of the light 10 may be received through one or more of several microlenses 154 on the outer surface 152.

[0045] Multiple wavelengths 12, 14, and 16 can be received at one or more corresponding color pixel locations 158A, 158B, and 158C of the color filter array 156, for example, in a manner that matches the microlens 154 that receives the light beam 10 through them. At each of the multiple color pixel locations 158A, 158B, and 158C, one or more of the multiple wavelengths 12, 14, and 16 of the light 10 can be prevented from passing through the color filter array 156 (e.g., filtered, excluded, rejected, or blocked) depending on the color filter of the color pixel locations 158A, 158B, and 158C. Therefore, the color filter configuration (e.g., red, green, blue, etc.) of the color pixel locations 158A, 158B, and 158C that receive the light 10 determines which of the wavelengths 12, 14, and 16 (e.g., red, blue, green, etc.) can pass through the color filter array 156 at those locations.

[0046] Therefore, it should be understood that each of the color pixel locations 158A, 158B, and 158C may be configured to allow only about one-third (e.g., 33%) of the incident light 10 to pass through the photosensor array 160 at that location. For example, in each of the plurality of first color pixel locations 158A (e.g., red filters), only wavelength 12 (e.g., the red range of the light spectrum) of the light 10 can pass through the color filter array 156, thereby allowing wavelengths 14 and 16 (e.g., blue and green, respectively) to be removed by the color filter array 156 at the first color pixel location 158A. It should be recognized that in this embodiment, each of the plurality of second color pixel locations 158B (e.g., green filters) may be configured to allow wavelength 14 (e.g., the green range of the light spectrum) to pass through, and each of the plurality of third color pixel locations 158C (e.g., blue filters) may be configured to allow wavelength 16 (e.g., the blue range of the light spectrum) to pass through.

[0047] Continuing to refer to Figure 3, the individual wavelengths 12, 14, and 16 of light 10 that pass through the color filter array 156 are detected along the photosensor array 160 and can be absorbed by one or more of the photosites 162 (i.e., those that match the color pixel locations 158A, 158B, and 158C that receive the light beam 10 passing through them). In this example, the portion of light 10 absorbed by each of the photosites 162 can be converted into an electric current. The raw digital image captured by the image sensor 150 may include a quantitative record of the light energy measured at each grid location of the photosites 162 along the photosensor array 160, with each photosite 162 configured to identify the color pixel values ​​of the wavelengths 12, 14, and 16 received thereon.

[0048] In this example, when the processor 102 of the image processing device 101 executes the imaging logic 104, it may cause the photosensor array 160 to transmit electrical signals of color pixel values ​​to the image processing device 101, for example, via the circuit configuration 164. The electrical signals of color pixel values ​​are stored in the memory 103 of the image processing device 101 and may be used to generate a processed image by the demosaicing logic 105, edge enhancement logic 106, and / or contrast enhancement logic 107.

[0049] Referring to Figure 5, in step 206, the wavelength shift input may be input by the user of the medical system 100, for example, via the user interface 108 of the image processing device 101. The wavelength shift input may include identification information, amplitude, and / or color pixel values, etc., for one or more optical wavelengths to adjust the demosaicing process of the original digital image captured by the image sensor 150. It should be noted that in some embodiments, step 206 may be performed before steps 202 and 204 and / or automatically pre-programmed in the memory 103 of the image processing device 101, so that method 200 can proceed to step 208 with the wavelength shift input already identified.

[0050] Next, referring to Figure 4, the original digital image received by the image processing device 101 may include a frame 20 (e.g., a grid) of original pixel values ​​formed by a plurality of color pixel values ​​22A, 22B, 22C measured by the image sensor 150. It should be understood that each grid location along the frame 20 of original pixel values ​​may correspond to the location of a photosite 162 on the photosensor array 160. Therefore, each grid location on the frame 20 of original pixel values ​​includes at least one of a first color pixel value 22A, a second color pixel value 22B, or a third color pixel value 22C, based on the relative position of the grid location with respect to color pixel locations 158A, 158B, 158C of the color filter array 156 through which the photosite 162 receives wavelengths 12, 14, 16.

[0051] In step 208, the processor 102 separates the frame 20 of original pixel values ​​by color pixel values ​​22A, 22B, and 22C, and may filter (e.g., exclude) at least one of the first color pixel value 22A, the second color pixel value 22B, or the third color pixel value 22C based on the wavelength shift input received in step 206. In other words, the image processing device 101 is configured to shift the demosaicing process of the original digital image captured by the image sensor 150 from a plurality of color pixel values ​​22A, 22B, and 22C in the frame 20 of original pixel values ​​to a small portion of the color pixel values ​​22A, 22B, and 22C.

[0052] Therefore, as will be explained in detail later, the demosaicing process (steps 210A, 210B), edge enhancement process (steps 212A, 212B), and / or contrast enhancement process (steps 214A, 214B) of the original digital image are performed only on a portion of the frame 20 of original pixel values ​​contained in the original digital image captured by the image sensor 150. For example, the processor 102 of the image processing device 101 removes a plurality of first color pixel values ​​22A according to the wavelength shift input, thereby so that the resulting processed image produced by the image processing device 101 includes partially resolved frames of pixels.

[0053] For example, continuing to refer to Figure 5, in step 210A, the processor 102 performs a demosaiking process of multiple second color pixel values ​​22B (e.g., green) to calculate estimates of second color pixel values ​​22B (e.g., green) at grid locations along the frame 20 of the original pixel values ​​that did not receive wavelength 14 (e.g., green), such as grid locations that received wavelength 12 (e.g., red) and wavelength 16 (e.g., blue). The processor 102 may estimate the amount of unknown second color pixel values ​​22B at grid locations that did not receive wavelength 14 on the frame 20 of the original pixel values ​​by interpolation from known measurements of second color pixel values ​​22B.

[0054] In this example, the processor 102, which executes the demosaicing logic 105, interpolates the missing second color pixel value 22B from an adjacent (e.g., neighboring) grid location containing the measured second color pixel value 22B. The processor 102 determines the amount of the unknown second color pixel value 22B from the neighboring grid location that received the wavelength 14. It should be understood that in step 210B, the processor 102 may execute the demosaicing logic 105 in substantially the same manner as described above with respect to step 210A, and reconstruct the unknown and / or missing third color pixel value 22C along the frame 20 of the original pixel value.

[0055] Continuing to refer to Figure 5, in step 212A, the processor 102 of the image processing apparatus 101 may execute edge enhancement logic 106 to further reconstruct the original digital image captured by the image sensor 150. In this example, the edge enhancement logic 106, when executed by the processor 102, may enhance the detail and / or sharpness of one or more edges in the frame 20 of the original pixel values. For example, the processor 102 may detect the location of one or more edges in each grid location in the frame 20 of the original pixel values ​​and minimize the noise level around those edges by adjusting one or more of the multiple second color pixel values ​​22B in that grid location. In some embodiments, the processor 102 may improve the sharpness of one or more edges in a grid location by increasing the gradient (e.g., magnitude) of the multiple second color pixel values ​​22B.

[0056] It should be understood that in step 212B, the processor 102 may reconstruct the edges of the digital image by performing edge enhancement logic 106 in substantially the same manner as described above with respect to step 212A, and by adjusting one or more of the third color pixel values ​​22C along the frame 20 of the original pixel values. It should be recognized that in other embodiments, various other suitable edge enhancement processes may be included in the edge enhancement logic 106 and performed by the processor 102.

[0057] Continuing to refer to Figure 5, in step 214A, the processor 102 of the image processing apparatus 101 may execute contrast enhancement logic 107 to further reconstruct the original digital image captured by the image sensor 150. In this example, the contrast enhancement logic 107, when executed by the processor 102, may enhance the contrast of the representation of the original pixel values ​​in frame 20. For example, the processor 102 may increase one or more brightness components (e.g., luminance) of a plurality of second color pixel values ​​22B within each grid location in frame 20 of the original pixel values. In some embodiments, the processor 102 may adjust the brightness of one or more grid locations in frame 20 of the original pixel values ​​by reducing the second color pixel values ​​22B within them, thereby reducing the contrast contribution provided by the second color pixel values ​​22B.

[0058] It should be understood that in step 214B, the processor 102 may enhance the local contrast of the digital image by performing the contrast enhancement logic 107 in substantially the same manner as described above with respect to step 214A, and by adjusting one or more of the third color pixel values ​​22C along the frame 20 of the original pixel values. It should be recognized that in other embodiments, various other suitable contrast enhancement processes may be included in the contrast enhancement logic 107 and performed by the processor 102.

[0059] Continuing to refer to Figure 5, in step 216, the processor 102 of the image processing apparatus 101 may determine whether the current iteration of the demosaicing process (steps 210A, 210B), the edge enhancement process (steps 212A, 212B), and the contrast enhancement process (steps 214A, 214B) is equal to or greater than a predetermined and / or dynamic iteration threshold. In some embodiments, the predetermined iteration threshold may be predetermined and stored in the memory 103 of the image processing apparatus 101, or may be selectively entered by the user of the medical system 100.

[0060] In other embodiments, the iteration threshold may be dynamically determined by the processor 102 based on one or more factors, including, for example, peak detail, contrast, and visibility from the initial transformation of the digital image through demosaiking, edge enhancement, and contrast enhancement processes. In this example, data showing the initial state of the original pixel values ​​20 (e.g., a histogram of the frame) may be analyzed by the processor 102 at the completion of the first iteration, and peak values ​​for detail, contrast, and visibility characteristics may be identified. Thus, the processor 102 can continuously determine the current detail, contrast, and visibility of the digital image with respect to the initial peak values ​​(i.e., the dynamic iteration threshold) at the completion of each iteration of the process.

[0061] In response to the processor 102 identifying in step 216 that the current iteration of method 200 is below a predetermined (or dynamic) threshold, the image processing apparatus 101 may be configured and capable of operating in such a way as to return to steps 210A, 210B and perform one or more of the demosaiking, edge enhancement, and / or contrast enhancement processes. In response to the processor 102 identifying in step 216 that the current iteration of method 200 is at least equal to or greater than a predetermined (or dynamic) threshold, the image processing apparatus 101 may be configured and capable of operating in such a way as to produce an interpolated output image. It should be recognized that, in response to the iterative enhancement of the frame 20 of the original pixel values ​​initially captured by the image sensor 150, an image with improved detail, contrast, and / or visibility may be provided.

[0062] Continuing to refer to Figure 5, in step 218, the processor 102 of the image processing apparatus 101 may generate a processed image obtained from the process of method 200 illustrated and described herein. In this example, the output image may include partial-resolution frames of color pixel values, for example, a plurality of second color pixel values ​​22B and a plurality of third color pixel values ​​22C. Thus, in the digital image generated by the processor 102 of the image processing apparatus 101, at least a plurality of first color pixel values ​​22A may be omitted from one or more grid locations of the frame 20 of the original pixel values ​​initially captured by the image sensor 150. In this example, the first color pixel values ​​22A are excluded from each grid location of the frame 20 of the original pixel values ​​so that the image processing apparatus 101 generates a partial-resolution image from the generated image.

[0063] With the display 109 of the medical system 100 communicably connected to the processor 102 of the image processing device 101, the processor 102 may be capable of transmitting partially resolved images to the display 109 so that the user of the medical system 100 can see them. In some examples, the medical system 100 may be configured and capable of operating in a manner that continuously performs the method 200 illustrated and described herein, thereby allowing the display 109 to output multiple partially resolved images to provide continuous (e.g., live, real-time, etc.) imaging of one or more target objects.

[0064] It should be recognized that by excluding at least one of the multiple color pixel values ​​22A, 22B, and 22C initially captured (by the image sensor 150) within frame 20 of the original pixel values ​​from the processed image, it may be easier to differentiate one or more features and / or structures (e.g., target objects) of the target treatment site. For example, by filtering out at least one of the color pixel values ​​22A, 22B, and 22C according to the wavelength shift input in step 206, blood vessels may be better distinguished from surrounding tissue. In some examples, the primary colors of one or more target objects may be substantially similar to each other, for example, tissue and blood vessels are generally displayed in red, making it difficult to distinguish the features in the generated image. In this example, by emphasizing the blue and / or green color pixel values ​​(e.g., the second color pixel value 22B and the third color pixel value 22C, respectively), the features may be better distinguishable.

[0065] Additionally, it should be understood that the contrast of one or more color components (e.g., red) may be lower than that of one or more other color components (e.g., green, blue) within the target treatment site. The subject's blood vessels can be highlighted by excluding one or more color components, for example, the first color pixel value 22A (e.g., red), from the resulting digital image, because only the second color pixel value 22B (e.g., blue) and the third color pixel value 22C (e.g., green) are retained from the original pixel values ​​in frame 20.

[0066] In an embodiment in which the optical fiber 146 of the medical device 140 is operable to generate ultraviolet light, the medical system 100 may be configured to generate a partially resolved image frame that can distinguish desirable (e.g., healthy) tissue from undesirable (e.g., unhealthy) tissue, since the tissue may fluoresce in different colors under ultraviolet light. In this example, the wavelength shift input in step 206 may include a first color pixel value 22A (e.g., red) and a third color pixel value 22C (e.g., green), so that the resulting processed image may contain only the second color pixel value 22B (e.g., blue).

[0067] In an embodiment in which the image sensor 150 of the medical device 110 includes an RGB-Ir sensor, the frame 20 of the original pixel values ​​detected and captured by the image sensor may include a plurality of first color pixel values ​​22A (e.g., red), a plurality of second color pixel values ​​22B (e.g., blue), a plurality of third color pixel values ​​22C (e.g., green), and a plurality of fourth color pixel values ​​(e.g., infrared) that are not included in the visible spectrum and are closer to the infrared spectrum. In this example, the medical system 100 may be configured to generate a partially resolved image frame in which relatively dark areas within the target treatment site (e.g., a body lumen) can be made more visible by highlighting the first color pixel values ​​22A, the third color pixel values ​​22C, and / or the fourth color pixel values. Accordingly, the wavelength shift input in step 206 may include a second color pixel value 22B (e.g., blue) to be excluded from the processed image.

[0068] In an embodiment in which the tip 122 of the shaft 120 includes a color image sensor 150A (e.g., an RGB-Ir sensor) and a monochrome image sensor 150B, the image processing device 101 may be configured to generate a partially resolved image captured by the color image sensor 150A, while simultaneously using the monochrome image sensor 150B to further enhance the contrast of relatively dark areas within the frame 20 of the original pixel values. In this example, since the monochrome image sensor 150B can sense near-infrared wavelengths irradiated in light 10, a fourth color pixel value (e.g., infrared) can be easily detected. It should be noted that providing one or more materials, such as fluorescent dyes, within the target treatment site can facilitate the visualization of one or more target objects by the monochrome image sensor 150B.

[0069] (Note) As a preferred embodiment, the technical concept that can be understood from the above embodiment is described below. [Item 1] It is a medical device, The medical device is equipped with a shaft, The medical device comprises a sensor connected to the distal end of the shaft and including a filter array, the sensor configured to capture an original image, and the filter array configured to filter the original image into a frame of original pixel values, the frame of original pixel values ​​including a plurality of first pixel values, a plurality of second pixel values, and a plurality of third pixel values. The medical device comprises a processor and a non-temporary computer-readable medium for storing demosaicing instructions, and when the demosaicing instructions are executed by the processor, the processor receives The plurality of first pixel values ​​are removed from the frame of the original pixel values. Generate a plurality of second predicted pixel values ​​at the locations of the plurality of excluded first pixel values ​​and the plurality of third pixel values ​​on the frame, Generate a plurality of third predicted pixel values ​​at the locations of the plurality of excluded first pixel values ​​and the plurality of second pixel values ​​on the frame, A processed image having a partially resolved image frame is generated from the plurality of second pixel values, the plurality of second predicted pixel values, the plurality of third pixel values, and the plurality of third predicted pixel values. Medical device. [Item 2] The demosaicing instructions stored in the non-temporary computer-readable medium are sent to the processor, To detect one or more edges in the original image, The medical device according to item 1, which causes one or more edges to perform sharpening enhancement in order to increase the edge detail of the processed image. [Item 3] The demosaicing instructions stored in the non-temporary computer-readable medium are sent to the processor, The sharpened enhancement image created by executing the sharpening enhancement step is output. A medical device according to item 2, which fuses the sharpened enhancement image with the processed image. [Item 4] The demosaicing instructions stored in the non-temporary computer-readable medium are sent to the processor, The brightness values ​​of each of the plurality of second pixel values ​​and the plurality of third pixel values ​​are set. The medical device according to item 3, which performs contrast enhancement between the plurality of second pixel values ​​and the plurality of third pixel values ​​by adjusting the brightness values ​​to increase the contrast of the processed image. [Item 5] The demosaicing instructions stored in the non-temporary computer-readable medium are sent to the processor, The contrast-enhanced image created from the execution of the aforementioned contrast enhancement is output. A medical device according to item 4, which fuses the contrast enhancement image with the processed image. [Item 6] The demosaicing instructions stored in the non-temporary computer-readable medium are sent to the processor, A medical device as described in item 5, which repeats all of the above steps up to a threshold. [Item 7] The demosaicing instructions stored in the non-temporary computer-readable medium are sent to the processor, A medical device according to any one of items 1 to 6, which receives a wavelength shift input in order to identify the color pixel values ​​of the plurality of first pixel values. [Item 8] The medical device according to item 7, further comprising a user interface configured to be communicatively connected to the processor and to transmit signals indicating the wavelength shift input to the processor. [Item 9] The sensor includes an RGB image sensor, and the filter array includes a red-green-blue Bayer color filter array. A medical device according to any one of items 1 to 8, wherein the plurality of first pixel values ​​include red pixel values, the plurality of second pixel values ​​include blue pixel values, and the plurality of third pixel values ​​include green pixel values. [Item 10] The medical device according to any one of items 1 to 9, wherein the sensor includes an RGB+Ir image sensor and the filter array includes a red-green-blue-infrared Bayer color filter array. [Item 11] The sensor includes an RGB+Ir image sensor, and the filter array includes a red-green-blue-infrared Bayer color filter array. A medical device according to any one of items 1 to 8, wherein the plurality of first pixel values ​​include blue pixel values, the plurality of second pixel values ​​include red pixel values ​​and green pixel values, and the plurality of third pixel values ​​include infrared pixel values. [Item 12] The aforementioned sensor is a medical device according to any one of items 1 to 11, including an RGB image sensor and a monochrome sensor. [Item 13] A medical device according to any one of items 1 to 12, wherein each location within the partially resolved image frame includes one captured color pixel value and one reconstructed color pixel value such that at least one color pixel value is excluded from the frame of the original pixel values. [Item 14] A medical device according to any one of items 1 to 13, further comprising a light source connected to the distal end of the shaft, wherein the light source is an optical fiber or a multicolor LED array. [Item 15] The demosaicing instructions stored in the non-temporary computer-readable medium are sent to the processor, A medical device according to any one of items 1 to 14, which outputs the processed image of the partially decomposed image frame to a display device. Each of the systems, apparatuses, assemblies, and methods described above may be used to generate partial-resolution frames of the subject's pixel values. By providing a medical device that includes an image processing system storing wavelength-shift demosaicing logic, the user can better visualize one or more features and / or characteristics of a target site within the subject's body during treatment without manipulating the light source. This medical device may enable the user to accurately locate the target site, thereby reducing overall treatment time, improving treatment efficiency, and avoiding unnecessary harm to the subject's body that may occur due to inaccurate locating of the target object at the target treatment site.

[0070] It will be apparent to those skilled in the art that various improvements and modifications can be made to the disclosed apparatus and methods without departing from the scope of this disclosure. It should be recognized that the disclosed apparatus may include various suitable computer systems and / or computing units containing multiple hardware components, such as a processor, and a non-transient computer-readable medium that enables the apparatus to perform one or more operations during the procedure in accordance with those described herein. Other aspects of this disclosure will also become apparent to those skilled in the art from the considerations herein and the practice of the features disclosed therein. This specification and the examples are to be considered illustrative only.

Claims

1. A method for processing an image, performed by one or more processors communicably connected to a medical device sensor, This method includes receiving an image frame captured by the medical device sensor from the medical device sensor, wherein the image frame includes a plurality of first pixel values ​​in a plurality of first locations, a plurality of second pixel values ​​in a plurality of second locations, and a plurality of third pixel values ​​in a plurality of third locations. This method comprises removing the plurality of first pixel values ​​from the image frame, This method comprises determining a plurality of second predicted pixel values ​​in the plurality of first locations and the plurality of third locations on the image frame, This method comprises determining a plurality of predicted pixel values ​​for a plurality of first locations and a plurality of predicted third locations on the image frame, The method comprises generating a processed image frame based on a plurality of second pixel values, a plurality of second predicted pixel values, a plurality of third pixel values, and a plurality of third predicted pixel values.

2. Receiving a wavelength shift input, The method according to claim 1, further comprising excluding the plurality of first pixel values ​​from the image frame based on a received wavelength shift input and a first color associated with the plurality of first pixel values ​​corresponding to the wavelength shift input.

3. The method according to claim 2, wherein the image frame is of a target area, and the target area includes at least a first and a second feature having a color substantially similar to the first color.

4. Determining the plurality of second predicted pixel values ​​in the plurality of first locations and the plurality of third locations on the image frame is: In order to determine the corresponding second expected pixel value in each of the plurality of first locations, interpolation is performed from one or more of the plurality of second pixel values ​​in one or more of the plurality of second locations adjacent to each of the plurality of first locations. In order to determine the corresponding second expected pixel value in each of the plurality of third locations, interpolation is performed from one or more of the plurality of second pixel values ​​in one or more of the plurality of second locations adjacent to each of the plurality of third locations. The method according to claim 1, comprising:

5. Determining the plurality of predicted pixel values ​​for the plurality of first locations and the plurality of second locations on the image frame is: In order to determine the corresponding third expected pixel value in each of the plurality of first locations, interpolation is performed from one or more of the plurality of third pixel values ​​in one or more of the plurality of third locations adjacent to each of the plurality of first locations, In order to determine the corresponding third expected pixel value in each of the plurality of second locations, interpolation is performed from one or more of the plurality of third pixel values ​​in one or more of the plurality of third locations adjacent to each of the plurality of second locations. The method according to claim 1, comprising:

6. To generate a sharpened image frame, the sharpness of one or more edges detected in one or more of the plurality of second locations or the plurality of third locations is increased, The sharpened image frame is merged with the processed image frame. The method according to claim 1, further comprising:

7. To generate a contrast-enhanced image frame, the luminance component of one or more of the aforementioned second pixel values ​​or one or more of the aforementioned third pixel values ​​is adjusted. The contrast-enhanced image frame is merged with the processed image frame. The method according to claim 1, further comprising:

8. To generate a sharpened image frame from the aforementioned image frame, The process involves generating a contrast-enhanced image frame from the aforementioned image frame, A partially resolved image frame is generated based on the processed image frame, the sharpened image frame, and the contrast-enhanced image frame. The method according to claim 1, further comprising:

9. Based on one or more of the aforementioned image frame, the processed image frame, the sharpened image frame, or the contrast-enhanced image frame, it is determined that the current iteration of image processing has reached a threshold. To generate the partial-resolved image frame in response to the above decision. The method according to claim 8, further comprising:

10. The method according to claim 1, wherein the plurality of first pixel values ​​correspond to a first color, the plurality of second pixel values ​​correspond to a second color different from the first color, and the plurality of third pixel values ​​correspond to a third color different from the first color and the second color.

11. The method according to claim 10, wherein the medical device sensor includes a Bayer color filter array, and the first color, the second color, and the third color correspond to red, green, or blue.

12. A method for processing an image, which is performed by one or more processors communicably connected to a medical device sensor, wherein the method is: The medical device sensor receives an image captured by the medical device sensor, which includes a plurality of pixels having a plurality of captured pixel values ​​at a plurality of locations. Removing from the image a first small portion of the multiple captured pixel values ​​for a first small portion of a plurality of pixels in a first small portion of the plurality of locations, Estimating multiple unknown pixel values ​​for the remaining second portion of the multiple pixels in the first portion of the multiple locations, A processed image is generated using the second portion of the multiple captured pixel values ​​for the remaining second portion of the multiple pixels in the first portion of the multiple locations, and the multiple unknown pixel values ​​estimated for the remaining second portion of the multiple pixels in the first portion of the multiple locations. A method that includes [a certain feature].

13. The method according to claim 12, wherein the first sub-part of the plurality of pixels includes a first color pixel in the first sub-part of the plurality of locations in the image, and the second sub-part of the plurality of pixels includes a second color pixel in the second sub-part of the plurality of locations in the image and a third color pixel in the third sub-part of the plurality of locations in the image.

14. To estimate multiple unknown pixel values ​​for the second color pixels in the third small portion of the multiple locations in the image, To estimate multiple unknown pixel values ​​for the third color pixels in the second small portion of the multiple locations in the image, Further generating the processed image based on the multiple unknown pixel values ​​estimated for the second color pixels in the third small portion of the multiple locations in the image, and the multiple unknown pixel values ​​estimated for the third color pixels in the second small portion of the multiple locations in the image. The method according to claim 13, further comprising:

15. The method according to claim 13, wherein the medical device sensor includes a Bayer color filter array, the first color pixel, the second color pixel, and the third color pixel correspond to one of red, green, or blue.

16. To generate one or more sharpened images or contrast-enhanced images from the aforementioned image, A partially resolved image is generated based on the processed image and one or more of the sharpened image or the contrast-enhanced image. The method according to claim 12, further comprising:

17. Generating the sharpened image from the aforementioned image is, The method according to claim 16, comprising increasing the sharpness of one or more edges detected in one or more of the plurality of locations relating to the remaining second small portion of the plurality of pixels in order to generate a sharpened image frame.

18. Generating the contrast enhancement image from the aforementioned image is The method according to claim 16, further comprising adjusting the luminance component of one or more of the remaining second small portions of the plurality of pixels in order to generate a contrast-enhanced image frame.

19. A method for processing an image, which is performed by one or more processors communicably connected to a medical device sensor, wherein the method is: The medical device sensor receives an image captured by the medical device sensor, which includes multiple captured pixel values ​​for multiple pixels. From the aforementioned image, a filter is used to remove the first small portion of the multiple captured pixel values ​​for a first small portion of the multiple pixels, To generate a plurality of predicted pixel values ​​for the remaining second portion of the plurality of pixels in a plurality of locations where the first portion of the plurality of captured pixel values ​​has been removed by the filter, A processed image is generated using the second small portion of the multiple captured pixel values ​​and the multiple predicted pixel values ​​for the remaining second small portion of the multiple pixels. The first small portion of the plurality of captured pixel values ​​that were removed by the filter remains removed from the processed image. A method that includes [a certain feature].

20. To generate a sharpened image from the aforementioned image, To generate a contrast-enhanced image from the aforementioned image, A partially resolved image is generated based on the processed image, the sharpened image, and the contrast-enhanced image. The method according to claim 19, further comprising:

Images