Image processing system and method of using the same

Abstract

A medical device includes a sensor including a filter array. The sensor is configured to capture an original image, and the filter array is configured to filter the original image to obtain a frame of original pixel values ​​including first, second, and third pixel values. The medical device includes a non-transitory computer-readable medium storing demosaicing instructions that, when executed, cause a processor to exclude a first pixel value from the frame of original pixel values. The processor generates second predicted pixel values ​​at locations on the frame of the excluded first and third pixel values ​​and generates third predicted pixel values ​​at locations on the frame of the excluded first and second pixel values. The processor generates a processed image having a partially resolved image frame from the second pixel value, the second predicted pixel value, the third pixel value, and the third predicted pixel value.

Description

Technical Field

[0001] Various aspects of the present disclosure generally relate to image processing systems, devices, and related methods. Examples of the present disclosure relate to systems, devices, and related methods for digital dye endoscopy and the like.

Background Art

[0002] With technological developments, users of medical systems, devices, and methods are increasingly able to perform more complex procedures on subjects. One problem in the technical field of minimally invasive surgery involves visualizing a target treatment site within a subject's body, such as a tumor or lesion within the subject's gastrointestinal tract. Dye endoscopy with dye injection can make it easier to detect changes on the mucosal surface within the gastrointestinal tract, which is a lumen. However, due to limited imaging methods and devices for visualizing blood vessels, procedures can be prolonged, their effectiveness limited, and / or harm can befall the patient.

Summary of the Invention

[0003] Aspects of the present disclosure relate to systems, devices, and methods for providing, among other things, an image processing system, and relate to, among other things, wavelength shift demosaicking logic. Each of the aspects disclosed herein may include one or more of the features described with respect to any of the other disclosed aspects.

[0004] According to an example, a medical device includes a shaft and a sensor coupled to a distal end of the shaft and including a filter array. The sensor is configured to image an original image, and the filter array is configured to filter the original image to obtain a frame of original pixel values including a plurality of first pixel values, a plurality of second pixel values, and a plurality of third pixel values. The medical device includes a processor and a non-transitory computer-readable medium that stores, when executed by the processor, a demosaicking instruction that causes the processor to exclude a plurality of first pixel values from the frame of original pixel values. The processor generates a plurality of second predicted pixel values at locations of the plurality of excluded first pixel values and the plurality of third pixel values on the frame. The processor generates a plurality of third predicted pixel values at locations of the plurality of excluded first pixel values and the plurality of second pixel values on the frame. The processor generates a processed image having a partial resolution frame from the plurality of second pixel values, the plurality of second predicted pixel values, the plurality of third pixel values, and the plurality of third predicted pixel values.

[0005] Any of the medical devices described herein may include any of the following features. The demosaicking instruction stored in the non-transitory computer-readable medium causes the processor to detect one or more edges in the original image and perform sharpened enhancement of the one or more edges to increase edge detail in the processed image. The demosaicking instruction stored in the non-transitory computer-readable medium causes the processor to output a sharpened enhancement image created from performing the sharpened enhancement step and fuse the sharpened enhancement image with the processed image. The demosaicking instruction stored in the non-transitory computer-readable medium causes the processor to set the luminance value of each of the plurality of second pixels Value and the plurality of third pixels Value and adjust the luminance value to increase the contrast of the processed image, whereby the plurality of second pixels Value and the plurality of third pixels ValueExecute contrast enhancement. The demosaicking instructions stored in the non-transitory computer-readable medium cause the processor to output a contrast-enhanced image created from the execution of the contrast enhancement step and fuse the contrast-enhanced image with the processed image. The demosaicking instructions stored in the non-transitory computer-readable medium cause the processor to repeat all of the above steps up to a threshold value. The demosaicking instructions stored in the non-transitory computer-readable medium cause the processor to receive a wavelength shift input to identify the color pixel values of a plurality of first pixels Value and further include a user interface communicably coupled to the processor and configured to transmit a signal indicating the wavelength shift input to the processor. The sensor includes an RGB image sensor, and the filter array includes a red-green-blue Bayer color filter array. The plurality of first pixels Value include red pixels Value , the plurality of second pixels Value include blue pixels Value , and the plurality of third pixels Value include green pixels Value . The sensor includes an RGB+Ir image sensor, and the filter array includes a red-green-blue-infrared Bayer color filter array. The plurality of first pixels Value include blue pixels Value , the plurality of second pixels Value include red pixels Value and green pixels Value , and the plurality of third pixels Value include infrared pixels Value . The sensor includes an RGB image sensor and a monochrome sensor Part Each location within the resolution frame includes one captured color pixel value and one reconstructed color pixel value, thereby the original pixel ValueAt least one color pixel value is excluded from the frame. Further, it includes a light source connected to the distal end of the shaft, and the light source is an optical fiber, ultraviolet light, or a multi-color LED array. The demosaicking instructions stored in the non-transitory computer-readable medium cause the processor to output the processed image of the sub-resolution frame to the display device.

[0006] According to another example, the image processing method includes the step of capturing an original image, and filtering the original image to obtain a frame of original pixels including a plurality of first pixels Value , a plurality of second pixels Value , and a plurality of third pixels Value . The method includes the step of excluding at least a plurality of first pixels Value , and generating missing second pixels Value at the pixel locations of the plurality of excluded first pixels Value along the frame and the plurality of third pixels Value . The method includes the step of generating missing third pixels Value at the pixel locations of the plurality of excluded first pixels Value along the frame and the plurality of second pixels Value , and constructing a digitally sampled image partially from the plurality of second pixels Value , the plurality of third pixels Value , the generated second pixels Value , and the generated third pixels Value Value .

[0007] Any of the methods described herein may include any of the following steps Part detecting edges within the frame, and enhancing the sharpness of the edges to increase edge details in the partially sampled digital image. A method including a plurality of second pixels Value in the partially sampled digital image and a plurality of third pixels ValueTo enhance the contrast, a plurality of second pixels Value and a plurality of third pixels Value A method including the step of changing the luminance value for each of. Receiving a wavelength shift input to identify the color pixel values of a plurality of first pixels Value A method including the step of. The original pixel Value The frame of further includes a plurality of fourth pixels Value A method including the step of excluding at least a plurality of fourth pixels Value

[0008] According to other examples, a processor, when executed by the processor, causes the processor to transmit light having a plurality of wavelengths from an illumination source, and at least a small portion of a plurality of pixel values detected by a digital image sensor communicatively coupled to the processor is removed. The digital image sensor includes a filter array configured to filter a plurality of wavelengths of light and obtain a plurality of pixel values. The processor generates a plurality of predicted pixel values, creates a processed image having a partial resolution frame including the plurality of pixel values and the plurality of predicted pixel values, and removes at least a small portion of the plurality of pixel values from the processed image.

[0009] It should be understood that both the above general description and the following detailed description are merely illustrative and explanatory and do not limit the invention according to the claims. The accompanying drawings incorporated herein and constituting a part thereof illustrate exemplary aspects of the present disclosure and serve to explain the principles of the present disclosure together with the description.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 3

Figure 4

Figure 5

[0011] Examples of the present disclosure include systems, devices, and methods for enhancing images of one or more target treatment sites within a subject (e.g., patient) by enhancing one or more features of the target sites (e.g., blood vessels, vasculature, etc.) in the processed image. Aspects of the present disclosure are described in detail below, examples of which are illustrated in the accompanying drawings. Wherever possible, the same or similar reference numerals are used throughout the drawings to refer to the same or similar parts. The term "distal" refers to the portion of the device that is furthest from the user when the device is introduced into the patient's body. In contrast, the term "proximal" refers to the portion of the device that is closest to the user when the device is placed in the subject's body. As used herein, the terms "comprises," "comprising," or any other variants thereof are intended to cover a non-exclusive inclusion, and a process, method, article, or device that includes recited elements does not necessarily include only those elements, but may include other elements not expressly recited or inherent to such process, method, article, or device. The term "exemplary" is used in the sense of "example" rather than "ideal." As used herein, the terms "about," "substantially," and "approximately" indicate a range of values ​​of + / - 10% of the stated value.

[0012] Examples of the present disclosure can be used to identify a target site within a subject's body by generating a processed image having a partial-resolution frame of pixel values that visually emphasizes one or more features and / or characteristics of the gastrointestinal tract, which is a lumen of the subject. Such features and / or characteristics include, for example, tumors, lesions, blood vessels, changes in the mucosal surface, and the like. In some embodiments, the medical device can include an image processing device, which includes a processor and a memory storing one or more algorithms for generating the partial-resolution frame. In an embodiment, the memory can include programmable instructions according to demosaicking logic, edge enhancement logic, and / or contrast enhancement logic. Further, the image processing device can include a user interface operable to receive user input therein, which is, for example, a wavelength shift input for removing at least one color pixel value with a filter, followed by interpolation and enhancement of other color pixel values captured by the image sensor. The processed image generated by the image processing device of the medical device can include a partial-resolution frame of pixel values, which can be output to a display device.

[0013] Examples of the present disclosure can relate to devices and methods for performing various medical procedures and / or treating the large intestine (colon), small intestine, cecum, esophagus, and any other part of the gastrointestinal tract and / or any other suitable part of the patient's internal structure (collectively referred to herein as the "target treatment site"). The various examples described herein include single-use or disposable medical devices. The examples of the present disclosure, described above and shown in the accompanying drawings, are explained in detail below. Whenever possible, the same reference numbers are used throughout the drawings to refer to the same or similar parts.

[0014] FIG. 1 shows a schematic diagram of an exemplary medical system 100 according to the present disclosure. The medical system 100 may include one or more light sources 130, an image processing device 101, a medical instrument 110, and a medical device 140. The image processing device 101 may be communicatively coupled to the medical instrument 110, for example, by a wired connection, a wireless connection, or the like. In an example, the image processing device 101 is a computer system incorporating a plurality of hardware components that enable the image processing device 101 to receive data (e.g., data from an image sensor), process information (e.g., wavelength data), and / or generate a processed image for output to a user of the medical system 100. Exemplary hardware components of the image processing device 101 may include at least one processor 102, at least one memory 103, at least one user interface 108, and at least one display 109.

[0015] The processor 102 of the image processing device 101 may include any computing device capable of executing machine-readable instructions, which may be stored on a non-transitory computer-readable medium, such as the memory 103 of the image processing device 101. For example, the processor 102 may include a controller, an integrated circuit, a microchip, a computer, and / or any other computer processing unit operable to perform the calculations and logic operations necessary to execute a program. As will be described in more detail herein, the processor 102 is configured to execute one or more operations, such as imaging logic 104, demosaicking logic 105, edge enhancement logic 106, contrast enhancement logic 107, etc., in accordance with instructions stored on the memory 103.

[0016] Continuing to refer to FIG. 1, the memory 103 of the image processing apparatus 101 may include a non-transitory computer-readable medium that stores thereon machine-readable instructions, such as imaging logic 104, demosaicking logic 105, edge enhancement logic 106, and contrast enhancement logic 107. The imaging logic 104 may include executable instructions that enable the medical system 100 to capture an original digital image by operating one or more components of the medical instrument 110, such as one or more image sensors 150, 150A, 150B (FIGS. 2A-2C).

[0017] Furthermore, the demosaicking logic 105 may include executable instructions that enable the medical system 100 to process a digital image (e.g., a mosaic image) by demosaicking the image and reconstructing missing and / or unknown pixel values in the mosaic image. It should be understood that a digital image captured by an image sensor using a color filter sensor array may provide an original image having various color pixel values arranged in a mosaic pattern. Each pixel array of the pattern may include only one color pixel value, whereby one or more color pixel values may be removed therefrom. As will be described in detail herein, a digital image includes a two-dimensional array of pixel values, and each pixel value corresponds to the light intensity (e.g., a color pixel value) in one of a plurality of spectral bands at a certain pixel location within the image.

[0018] Continuing to refer to FIG. 1, edge enhancement logic 106 may include executable instructions that enable medical system 100 to process a mosaic image of a target site and enhance the sharpness of one or more edges within the mosaic image. It should be understood that the demosaicking process may include inherent side effects, such as, for example, a decrease in the sharpness of one or more edges within the image. For example, the demosaicking process may attenuate high-frequency detail of the image and / or emphasize low-frequency detail of the image.

[0019] In this example, color fringing may occur at the edges of sharp contrast boundaries within the image, where the edges of the sharp contrast boundaries may include fringe artifacts within the color pixels of the mosaic image. Value As will be described further below, edge enhancement logic 106 may include executable instructions that enable medical system 100 to process a digital image (e.g., a mosaic image) by detecting edges, increasing the detail of the edges, and providing a sharper resolution of the image within the color pixels. Value

[0020] Continuing to refer to FIG. 1, contrast enhancement logic 107 may include executable instructions that enable medical system 100 to process a mosaic image of a target site and enhance the contrast of one or more pixels within the mosaic image. It should be understood that the demosaicking process may include inherent side reactions, such as, for example, a decrease in the contrast of the image due to a decrease in the color difference signal between pixels within the image. Value Value

[0021] In this example, the resolution frame of the color pixels Value may include various color pixels Value ​​​It may lack sufficient brightness to differentiate one or more features of the image. As will be described later, the contrast enhancement logic 107 enables the medical system 100 to process a digital image (e.g., a mosaic image) by scaling the luminance of specific color pixels Value and increasing the brightness of the resolution frame to provide a clearer sharpness of the image within the color pixels Value and may include executable instructions to enable this.

[0022] In some embodiments, the imaging logic 104, the demosaicking logic 105, the edge enhancement logic 106, and / or the contrast enhancement logic 107 may include executable instructions that enable the medical system 100 to automatically perform periodic image processing of the target site without requiring user input. In other embodiments, the image processing device 101 may be configured to receive user input from, for example, the user interface 108 of the image processing device 101 to initiate image processing of the target site. It should be noted that in some embodiments, the user interface 108 may be an integrated device with the image processing device 101, and in other embodiments, the user interface 108 may be a remote device that communicates with the image processing device 101 (e.g., wirelessly, wired, etc.).

[0023] It should be understood that all or part of the various programming algorithms and data that assist in the operation of the medical system 100 may be in the memory 103. The memory 103 may include any type of computer-readable medium suitable for storing data and algorithms, which may be, for example, random access memory (RAM), read-only memory (ROM), flash memory, hard drive, and / or any device capable of storing machine-readable instructions. The memory 103 may include one or more data sets, which may include, but are not limited to, image data from one or more components of the medical system 100 (e.g., medical instrument 110, medical device 140, etc.).

[0024] Continuing to refer to FIG. 1, the medical instrument 110 can be configured to facilitate positioning of one or more components of the medical system 100, such as the medical device 140, with respect to a subject (e.g., a patient). In an embodiment, the medical instrument 110 can be any type of endoscope, duodenoscope, gastroscope, colonoscope, ureteroscope, bronchoscope, catheter, or other delivery system, and can include a handle 112, an operating mechanism 114, at least one port 116, and a shaft 120. The handle 112 of the medical instrument 110 can have one or more lumens (not shown) that communicate with lumens of one or more other components of the medical system 100. The handle 112 further includes at least one port 116 that opens into one or more lumens of the handle 112. As described in more detail herein, the at least one port 116 is sized and shaped to receive one or more instruments, such as the medical device 140 of the medical system 100, therein.

[0025] The shaft 120 of the medical instrument 110 can include a sufficiently flexible tube, and the shaft 120 is configured to selectively bend, rotate, and / or twist when inserted into and / or through a tortuous internal structure of the subject to a target treatment site. The shaft 120 can have one or more lumens (not shown) extending therethrough, which can include, for example, a working lumen for receiving an instrument (e.g., the medical device 140). In other examples, the shaft 120 can include additional lumens such as a control wire lumen for receiving one or more control wires for operating one or more distal components / tools (e.g., articulation joints, elevators, etc.), a fluid lumen for delivering fluid, a lighting lumen for receiving at least a portion of a lighting assembly (not shown), and / or an imaging lumen for receiving at least a portion of an imaging assembly (not shown).

[0026] Continuing to refer to FIG. 1, the medical instrument 110 may further include a tip 122 at the distal end of the shaft 120. In some embodiments, the tip 122 may be attached to the distal end of the shaft 120, and in other embodiments, the tip 122 may be integral with the shaft 120. For example, the tip 122 may include a cap configured to receive the distal end of the shaft 120 therein. The tip 122 may include one or more openings that communicate with one or more lumens of the shaft 120. For example, the tip 122 may include a working port 123 through which the medical device 140 may exit from the working lumen of the shaft 120. It should be noted that one or more other openings of the tip 122 of the shaft 120 are not shown. The operating mechanism 114 of the medical instrument 110 is on the handle 112 and may include one or more knobs, buttons, levers, switches, and / or suitable actuators thereof. The operating mechanism 114 is configured to control at least the orientation of the shaft 120 (e.g., through the operation of a control wire).

[0027] The medical device 140 of the medical system 100 may include a catheter having a longitudinal body 142 between the proximal end 141 and the distal end 144 of the medical device 140. The longitudinal body 142 of the medical device 140 may be flexible, such that the medical device 140 is configured to bend, rotate, and / or twist when inserted into the working lumen of the medical instrument 110. The medical device 140 may include a handle at the proximal end 141 of the longitudinal body 142, which may be configured to move, rotate, and / or bend the longitudinal body 142. Further, the handle at the proximal end 141 of the medical device 140 may define one or more ports (not shown) sized to receive one or more tools through the longitudinal body 142 of the medical device 140.

[0028] Continuing to refer to FIG. 1, the medical instrument 110 can be configured to receive the medical device 140 from at least one port 116, through the working lumen, through the shaft 120, and to the working port 123 of the tip 122. In this example, the medical device 140 can extend distally from the working port 123 into the surrounding environment of the tip 122, for example at the target treatment site of the subject, as will be described in more detail later. The distal end 144 of the medical device 140 can extend distally from the tip 122 in response to the longitudinal body 142 moving within the working lumen of the shaft 120. The medical device 140 can include one or more end effectors (not shown) at the distal end 144 of the longitudinal body 142 for performing one or more operations at the target treatment site.

[0029] The medical instrument 110 can further be configured to receive one or more light sources 130 that pass through the shaft 120 through at least one of the lumens of the medical instrument 110 so as to be connected to the optical fiber 146. In this example, the one or more light sources 130 are shown as a component separate from the image processing device 101, whereby the light source 130 is connected to the medical instrument 110 separately from the image processing device (e.g., via a cable). It should be noted that in other embodiments, the one or more light sources 130 may be included in the image processing device 101, whereby the light source 130 can be communicatively connected to the medical instrument 110 together with the image processing device 101.

[0030] Next, referring to FIGS. 2A - 2C, the chip 122 of the medical instrument 110 according to one or more examples of the present disclosure is shown. First, referring to FIG. 2A, in one embodiment, the chip 122 of the medical instrument 110 may include an optical fiber 146 and an image sensor 150 on the chip 122. In this example, the optical fiber 146 may be coupled to one or more light sources 130 of the medical system 100, whereby each of the one or more light sources 130 may transmit light through one optical fiber 146. Although not shown, it should be appreciated that multiple light sources 130 may be coupled to the optical fiber 146 via a fiber splitter / combiner. The optical fiber 146 of the medical instrument 110 is configured to, and may be operable to, deliver light of various amplitudes from one or more light sources 130 distally from the tip 122 of the shaft 120. In some embodiments, the optical fiber 146 may be configured to deliver white light, ultraviolet light, near-infrared (NIR) light, and / or other various wavelengths within or beyond the visible spectrum.

[0031] Continuing to refer to FIG. 2A, the image sensor 150 of the medical instrument 110 may be communicatively coupled to the image processing device 101 of the medical system 100 via, for example, a wired connection and / or a wireless connection, etc. The image sensor 150 of the medical instrument 110 is configured to, and may be operable to, image a raw image (e.g., a digital image) of the surrounding environment of the tip 122 of the shaft 120. In some embodiments, the image sensor 150 may include an image sensor, such as an RGB (i.e., red - green - blue) digital sensor, an RGB - Ir (i.e., red - green - blue - infrared) digital sensor, and / or a monochrome sensor, etc. As will be described in more detail herein, the image sensor 150 may include one or more components for filtering colors from white light, ultraviolet light, near-infrared light, and / or other wavelengths within or beyond the visible spectrum.

[0032] In other embodiments, referring next to FIG. 2B, the medical device 110 may include a multi-color LED assembly at the tip 122 of the shaft 120. In this example, the multi-color LED assembly may include one or more light-emitting diodes (hereinafter, LEDs) 146A, 146B, 146C, 146D arranged as an annular array around the image sensor 150. Each of the LEDs 146A, 146B, 146C, 146D may be configured to transmit different wavelengths and / or amplitudes (e.g., colors) of light relative to each other and may be operable as such. It should be understood that different illumination sources may generate different spectra. It should be recognized that the LEDs 146A, 146B, 146C, 146D of the medical device 110 may include more and / or fewer diodes than those illustrated and described herein in the tip 122, and that too does not depart from the scope of the present disclosure.

[0033] In other embodiments, referring next to FIG. 2C, the medical device 110 may include a multi-sensor assembly at the tip 122 of the shaft 120. In this example, the multi-sensor assembly may include a color image sensor 150A and a monochrome image sensor 150B. As will be described in more detail herein, the color image sensor 150A may be configured to image a portion of the incident light corresponding to the color of the incident light at each of the individual pixel locations of the color image sensor 150A and may be operable as such. In some embodiments, the color image sensor 150A may include, for example, RGB (red-green-blue digital sensor), and RGB-Ir (red-green-blue-infrared) digital sensors, etc. As will be described in more detail herein, the monochrome image sensor 150B may be configured to image all of the incident light completely at each of the individual pixel locations of the monochrome sensor 150B regardless of the color of the incident light and may be operable as such.

[0034] Next, referring to FIG. 3, the image sensor 150 of the medical device 110 may include an outer surface 152 on which a plurality of microlenses 154 are disposed. In some examples, the outer surface 152 and / or the plurality of microlenses 154 may be formed of glass, plastic, and / or other transparent materials. The image sensor 150 may be a color image sensor that includes a color filter array 156 disposed relatively under the outer surface 152. The color filter array 156 may include an optical fiber device having a plurality of color pixel locations 158A, 158B, 158C arranged in a predetermined pattern. In this example, each of the plurality of microlenses 154 may be positioned in alignment with at least one of the plurality of color pixel locations 158A, 158B, 158C of the color filter array 156 disposed under the outer surface 152.

[0035] In some embodiments, the color filter array 156 may include a plurality of first color pixel locations 158A, a plurality of second color pixel locations 158B, and / or a plurality of third color pixel locations 158C. The plurality of color pixel locations 158A, 158B, 158C may be arranged in a mosaic pattern, such as a Bayer pattern, along the color filter array 156. In this example, the plurality of first color pixel locations 158A may include a red filter, the plurality of second color pixel locations 158B may include a green filter, and the plurality of third color pixel locations 158C may include a blue filter. In other embodiments, the plurality of color pixels Value 158A, 158B, 158C may include various suitable color filters and / or patterns other than those illustrated and described herein. For example, in embodiments where the image sensor 150 includes an RGB-Ir sensor, it should be understood that the color filter array 156 may additionally include a plurality of fourth color pixel locations corresponding to infrared color filters.

[0036] Continuing to refer to FIG. 3, the color filter array 156 of the image sensor 150 is configured and operable to selectively transmit one or more wavelengths 12, 14, 16 of the light beam 10 (e.g., light intensity, spectral band, color, etc.). For example, each of the color pixel locations 158A, 158B, 158C is configured to deliver and / or transmit a portion of the light beam 10 (e.g., at least one wavelength 12, 14, 16) through the color filter of the color pixel locations 158A, 158B, 158C. Thus, at each color pixel location 158A, 158B, 158C, only one color component (e.g., a color pixel value) can be measured by the image sensor 150. As further described herein, the color pixel value may include an amount of energy in a spectral colored range (e.g., a red range, a green range, a blue range, etc.).

[0037] In this example, each of the plurality of color pixel locations 158A, 158B, 158C may be configured such that only one wavelength 12, 14, 16 of the light beam 10 can pass through the color filter array 156. The image sensor 150 may further include a photosensor array 160 disposed relatively under the color filter array 156, and the color filter array 156 of the image sensor 150 may be positioned between the outer surface 152 and the photosensor array 160. The photosensor array 160 of the image sensor 150 may include a photodiode (e.g., a semiconductor device) having a plurality of photosites 162 and a circuit configuration 164 communicatively coupled to the plurality of photosites 162.

[0038] Continuing to refer to FIG. 3, a plurality of photosites 162 are arranged in an array (e.g., a grid), and each of the plurality of photosites 162 is positioned in alignment with at least one of a plurality of color pixel locations 158A, 158B, 158C of a color filter array 156 disposed above the photosensor array 160. The photosensor array 160 is configured and operable to convert the light beam 10 received through the outer surface 152 and the color filter array 156 into an electric current. In this example, when photons from the received light are absorbed by the plurality of photosites 162, an electric current can be generated by the photosensor array 160.

[0039] In this example, each of the plurality of photosites 162 can be configured to measure the quantity of only one color pixel value (e.g., red, green, blue) within the incident light 10 at the location of the photosite 162 along the surface of the photosensor array 160. Thus, the plurality of photosites 162 can image the incident light 10, generate an electrical signal, which can be quantified and stored as a numerical value in the resulting processed image file. It should be recognized that the photosensor array 160 can include various suitable shapes, sizes, and / or configurations other than those illustrated and described herein. In other embodiments, the image sensor 150 can be a monochrome sensor (e.g., monochrome sensor 150B), and thereby the color filter array 156 illustrated and described above can be completely eliminated from between the outer surface 152 and the photosensor array 160. In this example, each photosite 162 along the photosensor array 160 can be operable to receive, image, and absorb all three wavelengths 12, 14, 16 of the light beam 10.

[0040] Next, referring to FIGS. 3, 4 in conjunction with the flowchart of FIG. 5, an exemplary method 200 for generating a processed image of a target site using the medical system 100 is schematically shown. The figures of FIGS. 3 - 5 and the following description related thereto are not intended to limit the subject matter described herein to a particular method.

[0041] First, with respect to FIG. 1, the medical instrument 110 of the medical system 100 can be inserted into the body (not shown) of the subject such that the chip 122 is positioned adjacent to the target site. For example, the shaft 120 can be guided through the digestive tract of the subject (e.g., a patient) by inserting the chip 122 into the subject's nose or mouth (or other suitable natural opening of the body) and reaching the target site through the gastrointestinal tract (e.g., esophagus, stomach, small intestine, etc.) within the subject's body. It should be noted that the length of the shaft 120 can be sufficient such that the proximal end (including the handle 112) of the medical instrument 110 is outside the subject's body while at the same time the chip 122 of the medical instrument 110 is inside the subject's body. Although the present disclosure relates to the use of the medical system 100 in the digestive tract of the subject, it should be understood that the features of the present disclosure can also be used at various other locations within the subject's body (e.g., other organs, tissues, etc.).

[0042] Additionally, when the medical instrument 110 is received into the subject's body and the chip 122 of the shaft 120 is disposed relatively close to the target site, the medical device 140 can be received into the medical instrument 110 from at least one port 116. In this example, the longitudinal body 142 of the medical device 140 is moved through at least one (e.g., the working lumen) of the lumens of the shaft 120 within the shaft 120. The distal end 144 of the longitudinal body 142 can extend distally from the chip 122 of the shaft 120 through a working port 123 that communicates with the working lumen of the shaft 120. It should be noted that this step can be optional, and enabling the medical device 140 to be received into the medical instrument 110 can be performed in various other steps of the method 200 and / or can be completely omitted. The chip 122 can be positioned adjacent to and facing the target treatment site.

[0043] Referring to FIG. 5, in step 202, in response to the processor 102 of the imaging apparatus 101 executing the imaging logic 104 to activate one or more light sources 130, one or more target objects can be illuminated by the medical instrument 110. In an example of the medical instrument 110 including the optical fiber 146 (FIG. 2A) and / or the multi-color LED assemblies 146A, 146B, 146C, 146D (FIG. 2B), light from the one or more light sources 130 can be emitted from the chip 122 of the medical instrument 110 to illuminate the target object.

[0044] In step 204, when the target object is illuminated by the light from the medical instrument 110, the image sensor 150 can be activated by the processor 102 executing the imaging logic 104 to capture one or more raw digital images of the target object. It should be understood that the processor 102 of the imaging apparatus 101 can be communicably connected to the image sensor 150 of the medical instrument 110 via the circuit configuration 164. For example, referring to FIG. 3 again, the light 10 transmitted to the target object by the optical fiber 146 and / or the multi-color LED assemblies 146A, 146B, 146C, 146D can be reflected by the target object and received by the image sensor 150. In this example, a plurality of wavelengths 12, 14, 16 of the light 10 can be received through one or more of the plurality of microlenses 154 on the outer surface 152.

[0045] The plurality of wavelengths 12, 14, 16 can be received in correspondence with one or more corresponding color pixel locations 158A, 158B, 158C of the color filter array 156, for example, aligned with a microlens 154 that receives the light beam 10 therethrough. At each of the plurality of color pixel locations 158A, 158B, 158C, one or more of the plurality of wavelengths 12, 14, 16 of the light 10 can be blocked (e.g., filtered, excluded, eliminated, blocked) from passing through the color filter array 156 according to the color filters of the color pixel locations 158A, 158B, 158C. Therefore, depending on the color filter configuration (e.g., red, green, blue, etc.) of the color pixel locations 158A, 158B, 158C that receive the light 10, which of the wavelengths 12, 14, 16 (e.g., red, blue, green, etc.) can pass through the color filter array 156 at said locations is determined.

[0046] Accordingly, it should be understood that each of the color pixel locations 158A, 158B, 158C can be configured such that only about one-third (e.g., 33%) of the incident light 10 at said locations can pass through the photosensor array 160. For example, at each of the plurality of first color pixel locations 158A (e.g., red filter), only the wavelength 12 of the light 10 (e.g., the red range of the light spectrum) can pass through the color filter array 156, whereby the wavelengths 14, 16 (e.g., blue and green respectively) can be removed by the color filter array 156 at the first color pixel location 158A. It should be recognized that in this embodiment, each of the plurality of second color pixel locations 158B (e.g., green filter) can be configured to transmit the wavelength 14 (e.g., the green range of the light spectrum), and each of the plurality of third color pixel locations 158C (e.g., blue filter) can be configured to transmit the wavelength 16 (e.g., the blue range of the light spectrum).

[0047] Continuing to refer to FIG. 3, the individual wavelengths 12, 14, 16 of light 10 that pass through color filter array 156 are detected along photosensor array 160 and can be absorbed by one or more of a plurality of photosites 162 (i.e., those that coincide with color pixel locations 158A, 158B, 158C that receive light beam 10 passing therethrough). In this example, the portions of light 10 absorbed by each of the plurality of photosites 162 can be converted into an electric current. The original digital image captured by image sensor 150 can include a quantitative record of the light energy measured at each grid location of photosites 162 along photosensor array 160, and each photosite 162 is configured to identify the color pixel values of wavelengths 12, 14, 16 received thereon.

[0048] In this example, when processor 102 of image processing apparatus 101 executes imaging logic 104, it can cause an electrical signal of color pixel values to be transmitted to image processing apparatus 101 from photosensor array 160, for example via circuit configuration 164. The electrical signal of color pixel values is stored in memory 103 of image processing apparatus 101 and can be used to generate a processed image by demosaicking logic 105, edge enhancement logic 106, and / or contrast enhancement logic 107.

[0049] Referring to FIG. 5, at step 206, a wavelength shift input can be entered by a user of medical system 100, for example via user interface 108 of image processing apparatus 101. The wavelength shift input can include identification information, amplitude, and / or color pixel values, etc. of one or more light wavelengths for adjusting the demosaicking process of the original digital image captured by image sensor 150. It should be noted that in some embodiments, step 206 can be performed before steps 202 and 204 and / or can be automatically pre-programmed into memory 103 of image processing apparatus 101, whereby method 200 can proceed to step 208 with the wavelength shift input pre-specified.

[0050] Next, referring to FIG. 4, the original digital image received by the image processing apparatus 101 is composed of original pixels formed by a plurality of color pixel values 22A, 22B, and 22C measured by the image sensor 150. Frame 20 of the value It may include (for example, a grid). It should be understood that each of the grid locations along the original pixel Frame 20 of the value may correspond to the location of a photosite 162 on the photosensor array 160. Therefore, each of the grid locations on the original pixel Frame 20 of the value includes at least one of the first color pixel value 22A, the second color pixel value 22B, or the third color pixel value 22C based on the relative position of the grid location with respect to the color pixel locations 158A, 158B, 158C of the color filter array 156 through which the photosite 162 receives wavelengths 12, 14, and 16.

[0051] In step 208, the processor 102 separates the original pixel Frame 20 of the value into color pixel values 22A, 22B, and 22C, and may filter (for example, exclude) at least one of the first color pixel value 22A, the second color pixel value 22B, or the third color pixel value 22C based on the wavelength shift input received in step 206. In other words, the image processing apparatus 101 is configured to shift the demosaicking process of the original digital image captured by the image sensor 150 from a plurality of color pixel values 22A, 22B, and 22C in the frame 20 of the original pixel values to a small portion of the color pixel values 22A, 22B, and 22C.

[0052] Therefore, as will be described in detail later, the demosaicking process (steps 210A, 210B), edge enhancement process (steps 212A, 212B), and / or contrast enhancement process (steps 214A, 214B) of the original digital image are performed only on a part of the frame 20 of the original pixel values included in the original digital image captured by the image sensor 150. For example, the processor 102 of the image processing apparatus 101 removes a plurality of first color pixel values 22A according to the wavelength shift input, whereby the resulting processed image generated by the image processing apparatus 101 includes a partial resolution frame of pixels.

[0053] For example, continuing to refer to FIG. 5, in step 210A, the processor 102, for example, the original pixels that did not receive wavelength 14 (e.g., green), such as the grid locations that received wavelengths 12 (e.g., red) and 16 (e.g., blue). Frame 20 of the value To calculate the estimated value of the second color pixel value 22B (e.g., green) at the grid location along, the demosaicking process of the plurality of second color pixel values 22B is performed (when executing the demosaicking logic 105). The processor 102 can estimate the amount of the unknown second color pixel value 22B at the grid location on the frame 20 of the original pixel values that did not receive wavelength 14 by interpolation from the measured values of the known second color pixel values 22B.

[0054] In this example, the processor 102 that executes the demosaicking logic 105 interpolates the missing second color pixel value 22B from an adjacent (e.g., neighboring) grid location that includes the measured second color pixel value 22B. The processor 102 determines the amount of the unknown second color pixel value 22B from a neighboring grid location that received wavelength 14. It should be understood that the processor 102 executes the demosaicking logic 105 in substantially the same manner as described above with respect to step 210A in step 210B, and may reconstruct unknown and / or missing third color pixel values 22C along the frame 20 of the original pixel values.

[0055] Continuing to refer to FIG. 5, in step 212A, the processor 102 of the image processing apparatus 101 may further reconstruct the original digital image captured by the image sensor 150 by executing edge enhancement logic 106. In this example, when executed by the processor 102, the edge enhancement logic 106 may enhance the detail and / or sharpness of one or more edges within the original pixel values Frame 20 For example, the processor 102 may detect the position of one or more edges within each grid location in the frame 20 of the original pixel values and minimize the noise level around those edges by adjusting one or more of the plurality of second color pixel values 22B at that grid location. In some embodiments, the processor 102 may improve the sharpness of one or more edges within the grid location by increasing the gradient (e.g., magnitude) of the plurality of second color pixel values 22B.

[0056] It should be understood that the processor 102 executes the edge enhancement logic 106 in substantially the same manner as described above with respect to step 212A in step 212B, and the original pixel values Frame 20By adjusting one or more of the plurality of third color pixel values 22C along [it], the edges of the digital image can be reconstructed. As a point to note, in other embodiments, various other suitable edge enhancement processes can be included in the edge enhancement logic 106 and executed by the processor 102.

[0057] Continuing to refer to FIG. 5, in step 214A, the processor 102 of the image processing apparatus 101 can further reconstruct the original digital image captured by the image sensor 150 by executing the contrast enhancement logic 107. In this example, when the contrast enhancement logic 107 is executed by the processor 102, it can emphasize the contrast of the representation of the frame 20 of the original pixel values. For example, the processor 102 can increase one or more brightness components (e.g., luminance) of the plurality of second color pixel values 22B within each grid location in the frame 20 of the original pixel values. In some embodiments, the processor 102 can adjust the brightness of one or more grid locations within the frame 20 of the original pixel values by reducing the second color pixel values 22B therein to reduce the contrast contribution provided by the second color pixel values 22B.

[0058] It should be understood that in step 214B, the processor 102 executes the contrast enhancement logic 107 in a substantially similar manner as described above with respect to step 214A, and can enhance the local contrast of the digital image by adjusting one or more of the plurality of third color pixel values 22C along the frame 20 of the original pixel values. As a point to note, in other embodiments, various other suitable contrast enhancement processes can be included in the contrast enhancement logic 107 and executed by the processor 102.

[0059] Continuing to refer to FIG. 5, in step 216, the processor 102 of the image processing apparatus 101 can identify whether the current iteration of the demosaicking process (steps 210A, 210B), the edge enhancement process (steps 212A, 212B), and the contrast enhancement process (steps 214A, 214B) is equal to or greater than a predetermined and / or dynamic iteration threshold. In some embodiments, the predetermined iteration threshold can be pre-determined and stored in the memory 103 of the image processing apparatus 101, or selectively input by a user of the medical system 100.

[0060] In other embodiments, the iteration threshold can be determined dynamically by the processor 102 based on one or more factors including, for example, peak sharpness, contrast, and visibility from an initial transformation of the digital image through the demosaicking, edge enhancement, and contrast enhancement processes. In this example, data representing an initial state of an original pixel value of 20 (e.g., a histogram of the frame) can be analyzed by the processor 102 at the completion of the first iteration, and peak values regarding sharpness, contrast, and visibility characteristics can be identified. Thus, the processor 102 can continuously identify the current sharpness, contrast, and visibility of the digital image regarding the initial peak values (i.e., the dynamic iteration threshold) at the completion of each iteration of the process.

[0061] In response to the processor 102 in step 216 identifying that the current iteration of method 200 is lower than a predetermined (or dynamic) threshold, the image processing apparatus 101 may be configured and operable to return to steps 210A and 210B and perform one or more of the demosaicking, edge enhancement, and / or contrast enhancement processes. In response to the processor 102 in step 216 identifying that the current iteration of method 200 is at least equal to or greater than a predetermined (or dynamic) threshold, the image processing apparatus 101 may be configured and operable to generate an interpolated output image. It should be noted that the original pixel values initially captured by the image sensor 150 Frame 20 In response to iterative enhancement, an image with improved sharpness, contrast, and / or visibility may be provided.

[0062] Continuing to refer to FIG. 5, in step 218, the processor 102 of the image processing apparatus 101 may generate a processed image obtained from the process of method 200 illustrated and described herein. In this example, the output image may include a sub-resolution frame of color pixel values, such as a plurality of second color pixel values 22B and a plurality of third color pixel values 22C. Therefore, in the digital image generated by the processor 102 of the image processing apparatus 101, at least a plurality of first color pixel values 22A may be omitted from one or more of the grid locations of the original pixel values Frame 20 initially captured by the image sensor 150. In this example, from the generated image, the image processing apparatus 101 may generate a partial-resolution image by excluding the first color pixel value 22A at each of the grid locations of the original pixel values Frame 20 In each of the grid locations of the original pixel values.

[0063] With the display 109 of the medical system 100 communicably connected to the processor 102 of the image processing apparatus 101, the processor 102 can be operable to transmit a partial resolution image to the display 109 so that it can be seen by the user of the medical system 100. In some examples, the medical system 100 can be configured to continuously execute the method 200 illustrated and described herein and can be operable as such, whereby the display 109 can output a plurality of partial resolution images for providing continuous (e.g., live, real-time, etc.) imaging of one or more target objects.

[0064] As a point to be recognized, by excluding at least one of the original pixel values Frame 20 within the processed image (by the image sensor 150), i.e., at least one of the plurality of color pixel values 22A, 22B, 22C originally captured, the differentiation of one or more features and / or structures (e.g., target object) of the target treatment site can be facilitated. For example, by the processor 102 excluding at least one of the color pixel values 22A, 22B, 22C by filtering according to the wavelength shift input in step 206, the blood vessels can be better distinguished from the surrounding tissues. In some examples, the main colors of one or more target objects can be substantially similar to each other. For example, tissues and blood vessels are generally displayed in red, whereby the features are difficult to distinguish in the generated image. In this example, by emphasizing the blue and / or green color pixels Value (e.g., the second color pixel value 22B and the third color pixel value 22C respectively), the features can be made more distinguishable.

[0065] Additionally, as a point to be understood, the contrast of one or more color components (e.g., red) may be lower compared to one or more other color components (e.g., green, blue) within the target treatment site. By excluding one or more of the color components, e.g., the first color pixel value 22A (e.g., red), from the resulting digital image, the vasculature of the subject can be emphasized, which is the original pixel value Frame 20This is because only the second color pixel value 22B (e.g., blue) and the third color pixel value 22C (e.g., green) are retained from

[0066] In an embodiment where the optical fiber 146 of the medical device 140 is operable to generate ultraviolet light, the medical system 100 can be configured to generate a partial resolution frame that can distinguish desirable (e.g., healthy) tissue from undesirable (e.g., unhealthy) tissue, because the tissue can emit fluorescence of different colors under ultraviolet light. In this example, the wavelength shift input of step 206 can include the first color pixel value 22A (e.g., red) and the third color pixel value 22C (e.g., green), such that only the second color pixel value 22B (e.g., blue) can be included in the resulting processed image.

[0067] In an embodiment where the image sensor 150 of the medical instrument 110 includes an RGB-Ir sensor, the raw pixel values detected and imaged by the image sensor Frame 20 can include a plurality of first color pixel values 22A (e.g., red), a plurality of second color pixel values 22B (e.g., blue), a plurality of third color pixel values 22C (e.g., green), and a plurality of fourth color pixel values (e.g., infrared) that are not included in the visible spectrum and are closer to the infrared spectrum. In this example, the medical system 100 can be configured to generate a partial resolution frame that makes a relatively dark region within the target treatment site (e.g., a body lumen) more visible by emphasizing the first color pixel value 22A, the third color pixel value 22C, and / or the fourth color pixel value. Accordingly, the wavelength shift input of step 206 can include the second color pixel value 22B (e.g., blue) for exclusion from the processed image.

[0068] In an embodiment where the chip 122 of the shaft 120 includes a color image sensor 150A (e.g., RGB-Ir sensor) and a monochrome image sensor 150B, the image processing apparatus 101 generates a partial resolution image captured by the color image sensor 150A, while the original pixel values Frame 20 within the relatively dark regions in Frame 20 can be further configured to enhance the contrast. In this example, since the monochrome image sensor 150B can sense the near-infrared wavelength irradiated within the light 10, a fourth color pixel value (e.g., infrared) can be easily detected. It should be noted that by providing one or more materials, such as a fluorescent dye, within the target treatment site, visualization of one or more target objects by the monochrome image sensor 150B can be facilitated.

[0069] Each of the above-described systems, devices, assemblies, and methods can be used to generate a partial resolution frame of the pixel values of a subject. By providing a medical device that includes an image processing system storing wavelength shift demosaicking logic, a user can better visualize one or more features and / or characteristics of a target site within a subject's body during a procedure without operating a light source. With this medical device, the user may be able to accurately identify the position of the target site, thereby shortening the overall procedure time, improving the efficiency of the procedure, and avoiding unnecessary harm to the subject's body that may occur if the identification of the position of the target object at the target treatment site is inaccurate.

[0070] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed apparatus and methods without departing from the scope of the present disclosure. It should be recognized that the disclosed apparatus may include a plurality of hardware components, such as a processor, and various suitable computer systems and / or computing units incorporating a non-transitory computer-readable medium enabling the apparatus to execute one or more operations while being disposed in accordance with what is described herein. Other aspects of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. The specification and examples are to be considered exemplary only.

Claims

1. A medical device, the medical device comprises a shaft, the medical device comprises a sensor connected to the distal end of the shaft and including a filter array, the sensor being configured to capture a raw image, and the filter array being configured to filter the raw image into a frame of raw pixel values, the frame of raw pixel values including a plurality of first pixel values, a plurality of second pixel values, and a plurality of third pixel values, the medical device comprises a processor and a non-transitory computer-readable medium storing demosaicking instructions, the demosaicking instructions, when executed by the processor, causing the processor to, exclude the plurality of first pixel values from the frame of raw pixel values, generate a plurality of second predicted pixel values at locations of the plurality of excluded first pixel values on the frame and the plurality of third pixel values, generate a plurality of third predicted pixel values at locations of the plurality of excluded first pixel values on the frame and the plurality of second pixel values, generate a processed image having a sub-resolution frame from the plurality of second pixel values, the plurality of second predicted pixel values, the plurality of third pixel values, and the plurality of third predicted pixel values, A medical device.

2. the demosaicking instructions stored in the non-transitory computer-readable medium cause the processor to, detect one or more edges in the raw image, perform edge sharpening enhancement of the one or more edges to increase edge detail of the processed image, the medical device according to claim 1.

3. the demosaicking instructions stored in the non-transitory computer-readable medium cause the processor to, Output a sharpened enhancement image created by performing a sharpened enhancement step, The medical device according to claim 2, wherein the sharpened enhancement image is fused with the processed image.

4. The demosaicking instruction stored in the non-transitory computer-readable medium causes the processor to, Set the luminance value of each of the plurality of second pixel values and the plurality of third pixel values, The medical device according to claim 3, wherein the processor executes contrast enhancement between the plurality of second pixel values and the plurality of third pixel values by adjusting the luminance value to enhance the contrast of the processed image.

5. The demosaicking instruction stored in the non-transitory computer-readable medium causes the processor to, Output a contrast enhancement image created from the execution of the contrast enhancement, The medical device according to claim 4, wherein the contrast enhancement image is fused with the processed image.

6. The demosaicking instruction stored in the non-transitory computer-readable medium causes the processor to, The medical device according to claim 5, wherein the processor repeats all of the above steps up to a threshold value.

7. The demosaicking instruction stored in the non-transitory computer-readable medium causes the processor to, Receive a wavelength shift input to identify the color pixel value of the plurality of first pixel values, the medical device according to any one of claims 1 to 6.

8. The medical device according to claim 7, further comprising a user interface communicably coupled to the processor and configured to transmit a signal indicating the wavelength shift input to the processor.

9. The sensor includes an RGB image sensor, and the filter array includes a red-green-blue Bayer color filter array. The plurality of first pixel values include red pixel values, the plurality of second pixel values include blue pixel values, and the plurality of third pixel values include green pixel values. The medical device according to any one of claims 1 to 8.

10. The sensor includes an RGB+Ir image sensor, and the filter array includes a red-green-blue-infrared Bayer color filter array. The medical device according to any one of claims 1 to 9.

11. The sensor includes an RGB+Ir image sensor, and the filter array includes a red-green-blue-infrared Bayer color filter array. The plurality of first pixel values include blue pixel values, the plurality of second pixel values include red pixel values and green pixel values, and the plurality of third pixel values include infrared pixel values. The medical device according to any one of claims 1 to 8.

12. The sensor includes an RGB image sensor and a monochrome sensor. The medical device according to any one of claims 1 to 11.

13. Each location in the partial resolution frame includes one captured color pixel value and one reconstructed color pixel value such that at least one color pixel value is excluded from the frame of the original pixel values. The medical device according to any one of claims 1 to 12.

14. The medical device according to any one of claims 1 to 13 further includes a light source connected to the distal end of the shaft, and the light source is an optical fiber or a multi-color LED array.

15. The demosaicking instructions stored in the non-transitory computer-readable medium cause the processor to output the processed image of the partial resolution frame to a display device. The medical device according to any one of claims 1 to 14.

Images