Image processing system and method using the same

Abstract

A medical device including a sensor including a filter array. The sensor is configured to capture a raw image and the filter array is configured to filter the raw image into a frame of raw pixel values including a first pixel value, a second pixel value, and a third pixel value. The medical device includes a non-transitory computer readable medium storing demosaicing instructions that when executed cause a processor to exclude the first pixel value from the frame of raw pixel values. The processor generates an estimated second pixel value at a location of the frame where the first pixel value and the third pixel value are excluded and generates an estimated third pixel value at a location of the frame where the first pixel value and the second pixel value are excluded. The processor creates a processed image from the second pixel value, the estimated second pixel value, the third pixel value, and the estimated third pixel value, the processed image having a local resolution frame.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 005,832, filed April 6, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates in general to image processing systems, apparatus, and related methods. Examples of this disclosure relate to systems, apparatus, and related methods for digital chromoendoscopy, etc. Background Technology

[0004] Technological advancements have empowered users of medical systems, devices, and methods to perform increasingly complex surgeries on patients. One challenge in the field of minimally invasive surgery is visualizing target treatment sites within the patient's body, such as tumors or lesions located in the gastrointestinal tract. Dye-injected chromoendoscopy can facilitate the detection of changes in the mucosal surfaces of the gastrointestinal tract. However, limitations in imaging methods and devices used to provide visualization of blood vessels can prolong surgical time, limit their effectiveness, and / or cause harm to the patient. Summary of the Invention

[0005] This disclosure relates to systems, apparatus, and methods for providing image processing systems, as well as wavelength-shifting demosaic logic, etc. Each of the aspects disclosed herein may include one or more features described in conjunction with any of the other disclosed aspects.

[0006] According to one example, a medical device includes an axis and a sensor coupled to a distal end of the axis and including a filter array. The sensor is configured to capture a raw image, and the filter array is configured to filter the raw image into frames of raw pixel values, the frames including a plurality of first pixel values, a plurality of second pixel values, and a plurality of third pixel values. The medical device includes a processor and a non-transitory computer-readable medium storing demosaic instructions that, when executed by the processor, cause the processor to exclude a plurality of first pixel values ​​from the frames of raw pixel values. The processor generates a plurality of estimated second pixel values ​​at the locations of the excluded first and third pixel values ​​on the frame. The processor generates a plurality of estimated third pixel values ​​at the locations of the excluded first and second pixel values ​​on the frame. The processor creates a processed image with local resolution frames from the plurality of second pixel values, the plurality of estimated second pixel values, the plurality of third pixel values, and the plurality of estimated third pixel values.

[0007] Any of the medical devices described herein may include any of the following features: Demosaic instructions stored in a non-transitory computer-readable medium cause the processor to detect one or more edges within an original image and perform sharpening enhancement on the one or more edges to increase edge detail in the processed image. Demosaic instructions stored in a non-transitory computer-readable medium cause the processor to output a sharpened enhanced image created from the sharpening enhancement step and blend the sharpened enhanced image with the processed image. Demosaic instructions stored in a non-transitory computer-readable medium cause the processor to set a brightness value for each of a plurality of second pixels and a plurality of third pixels and perform contrast enhancement on the plurality of second pixels and a plurality of third pixels by modifying the brightness values ​​to increase the contrast of the processed image. Demosaic instructions stored in a non-transitory computer-readable medium cause the processor to output a contrast-enhanced image created from the contrast enhancement step and blend the contrast-enhanced image with the processed image. Demosaic instructions stored in a non-transitory computer-readable medium cause the processor to repeat all the previous steps until a threshold is reached. Demosaic instructions stored in a non-transitory computer-readable medium cause the processor to receive a shift wavelength input to determine color pixel values ​​for a plurality of first pixels. The medical device also includes a user interface communicatively coupled to a processor and configured to transmit a signal indicating a shift wavelength input to the processor. The sensor includes an RGB image sensor, and the filter array includes a red-green-blue Bayer color filter array. A plurality of first pixels include red pixels, a plurality of second pixels include blue pixels, and a plurality of third pixels include green pixels. The sensor also includes an RGB+Ir image sensor, and the filter array includes a red-green-blue-infrared Bayer color filter array. A plurality of first pixels include blue pixels, a plurality of second pixels include red and green pixels, and a plurality of third pixels include infrared pixels. The sensor includes both an RGB image sensor and a monochrome sensor. Each location in a local resolution frame of a pixel includes a captured color pixel value and a reconstructed color pixel value, such that at least one color pixel value is excluded from the frame of the original pixel. The medical device also includes a light source coupled to a distal end of an axis, wherein the light source is an optical fiber, ultraviolet light, or a multicolor LED array. Demosaic instructions stored in a non-transitory computer-readable medium cause the processor to output a processed image of the local resolution frame to a display device.

[0008] According to another example, the image processing method includes capturing an original image and filtering the original image into frames of original pixels, the original pixels including a plurality of first pixels, a plurality of second pixels, and a plurality of third pixels. The method includes excluding at least the plurality of first pixels, and generating missing second pixels along the frame at pixel locations of the excluded first pixels and the plurality of third pixels. The method includes generating missing third pixels along the frame at pixel locations of the excluded first pixels and the plurality of second pixels, and constructing a partially sampled digital image from the plurality of second pixels, the plurality of third pixels, the generated second pixels, and the generated third pixels.

[0009] Any of the methods described herein may include any of the following steps. The method includes detecting edges within a frame of original pixels and enhancing the sharpness of the edges to increase edge detail in the partially sampled digital image. The method includes modifying the brightness value of each of a plurality of second pixels and a plurality of third pixels to enhance the contrast of the plurality of second pixels and the plurality of third pixels in the partially sampled digital image. The method includes receiving a shifted wavelength output to determine the color pixel values ​​of a plurality of first pixels. The frame of original pixels also includes a plurality of fourth pixels. The method includes excluding at least a plurality of fourth pixels.

[0010] According to another example, the medical device includes a processor and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the processor to transmit light having multiple wavelengths from an illumination source and at least remove a subgroup of multiple pixel values ​​detected by a digital image sensor communicatively coupled to the processor. The digital image sensor includes a filter array configured to filter the multiple wavelengths of light into multiple pixel values. The processor generates multiple estimated pixel values ​​and creates a processed image having a local resolution frame that includes the multiple pixel values ​​and the multiple estimated pixel values, and at least omits a subgroup of multiple pixel values ​​from the processed image.

[0011] It is understood that the foregoing general description and the following detailed description are merely exemplary and explanatory, and do not limit the claimed invention. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary aspects of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0013] Figure 1 A schematic diagram of an exemplary medical system according to aspects of this disclosure;

[0014] Figure 2A For the purposes of this disclosure Figure 1 A partial three-dimensional view of a medical device in a medical system, which includes sensors and a light source;

[0015] Figure 2B For the purposes of this disclosure Figure 1 A partial perspective view of another medical device in a medical system, which includes sensors and multiple light sources;

[0016] Figure 2C For the purposes of this disclosure Figure 1 A partial perspective view of another medical device in a medical system, which includes a pair of sensors and a light source;

[0017] Figure 3 For the purposes of this disclosure Figure 2A A schematic diagram of an exemplary image sensor for a medical device;

[0018] Figure 4 This is a schematic diagram of a frame of raw pixel data received from an image according to aspects of this disclosure, the image being... Figure 3 Image sensors are used to capture; and

[0019] Figure 5 In accordance with the aspects of this disclosure Figure 1 A block diagram of an exemplary method for imaging a target site using a medical system. Detailed Implementation

[0020] Examples of this disclosure include systems, apparatus, and methods for enhancing images of one or more target treatment sites within a subject (e.g., a patient) by emphasizing one or more features (e.g., blood vessels, vascular systems, etc.) of a target site in the processed image. Reference will now be made in detail to several aspects of this disclosure, examples of which are illustrated in the accompanying drawings. Where possible, the same or similar reference numerals will be used in the drawings to refer to the same or similar portions. When the device is introduced to a patient, the term "distal" refers to the portion furthest from the user. In contrast, when the device is placed within a subject, the term "proximal" refers to the portion closest to the user. As used herein, the terms "comprises," "comprising," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article of manufacture, or apparatus that includes a list of elements not only necessarily includes those elements but may also include other elements not expressly listed or inherent to such process, method, article of manufacture, or apparatus. The term "exemplary" is used in the sense of "example" rather than "ideal." As used herein, the terms “about,” “roughly,” and “approximately” refer to values ​​within a range of + / -10% of the stated value.

[0021] Examples of this disclosure can be used to identify target sites within a subject's body by generating processed images having local resolution frames of pixel values ​​that visually emphasize one or more features and / or characteristics of the subject's gastrointestinal tract. Such features and / or characteristics include, for example, tumors, lesions, blood vessels, changes in the mucosal surface, and so on. In some embodiments, the medical device may include image processing means including a processor and memory storing one or more algorithms for generating local resolution frames. In embodiments, the memory may include programmable instructions based on demosaicing logic, edge enhancement logic, and / or contrast enhancement logic. Additionally, the image processing means may include a user interface operable to receive user input thereon, such as, for example, filtering wavelength-shifted input of at least one color pixel value before interpolation and enhancement of other color pixel values ​​captured by an image sensor. The processed image generated by the image processing means of the medical device may include local resolution frames of pixel values ​​that may be output to a display device.

[0022] Examples of this disclosure may relate to apparatus and methods for performing various medical procedures and / or for treating portions of the large intestine (colon), small intestine, cecum, esophagus, any other part of the gastrointestinal tract, and / or any other suitable patient anatomy (collectively referred to herein as "target treatment sites"). The various examples described herein include single-use or disposable medical devices. Reference will now be made in detail to the examples of this disclosure described above and illustrated in the accompanying drawings. Wherever possible, the same reference numerals will be used in all figures to refer to the same or similar parts.

[0023] Figure 1 A schematic diagram of an exemplary medical system 100 is shown according to an example of this disclosure. The medical system 100 may include one or more light sources 130, an image processing device 101, a medical device 110, and a medical apparatus 140. The image processing device 101 may be communicatively coupled to the medical device 110 via, for example, a wired connection, a wireless connection, etc. In this example, the image processing device 101 is a computer system incorporating multiple hardware components that allow the image processing device 101 to receive data (e.g., image sensor data), process information (e.g., wavelength data), and / or generate processed images for output to a user of the medical system 100. Exemplary hardware components of the image processing device 101 may include at least one sensor 102, at least one memory 103, at least one user interface 108, and at least one display 109.

[0024] The processor 102 of the image processing apparatus 101 may include any computing device capable of executing machine-readable instructions, which may be stored on a non-transitory computer-readable medium, such as, for example, the memory 103 of the image processing apparatus 101. By way of example, the processor 102 may include a controller, integrated circuit, microchip, computer, and / or any other computer processing unit operable to perform computational and logical operations required to execute a program. As described in more detail herein, the processor 102 is configured to perform one or more operations based on instructions stored in the memory 103, such as, for example, imaging logic 104, de-mosaic logic 105, edge enhancement logic 106, contrast enhancement logic 107, etc.

[0025] Still referencing Figure 1 The memory 103 of the image processing device 101 may include a non-transitory computer-readable medium storing machine-readable instructions thereon, such as, for example, imaging logic 104, de-mosaic logic 105, edge enhancement logic 106, and contrast enhancement logic 107. Imaging logic 104 may include executable instructions that allow the medical system 100 to activate one or more components of the medical device 110 (such as, for example, one or more image sensors 150, 105A, 150B). Figures 2A to 2C And capture the original digital image.

[0026] Additionally, the demosaic logic 105 may include executable instructions that allow the medical system 100 to process digital images (e.g., mosaic images) by demosaicing the image and reconstructing missing and / or unknown pixel values ​​in the mosaic image. It should be understood that digital images captured by an image sensor (using a color filter sensor array) provide a raw image having various color pixel values ​​arranged in a mosaic pattern. Each pixel array of the pattern includes only a single color pixel value, such that one or more color pixel values ​​thereon can be omitted. As discussed in detail herein, digital images comprise a two-dimensional array of pixel values, where each pixel value at a pixel location in the image corresponds to the light intensity of one of multiple spectral bands (e.g., color pixel values).

[0027] Still referencing Figure 1 Edge enhancement logic 106 may include executable instructions that allow medical system 100 to process a mosaic image of a target area and enhance the sharpness of one or more edges in the mosaic image. It should be understood that the demosaic process may include inherent side effects, such as, for example, reducing the sharpness of one or more edges in the image. For example, the demosaic process may weaken high-frequency details of the image and / or emphasize low-frequency details of the image.

[0028] In this scenario, colored stripes may appear at the edges of significant contrast boundaries within the image, where the edges of significant contrast boundaries may include stripe artifacts within the color pixels of the mosaic image. As further described below, the edge enhancement logic 106 may include executable instructions that allow the medical system 100 to process digital images (e.g., mosaic images) by monitoring edges and increasing the detail of said edges to provide greater clarity to the image at the color pixels.

[0029] Still referencing Figure 1 The contrast enhancement logic 107 may include executable instructions that allow the medical system 100 to process a mosaic image of a target area and enhance the contrast of one or more pixels in the mosaic image. It should be understood that the demosaic process may include inherent side effects, such as, for example, reduced image contrast due to reduced color difference signals between pixels in the image.

[0030] In this scenario, the resolution frame of the color pixels may lack sufficient brightness between the individual color pixels to distinguish one or more features of the image. As further described below, the contrast enhancement logic 107 may include executable instructions that allow the medical system 100 to process digital images (e.g., mosaic images) by scaling the brightness of specific color pixels and increasing the brightness of the resolution frame, thereby providing greater clarity to the image at the color pixels.

[0031] In some embodiments, imaging logic 104, demosaic logic 105, edge enhancement logic 106, and / or contrast enhancement logic 107 may include executable instructions that allow the medical system 100 to automatically perform periodic image processing of the target area without user input. In other embodiments, image processing device 101 may be configured to receive user input to initiate image processing of the target area, such as, for example, from user interface 108 of image processing device 101. It should be understood that in some embodiments, user interface 108 may be a device integrated with image processing device 101, and in other embodiments, user interface 108 may be a remote device communicating with image processing device 101 (e.g., wireless communication, wired communication, etc.).

[0032] It should be understood that various programming algorithms and data (which support the operation of medical system 100) may reside wholly or partially in memory 103. Memory 103 may include any type of computer-readable medium suitable for storing data and algorithms, such as, for example, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive, and / or any means capable of storing machine-readable instructions. Memory 103 may include one or more sets of data, including but not limited to image data of one or more components of medical system 100 (e.g., medical device 110, medical apparatus 140, etc.).

[0033] Still referencing Figure 1 The medical device 110 may be configured to facilitate the positioning of one or more components of the medical system 100 (such as, for example, medical device 140) relative to a subject (e.g., a patient). In embodiments, the medical device 110 may be any type of endoscope, duodenoscope, gastroscope, colonoscope, ureteroscope, bronchoscope, catheter, or other delivery system, and may include a handle 112, an actuation mechanism 114, at least one port 116, and a shaft 120. The handle 112 of the medical device 110 may have one or more lumens (not shown) communicating with lumens of one or more other components of the medical system 100. The handle 112 also includes at least one port 116 leading to one or more lumens of the handle 112. As described in more detail herein, the at least one port 116 is sized and shaped to receive one or more instruments, such as, for example, medical device 140 of the medical system 100, passing through it.

[0034] The shaft 120 of the medical device 110 may include a tube that is sufficiently flexible to selectively bend, rotate, and / or twist when inserted into and / or through the tortuous anatomy of a subject to a target treatment site. The shaft 120 may have one or more lumens (not shown) extending through it, including, for example, a working lumen for receiving instruments (e.g., medical device 140). In other instances, the shaft 120 may include additional lumens, such as control wire lumens, fluid lumens, illumination lumens, and / or imaging lumens; the control wire lumen receiving one or more control wires for actuating one or more distal portions / tools (e.g., articulated joints, lifters, etc.); the fluid lumen for delivering fluid; the illumination lumen receiving at least a portion of an illumination assembly (not shown); and the imaging lumen receiving at least a portion of an imaging assembly (not shown).

[0035] Still referencing Figure 1The medical device 110 may also include a distal end 122 at the distal end of the shaft 120. In some embodiments, the distal end 122 may be attached to the distal end of the shaft 120, while in other embodiments, the distal end 122 may be integral with the shaft 120. For example, the distal end 122 may include a cap configured to receive the distal end of the shaft 120 therein. The distal end 122 may include one or more openings communicating with one or more lumens of the shaft 120. For example, the distal end 122 may include a working opening 123 through which the medical device 140 exits from a working lumen of the shaft 120. It should be understood that one or more other openings at the distal end 122 of the shaft 120 are not shown. An actuation mechanism 114 of the medical device 110 is located on the handle 112 and may include one or more knobs, buttons, levers, switches, and / or other suitable actuators. The actuation mechanism 114 is configured to at least control the deflection of the shaft 120 (e.g., by controlling the actuation of the wire).

[0036] The medical device 140 of the medical system 100 may include a catheter having a longitudinal body 142 between a proximal end 141 and a distal end 144 of the medical device 140. The longitudinal body 142 of the medical device 140 may be flexible, such that the medical device 140 is configured to bend, rotate, and / or twist when inserted into the working lumen of the medical instrument 110. The medical device 140 may include a handle at the proximal end 141 of the longitudinal body 142, which may be configured to allow the longitudinal body 142 to move, rotate, and / or bend. Additionally, the handle at the proximal end 141 of the medical device 140 may define one or more ports (not shown) sized to receive one or more tools through the longitudinal body 142 of the medical device 140.

[0037] Still referencing Figure 1 The medical device 110 may be configured to receive a medical device 140 via at least one port 116, the medical device 140 passing through a working lumen through a shaft 120 and extending to a working opening 123 at a distal end 122. In this configuration, the medical device 140 may extend distally away from the working opening 123 and into the surrounding environment of the distal end 122, such as, for example, at a target treatment site of a subject, as described in further detail below. The distal end 144 of the medical device 140 may extend distally from the distal end 122 in response to translation of the longitudinal body 142 through the working lumen of the shaft 120. The medical device 140 may include one or more end effectors (not shown) at the distal end 144 of the longitudinal body 142 for performing one or more operations at the target treatment site.

[0038] The medical device 110 may also be configured to receive one or more light sources 130, which pass through at least one of the lumen of the medical device 110 via the shaft 120 for connection to the optical fiber 146. In this example, the one or more light sources 130 are shown as components independent of the image processing device 101, such that the light sources 130 are coupled to the medical device 101 independently of the image processing device (e.g., via cable). It should be understood that in other embodiments, the one or more light sources 130 may be included on the image processing device 101, such that the light sources 130 are communicatively coupled to the medical device 110 together with the image processing device 101.

[0039] For reference Figures 2A to 2C The distal end 122 of the medical device 110 is shown according to one or more embodiments of this disclosure. Initial reference Figure 2A In one embodiment, the distal end 122 of the medical device 110 may include an optical fiber 146 and an image sensor 150 at the distal end 122. In this example, the optical fiber 146 may be coupled to one or more light sources 130 of the medical system 100, such that each of the one or more light sources 130 can transmit light through a single optical fiber 146. Although not shown, it should be understood that multiple light sources 130 may be coupled to the optical fiber 146 via an optical fiber splitter / combiner. The optical fiber 146 of the medical device 110 may be configured and operable to deliver light of various amplitudes from one or more light sources 130 distally from the distal end 122 of the axis 120. In some embodiments, the optical fiber 146 may be configured to deliver white light, ultraviolet light, near-infrared (NIR) light, and / or various other wavelengths within or outside the visible spectrum.

[0040] Still referencing Figure 2A The image sensor 150 of the medical device 110 is communicatively coupled to the image processing device 101 of the medical system 100, such as via a wired connection, a wireless connection, etc. The image sensor 150 of the medical device 110 is configured and operable to capture raw images (e.g., digital images) of the environment surrounding the end point 122 of the axis 120. In some embodiments, the image sensor 150 may include an image sensor, such as, for example, an RGB (i.e., red, green, blue) digital sensor, an RGB-Ir (red, green, blue, infrared) digital sensor, a monochrome sensor, etc. As described in further detail herein, the image sensor 150 may include one or more components for filtering colors within or outside the white light, ultraviolet light, near-infrared light, and / or the visible spectrum.

[0041] In other embodiments, reference is now made to Figure 2BThe medical device 110 may include a multicolor LED assembly at the end 122 of the shaft 120. In this case, the multicolor LED assembly may include one or more light-emitting diodes (hereinafter referred to as LEDs) 146A, 146B, 146C, 146D arranged in a circular array around the image sensor 150. Each of the LEDs 146A, 146B, 146C, 146D may be configured and operable to transmit different wavelengths and / or amplitudes (e.g., colors) relative to each other. It should be understood that different illumination sources may generate different spectra. It should be understood that the LEDs 146A, 146B, 146C, 146D of the medical device 110 may include additional and / or fewer diodes at the end 122 compared to those shown and described herein, without departing from the scope of this disclosure.

[0042] In other embodiments, reference is now made to Figure 2C The medical device 110 may include a multi-sensor assembly at the end 122 of the shaft 120. In this case, the multi-sensor assembly may include a color image sensor 150A and a monochrome image sensor 150B. As described in further detail herein, the color image sensor 150A may be configured and operable to capture a portion of the incoming light at each individual pixel location of the color image sensor 150A according to the color of the incoming light. In some embodiments, the color image sensor 150A may include, for example, an RGB (red, green, blue) digital sensor, an RGB-Ir (red, green, blue, infrared) digital sensor, etc. As described in further detail herein, the monochrome image sensor 150B may be configured and operable to completely capture all incoming light at each individual pixel location of the monochrome sensor 150B, regardless of the color of the incoming light.

[0043] For reference Figure 3 The image sensor 150 of the medical device 110 may include an outer surface 152 having a plurality of microlenses 154 disposed thereon. In some embodiments, the outer surface 152 and / or the plurality of microlenses 154 may be formed of glass, plastic and / or other transparent materials. The image sensor 150 may be a color image sensor including a color filter array 156 disposed opposite to the outer surface 152. The color filter array 156 may include a filter device having a plurality of color pixel positions 158A, 158B, 158C arranged in a predetermined pattern. In this embodiment, each of the plurality of microlenses 154 may be positioned to align with at least one of the plurality of color pixel positions 158A, 158B, 158C of the color filter array 156 disposed below the outer surface 152.

[0044] In some embodiments, the color filter array 156 may include a plurality of first color pixel positions 158A, a plurality of second color pixel positions 158B, and / or a plurality of third color pixel positions 158C. The plurality of color pixel positions 158A, 158B, 158C may be arranged in a mosaic pattern (such as, for example, a Bayer pattern) along the color filter array 156. In this example, the plurality of first color pixel positions 158A may include a red color filter, the plurality of second color pixel positions 158B may include a green color filter, and the plurality of third color pixel positions 158C may include a blue color filter. In other embodiments, the plurality of color pixels 158A, 158B, 158C may include a variety of other suitable color filters and / or patterns compared to those shown and described herein. For example, in embodiments where the image sensor 150 includes an RGB-Ir sensor, it should be understood that the color filter array 156 may additionally include a plurality of fourth color pixel positions corresponding to infrared color filters.

[0045] Still referencing Figure 3 The color filter array 156 of the image sensor 150 may be configured and operable to selectively transmit one or more wavelengths 12, 14, 16 (e.g., light intensity, spectral band, color, etc.) of the light beam 10. For example, each of the color pixel locations 158A, 158B, 158C is configured to allow delivery and / or passage of a local portion of the light beam 10 (e.g., at least one wavelength 12, 14, 16), depending on the color filter at said color pixel location 158A, 158B, 158C. Thus, only a single color component (e.g., color pixel value) can be measured by the image sensor 150 at each color pixel location 158A, 158B, 158C. As further described herein, the color pixel value may include a certain amount of electrical energy within a color range of the spectrum (e.g., red range, green range, blue range, etc.).

[0046] In this example, each of the plurality of color pixel locations 158A, 158B, 158C allows only one wavelength 12, 14, 16 of the light beam 10 to pass through the color filter array 156. The image sensor 150 may also include a photosensor array 160 disposed opposite to the color filter array 156, such that the color filter array 156 of the image sensor 150 may be located between the outer surface 152 and the photosensor array 160. The photosensor array 160 of the image sensor 150 may include a photodiode (e.g., a semiconductor device) having a plurality of photosensitive elements 162 and circuitry 164 communicatively coupled to the plurality of photosensitive elements 162.

[0047] Still referencing Figure 3A plurality of photosensitive elements 162 are arranged in an array (e.g., a grid), wherein each of the plurality of photosensitive elements 162 is positioned aligned with at least one of a plurality of color pixel positions 158A, 158B, 158C of a color filter array 156 disposed on the photosensitive array 160. The photosensitive array 160 may be configured and operable to convert a light beam 10 received through the outer surface 152 and the color filter array 156 into an electric current. In this example, an electric current may be generated by the photosensitive array 160 when photons of the received light are absorbed in the plurality of photosensitive elements 162.

[0048] In this example, each of the plurality of photosensitive elements 162 may be configured to measure the amount of a single color pixel value (e.g., red, green, blue) in the incoming light 10 at a position along the surface of the photosensitive element 162. Thus, the plurality of photosensitive elements 162 may capture the incoming light 10 and may generate an electrical signal that is quantized and stored as a value in the resulting processed image file. It should be understood that the photosensitive array 160 may include various suitable shapes, sizes, and / or configurations different from those shown and described herein. In other embodiments, the image sensor 150 may be a monochrome sensor (e.g., monochrome sensor 150B), such that the color filter array 156 shown and described above may be completely omitted between the outer surface 152 and the photosensitive array 160. In this case, each photosensitive element 162 along the photosensitive array 160 is operable to receive, capture, and absorb all three wavelengths 12, 14, 16 of the light beam 10.

[0049] For reference Figures 3 to 4 , combined Figure 5 The flowchart schematically illustrates an exemplary method 200 for generating processed images of a target area using a medical system 100. Figures 3 to 5 The descriptions and accompanying descriptions below are not intended to limit the subject matter described herein to any particular method.

[0050] Initially, reference Figure 1The medical device 110 of the medical system 100 can be inserted into the body of a subject (not shown) to position the distal end 122 adjacent to a target site. For example, the shaft 120 can be guided through the digestive tract of the subject (e.g., a patient) by inserting the distal end 122 into the nose or mouth (or other suitable natural body opening) of the subject's body, and through the gastrointestinal tract (e.g., esophagus, stomach, small intestine, etc.) of the subject's body until reaching the target site. It should be understood that the length of the shaft 120 may be sufficient such that the proximal end of the medical device 110 (including the handle 112) is outside the subject, while the distal end 122 of the medical device 110 is inside the subject's body. Although this disclosure relates to the use of the medical system 100 in the digestive tract of a subject, it should be understood that the features of this disclosure can be used in various other locations within the subject's body (e.g., other organs, tissues, etc.).

[0051] Furthermore, since the medical device 110 is received within the subject's body and the distal end 122 of the shaft 120 is positioned relative to the target site, the medical device 140 may be received within the medical device 110 via at least one port 116. In this case, the longitudinal body 142 of the medical device 140 translates through the shaft 120 via at least one of the lumen of the shaft 120 (e.g., the working lumen). The distal end 144 of the longitudinal body 142 may extend distally from the distal end 122 of the shaft 120 via a working opening 123 communicating with the working lumen of the shaft 120. It should be understood that this step may be optional, such that receiving the medical device 140 through the medical device 110 may be performed in various other steps of method 200 and / or may be completely omitted. The distal end 122 may be positioned adjacent to and facing the target treatment site.

[0052] refer to Figure 5 In step 202, one or more target objects may be illuminated by medical device 110 in response to the processor 102 of image processing device 101, and image processing device 101 executes imaging logic 104 to activate one or more light sources 130. The medical device 110 includes an optical fiber 146. Figure 2A ) and / or multicolor LED components 146A, 146B, 146C, 146D ( Figure 2B In one example, light from one or more light sources 130 can be emitted from the medical device 110 at the end 122 to illuminate the target object.

[0053] In step 204, since the target object is illuminated by the light from the medical device 110, the image sensor 150 can be activated by the processor 102, which executes imaging logic 104 to capture one or more raw digital images of the target object. It should be understood that the processor 102 of the image processing device 101 can be communicatively coupled to the image sensor 150 of the medical device 110 via circuitry 164. For example, referencing... Figure 3 The light 10 transmitted to the target object by the optical fiber 146 and / or multi-color LED components 146A, 146B, 146C, 146D can be reflected away from the target object and received at the image sensor 150. In this case, multiple wavelengths 12, 14, 16 of the light 10 can be received on the outer surface 152 by one or more of the multiple microlenses 154.

[0054] Multiple wavelengths 12, 14, 16 may be received at one or more corresponding color pixel positions 158A, 158B, 158C of the color filter array 156, such as those aligned with microlens 154, which receives the light beam 10 passing through it. Depending on the color filters at the color pixel positions 158A, 158B, 158C, one or more of the multiple wavelengths 12, 14, 16 of the light 10 at each of the multiple color pixel positions 158A, 158B, 158C may be suppressed (e.g., filtered, excluded, omitted, blocked) from passing through the color filter array 156. Therefore, the color filter configuration (e.g., red, green, blue, etc.) of the color pixel positions 158A, 158B, 158C receiving the light 10 determines which of the wavelengths 12, 14, 16 (e.g., red, blue, green, etc.) is allowed to pass through the color filter array 156 at said position.

[0055] Therefore, it should be understood that each of the color pixel positions 158A, 158B, and 158C may allow only about one-third (e.g., 33%) of the incoming light 10 to pass through to the photosensitive array 160 at said position. For example, at each of the plurality of first color pixel positions 158A (e.g., a red filter), only wavelength 12 of light 10 (e.g., the spectrum in the red range) is allowed to pass through the filter array 156, such that wavelengths 14 and 16 (e.g., blue and green, respectively) can be filtered by the filter array 156 at the first color pixel positions 158A. It should be understood that, in this embodiment, the plurality of second color pixel positions 158B (e.g., a green filter) may be configured to allow wavelength 14 (e.g., the spectrum in the green range), and the plurality of third color pixel positions 158C (e.g., a blue filter) may be configured to allow wavelength 16 (e.g., the spectrum in the blue range).

[0056] Still referencing Figure 3Individual wavelengths 12, 14, 16 of the light 10 passing through the color filter array 156 can be detected along the photosensitive array 160 and absorbed by one or more of the plurality of photosensitive elements 162 (i.e., those aligned with the color pixel positions 158A, 158B, 158C of the light beam 10 received therethrough). In this case, a portion of the light 10 absorbed by each of the plurality of photosensitive elements 162 can be converted into an electric current. The raw digital image captured by the image sensor 150 may include a quantitative record of the light energy measured at each grid position of the photosensitive elements 162 along the photosensitive array 160, wherein each photosensitive element 162 is configured to determine the color pixel value of wavelengths 12, 14, 16 received therethrough.

[0057] In this configuration, when the image processing unit 102 executes the imaging logic 104, it causes the photosensitive array 160 (e.g., via circuitry 164) to transmit electrical signals of color pixel values ​​to the image processing unit 101. These electrical signals of color pixel values ​​can be stored in the memory 103 of the image processing unit 101 and utilized by the de-mosaic logic 105, edge enhancement logic 106, and / or contrast enhancement logic 107 to generate a processed image.

[0058] refer to Figure 5 In step 206, the shift wavelength input can be input by a user of the medical system 100 (e.g., via the user interface 108 of the image processing device 101). The shift wavelength input may include identifiers of one or more light wavelengths, amplitudes, color pixel values, etc., for use in modifying the demosaic process of the raw digital image captured by the image sensor 150. It should be understood that in some embodiments, step 206 may occur prior to steps 202 and 204, and / or may be automatically programmed in the memory 103 of the image processing device 101 such that method 200 can proceed to step 208 with a predetermined shift wavelength input.

[0059] For reference Figure 4 The raw digital image received at the image processing device 101 may include frames (e.g., grids) of raw pixels 20, which are formed by a plurality of color pixel values ​​22A, 22B, 22C measured by the image sensor 150. It should be understood that each grid position along the frame of raw pixels 20 may correspond to a position of the photosensitive element 162 on the photosensitive array 160. Therefore, based on the relative position of the grid position with respect to the color pixel positions 158A, 158B, 158C of the color filter array 156 (through which the photosensitive element 162 receives wavelengths 12, 14, 16), each grid position on the frame of raw pixels 20 includes at least one of a first color pixel value 22A, a second color pixel value 22B, or a third color pixel value 22C.

[0060] In step 208, the processor 102 may separate the original pixels 20 based on the shift wavelength input received in step 206 using color pixel values ​​22A, 22B, 22C, and filter out (e.g., exclude) at least one of the first color pixel value 22A, the second color pixel value 22B, or the third color pixel value 22C. In other words, the image processing apparatus 101 is configured to shift the demosaic process of the original digital image captured by the image sensor 150 from the plurality of color pixel values ​​22A, 22B, 22C in the frame of the original pixel values ​​20 to a subgroup of color pixel values ​​22A, 22B, 22C.

[0061] Therefore, as described in detail below, the demosaic process (steps 210A, 210B), edge enhancement process (steps 212A, 212B), and / or contrast enhancement process (steps 214A, 214B) of the original digital image are performed only on a local portion of the frame of the original pixel values ​​20, which are included in the original digital image captured by the image sensor 150. In this example, the processor 102 of the image processing apparatus 101 removes a plurality of first color pixel values ​​22A according to the shift wavelength input, such that the resulting processed image generated by the image processing apparatus 101 includes a local resolution frame of pixels.

[0062] For example, still refer to Figure 5 In step 210A, processor 102 may perform a demosaic process for multiple second color pixel values ​​22B in step 210A (when demosaic logic 105 is executed) to calculate an inferred measurement of the second color pixel value 22B (e.g., green) at a grid location along the frame of the original pixel 20 (e.g., a grid location receiving wavelength 12 (e.g., red) and wavelength 16 (e.g., blue)) where wavelength 14 (e.g., green) is not received. By interpolating from the measurement of the known second color pixel value 22B, processor 102 may infer the measurement of the unknown second color pixel value 22B at a grid location on the frame of the original pixel value 20 where wavelength 14 is not received.

[0063] In this example, processor 102, performing demosaic logic 105, interpolates the missing second color pixel value 22B from adjacent (e.g., neighboring) grid locations, including the measured second color pixel value 22B. Processor 102 determines the measurement of the unknown second color pixel value 22B from the neighboring grid locations of the received wavelength 14. It should be understood that processor 102 may perform demosaic logic 105 to reconstruct the unknown and / or missing third color pixel value 22C along the frame of the original pixel value 20 in step 210B in a manner substantially similar to that described above with respect to step 210A.

[0064] Still referencing Figure 5 In step 212A, the processor 102 of the image processing apparatus 101 may execute edge enhancement logic 106 to further reconstruct the original digital image captured by the image sensor 150. In this example, the edge enhancement logic 106 (when executed by the processor 102) may enhance the detail and / or sharpness of one or more edges in the frame of original pixel values ​​20. For example, the processor 102 may detect the position of one or more edges within each grid location in the frame of original pixel values ​​20 and minimize the noise level around the edges by adjusting one or more of a plurality of second color pixel values ​​22B at the grid location. In some embodiments, the processor 102 may increase the sharpness of one or more edges in the grid location by enhancing the gradient (e.g., magnitude) of the plurality of second color pixel values ​​22B.

[0065] It should be understood that processor 102 may execute edge enhancement logic 106 to reconstruct the edges of the digital image by adjusting one or more of a plurality of third pixel values ​​22C along the frame of the original pixel value 20 in a manner substantially similar to that described above with respect to step 212A in step 212B. It should be understood that in other embodiments, various other suitable edge enhancement procedures may be included in edge enhancement logic 106 and executed by processor 102.

[0066] Still referencing Figure 5 In step 214A, the processor 102 of the image processing apparatus 101 may execute contrast enhancement logic 107 to further reconstruct the original digital image captured by the image sensor 150. In this example, the contrast enhancement logic 107 (when executed by the processor 102) may enhance the representational contrast in the frame of the original pixel values ​​20. For example, the processor 102 may increase the luminance component (e.g., brightness) of one or more of the plurality of second color pixel values ​​22B within each grid location in the frame of the original pixel values ​​20. In some embodiments, the processor 102 may modify the luminance of one or more grid locations within the frame of the original pixel values ​​20 by reducing the second color pixel values ​​22B located therein, thereby reducing the contrast contribution provided by the second color pixel values ​​22B.

[0067] It should be understood that processor 102 may execute contrast enhancement logic 107 to increase the local contrast of the digital image by adjusting one or more of a plurality of third color pixel values ​​22C along the frame of the original pixel value 20 in a manner substantially similar to that described above with respect to step 214A in step 214B. It should be understood that in other embodiments, various other suitable contrast enhancement processes may be included in contrast enhancement logic 107 and executed by processor 102.

[0068] Still referencing Figure 5In step 216, the processor 102 of the image processing device 101 may determine whether the current iteration of the demosaic process (steps 210A, 210B), the edge enhancement process (steps 212A, 212B), and the contrast enhancement process (steps 214A, 214B) is equal to or greater than a predetermined and / or dynamic iteration threshold. In some embodiments, the predetermined iteration threshold may be predefined and stored in the memory 103 of the image processing device 101, or selectively input by the user of the medical system 100.

[0069] In other embodiments, the iteration threshold may be dynamically determined by processor 102 from the initial transformation of the digital image (through a demosaicing process, an edge enhancement process, and a contrast enhancement process) based on one or more factors, including, for example, peak definition, contrast, and visibility values. In this case, data indicating the initial state of the original pixel values ​​20 (e.g., a histogram of the frame) may be analyzed when processor 102 completes the first iteration, and the peak value may be determined with respect to sharpness, contrast, and visibility characteristics. Therefore, processor 102 may continuously determine the current sharpness, contrast, and visibility of the digital image as it completes each iteration of these processes relative to the initial peak value (i.e., the dynamic iteration threshold).

[0070] In response to processor 102 determining in step 216 that the current iteration of method 200 is less than a predetermined (or dynamic) threshold, image processing device 101 may be configured and operable to return to steps 210A, 210B to perform one or more of the demosaic process, edge enhancement process, and / or contrast enhancement process. In response to processor 102 determining in step 216 that the current iteration of method 200 is at least equal to or greater than the predetermined (or dynamic) threshold, image processing device 101 may be configured and operable to produce an interpolated output image. It should be understood that an image with improved sharpness, contrast, and / or visibility may be provided in response to iterative enhancement of frames of raw pixel values ​​20 initially captured by image sensor 150.

[0071] Still referencing Figure 5In step 218, due to the process of the method 200 shown and described herein, the processor 102 of the image processing apparatus 101 can generate a processed image. In this example, the output image may include local resolution frames of color pixel values, such as, for example, a plurality of second color pixel values ​​22B and a plurality of third color pixel values ​​22C. Therefore, the digital image generated by the processor 102 of the image processing apparatus 101 may omit at least a plurality of first color pixel values ​​22A from one or more of the grid positions of the frames of original pixel values ​​20 initially captured by the image sensor 150. In this example, the generated image excludes the first color pixel values ​​22A in each grid position of the frames of original pixel values ​​20, such that the image processing apparatus 101 generates a local resolution image.

[0072] The display 109 of the medical system 100 is communicatively connected to a processor 102 of the image processing apparatus 101. The processor 102 is operable to transmit local resolution images to the display 109 for viewing by a user of the medical system 100. In some instances, the medical system 100 may be configured and operable to continuously perform the methods 200 shown and described herein, such that the display 109 may output multiple local resolution images to provide continuous (e.g., on-site, real-time, etc.) imaging of one or more target objects.

[0073] It should be understood that removing at least one of the multiple color pixel values ​​22A, 22B, 22C initially captured (by image sensor 150) in the frame of the original pixel value 20 from the processed image can facilitate the differentiation of one or more features and / or structures (e.g., target objects) at the target treatment site. For example, improved distinguishability of blood vessels from surrounding tissue can be provided in response to the shift wavelength input in step 206 by processor 102, which filters out at least one of the color pixel values ​​22A, 22B, 22C. In some instances, the primary colors of one or more target objects may be substantially similar to each other, such as tissues and blood vessels generally appearing as red, thereby minimizing the distinction of said features in the generated image. In this case, emphasizing blue and / or green color pixels (e.g., second color pixel value 22B and third color pixel value 22C, respectively) can enhance the distinction of said features.

[0074] Furthermore, it should be understood that one or more color components may include lower contrast (e.g., red) compared to one or more other color components (e.g., green, blue) within the target treatment area. Excluding one or more color components (e.g., first color pixel value 22A (e.g., red)) from the resulting digital image can enhance the subject's vascular system, since only the second color pixel value 22B (e.g., blue) and the third color pixel value 22C (e.g., green) can be maintained from the frame of the original pixel value 20.

[0075] In an embodiment where the optical fiber 146 of the medical device 140 is operable to generate ultraviolet light, the medical system 100 can be configured to generate local resolution frames capable of distinguishing desired (e.g., healthy) tissue from undesirable (e.g., unhealthy) tissue because the tissue may fluoresce in different colors under ultraviolet illumination. In this example, the shift wavelength input in step 206 may include a first color pixel value 22A (e.g., red) and a third color pixel value 22C (e.g., green), such that only a second color pixel value 22B (e.g., blue) may be included in the resulting processed image.

[0076] In embodiments where the image sensor 150 of the medical device 110 includes an RGB-Ir sensor, the frames of raw pixel values ​​20 detected and captured by the image sensor may include a plurality of first color pixel values ​​22A (e.g., red), a plurality of second color pixel values ​​22B (e.g., blue), a plurality of third color pixel values ​​22C (e.g., green), and a plurality of fourth color pixel values ​​(e.g., infrared), which extend beyond the visible spectrum and are closer to the infrared spectrum. In this case, the medical system 100 may be configured to generate local resolution frames that can enhance darker areas within a target treatment site (e.g., body cavity) by emphasizing the first color pixel values ​​22A, the third color pixel values ​​22C, and / or the fourth color pixel values. Therefore, the shift wavelength input in step 206 may include second color pixel values ​​22B (e.g., blue) for removal from the processed image.

[0077] In embodiments where the end 122 of the central axis 120 includes a color image sensor 150A (e.g., an RGB-Ir sensor) and a monochrome image sensor 150B, the image processing device 101 can be configured to generate a local resolution image captured by the color image sensor 150A, while also enhancing the contrast of darker areas within the frame of the original pixel value 20 using the monochrome image sensor 150B. In this case, the monochrome image sensor 150B can be sensitive to near-infrared wavelengths illuminated by light 10, making the fourth color pixel value (e.g., infrared) easily detectable. It should be understood that providing one or more materials to the target treatment site can facilitate the visualization of one or more target objects through the monochrome image sensor 150B, such as, for example, fluorescent dyes.

[0078] Each of the aforementioned systems, devices, components, and methods can be used to generate local resolution frames of pixel values ​​for a subject. By providing a medical device that includes an image processing system (which stores shifted wavelength demosaic logic), a user can enhance the visualization of one or more features and / or properties of a target site within the subject's body during surgery without manipulating a light source. The medical device allows the user to accurately identify the location of the target site, thereby reducing overall surgical time, increasing surgical efficiency, and avoiding unnecessary harm to the subject's body caused by inaccurate positioning of the target object in the treatment area.

[0079] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed apparatus and methods without departing from the scope of this disclosure. It should be understood that the disclosed apparatus may include various suitable computer systems and / or computing units incorporating multiple hardware components (such as, for example, a processor and non-transitory computer-readable media) that allow the apparatus to perform one or more operations during surgery, according to those disclosed herein. Other aspects of this disclosure will be apparent to those skilled in the art in light of the practice of the specifications and features disclosed herein. It is contemplated that the specifications and examples are to be considered exemplary only.

Claims

1. A medical device comprising: axis; A sensor coupled to the distal end of the shaft and including a filter array, wherein the sensor is configured to capture a raw image and the filter array is configured to filter the raw image into frames of raw pixel values, the frames of raw pixel values ​​including a plurality of first pixel values, a plurality of second pixel values ​​and a plurality of third pixel values; and A processor and a non-transitory computer-readable medium, the non-transitory computer-readable medium storing demosaic instructions, which, when executed by the processor, cause the processor to: Exclude the plurality of first pixel values ​​from the frame containing the original pixel values; A plurality of estimated second pixel values ​​are generated at the positions of the plurality of first pixel values ​​and the plurality of third pixel values ​​on the frame; A plurality of estimated third pixel values ​​are generated at the positions of the plurality of first pixel values ​​and the plurality of second pixel values ​​on the frame; and A processed image is created from the plurality of second pixel values, the plurality of estimated second pixel values, the plurality of third pixel values, and the plurality of estimated third pixel values, the processed image having a local resolution frame.

2. The medical device of claim 1, wherein the demosaic instructions stored in the non-transitory computer-readable medium cause the processor to: Detect one or more edges within the original image; and Perform sharpening enhancement on one or more edges to increase edge details in the processed image.

3. The medical device of claim 2, wherein the demosaic instructions stored in the non-transitory computer-readable medium cause the processor to: Output the sharpened and enhanced image created from the sharpening and enhancement steps described above; and The sharpened and enhanced image is then blended with the processed image.

4. The medical device of claim 3, wherein the demosaic instructions stored in the non-transitory computer-readable medium cause the processor to: Set the brightness value of each of the plurality of second pixel values ​​and the plurality of third pixel values; and The contrast of the processed image is increased by performing contrast enhancement on the plurality of second pixel values ​​and the plurality of third pixel values ​​by modifying the brightness values.

5. The medical device of claim 4, wherein the demosaic instructions stored in the non-transitory computer-readable medium cause the processor to: Output the contrast-enhanced image created from the contrast enhancement step; and The contrast-enhanced image is then blended with the processed image.

6. The medical device according to claim 5, wherein: Repeat one or more of the following: demosaicing, sharpening enhancement, or contrast enhancement, until one or more predetermined thresholds associated with the processed image are met.

7. The medical device according to any one of claims 1-6, wherein the demosaic instructions stored in the non-transitory computer-readable medium cause the processor to: Receive a shift wavelength input to determine the color pixel value of the plurality of first pixel values.

8. The medical device of claim 7, further comprising a user interface communicatively connected to the processor and configured to transmit a signal indicating the shift wavelength input to the processor.

9. The medical device according to any one of claims 1-8, wherein the sensor comprises an RGB image sensor, and the filter array comprises a red-green-blue Bayer color filter array; and The plurality of first pixel values ​​include red pixel values, the plurality of second pixel values ​​include blue pixel values, and the plurality of third pixel values ​​include green pixel values.

10. The medical device according to any one of claims 1-8, wherein the sensor comprises an RGB+Ir image sensor, and the filter array comprises a red-green-blue-infrared Bayer color filter array.

11. The medical device according to any one of claims 1-9, wherein the sensor comprises an RGB image sensor and a monochrome sensor.

12. The medical device according to any one of claims 1-11, wherein each pixel location in the local resolution frame includes a captured color pixel value and a reconstructed color pixel value, such that at least one color pixel value is excluded from the frame of the original pixel value.

13. The medical device according to any one of claims 1-12, further comprising a light source coupled to the distal end of the shaft, wherein the light source is an optical fiber, an ultraviolet light source, or a multicolor LED array.

14. The medical device according to any one of claims 1-13, wherein the demosaic instructions stored in the non-transitory computer-readable medium cause the processor to: The processed image of the local resolution frame is output to the display device.

Images