Method and apparatus for gamma correction

Abstract

A medical system including a shaft having a proximal end and a distal end, an imaging device at the distal end of the shaft, and a controller that receives image data including pixel data from the imaging device, the pixel data including a plurality of individual pixel values; converts the pixel data from an RGB format to a YC b C r format; and forms adjusted pixel data by applying a gain to the pixel data, wherein the gain is based on a Y value, a C b value, and a C r value of the individual pixel values of the converted pixel data.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 060,885, filed August 4, 2020, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to medical devices and methods of use. More specifically, in some embodiments, this disclosure relates to endoscopic imaging tools and methods that involve enhancing images generated by imaging devices associated with an endoscope. Background Technology

[0004] Medical instruments (such as endoscopes or catheters) may include associated imaging devices, such as cameras for imaging body tissues or cameras associated with a medical endoscope. Disadvantages of endoscopes using such imaging devices include, for example, intensity gradients resulting from changes in the distance between the target object and the body, and / or shadows created by other structures within the body that block light from reaching the target. Light control exists for globally adjusting the light output from a light source (e.g., a light-emitting diode, LED) to the target area. However, if the light output increases, changing the amount of light from the light source may increase hotspots, or if the light output decreases, it may increase shadows. Therefore, there is a need to adjust local areas of the image to improve the overall visibility of the entire image. This disclosure can solve one or more of these or other problems in the art. However, the scope of this disclosure is defined by the appended claims, not by the ability to solve the specific problems. Summary of the Invention

[0005] According to one aspect, a medical system includes an axis having a proximal end and a distal end, an imaging device located at the distal end of the axis, and a controller, wherein the controller is configured to: receive image data including pixel data from the imaging device, the pixel data including a plurality of individual pixel values; and convert the pixel data from RGB format to YC format. b C r Format; and adjusted pixel data formed by applying gain to pixel data, wherein the gain is based on the Y value, C value, and C value of the individual pixel values ​​of the transformed pixel data. b Value and C r value.

[0006] The controller can also be configured to calculate the magnitude (e.g., intensity) of each pixel value, where the magnitude of each pixel value can be equal to... Furthermore, the gain of an individual pixel is a function of the magnitude of all pixels.

[0007] This gain can include a first gain and a second gain, and the first gain can be applied to the Y value, while the second gain can be applied to the C value corresponding to the pixel value. b Value and C r Both are worthwhile.

[0008] The medical system may also include a memory that may include a look-up table (LUT) for each of the luminance and chroma values.

[0009] The controller can also be configured to: transfer adjusted pixel data from YC b C r Convert the format to RGB format.

[0010] The medical system may also include a display, and the controller may be configured to generate an image based on adjusted and transformed pixel data.

[0011] The medical system may also include a light-emitting device located at the distal end of the axis.

[0012] The medical system may also include a user input device configured to receive at least one user input for controlling the medical system.

[0013] The controller may include a field-programmable gate array (FPGA).

[0014] The medical system may also include a handle, wherein the controller may be mounted on or in the handle.

[0015] According to one aspect, a method for controlling an imaging device for a medical system includes: receiving an image comprising pixel data from the imaging device of the medical system, the pixel data comprising a plurality of individual pixel values; converting the pixel data from an RGB format to a second multichannel color format having brightness values ​​and color values; calculating an amplitude for each pixel value, wherein the gain amplitude is based on the brightness value and color value of each corresponding pixel; applying the gain amplitude to adjust the corresponding brightness or corresponding color of each pixel; and converting each of the plurality of pixels from the second multichannel color format to an RGB format.

[0016] The method may also include applying a gain to the brightness value that is different from the color value of the corresponding pixel value.

[0017] The method may also include generating an image on a display of a medical system based on adjusted and transformed pixel data.

[0018] The method may further include: applying the gain to each pixel value by: matching the magnitude with a corresponding value in a luminance lookup table (LUT) for the corresponding luminance value, and matching the gain magnitude with a corresponding value in a color LUT for the corresponding color value.

[0019] Gain magnitude can be applied to each of the brightness and color values ​​of the corresponding pixel.

[0020] According to one aspect, a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for processing electronic images from a medical system, the method comprising: receiving an image including pixel data from an imaging device of the medical system, the pixel data including a plurality of individual pixel values; converting the pixel data from RGB format to YC format. b C r The format; and the calculation of the magnitude of each pixel value, where the gain magnitude of each pixel is equal to Furthermore, the gain magnitude is based on the Y value and C value of each corresponding pixel. b Value and C r Value; apply gain magnitude to adjust the corresponding brightness or corresponding chromaticity of each pixel; and transfer each of multiple pixels from YC b C r Convert the format to RGB format.

[0021] The method may further include: applying a C value to the Y value corresponding to the pixel value. b Value and C r Different values ​​of gain.

[0022] The method may also include generating an image on a display of a medical system based on adjusted and transformed pixel data.

[0023] Applying gain to each pixel value can include: matching the magnitude with the corresponding value in the luminance lookup table (LUT) for the corresponding Y value, and for the corresponding C value... b Value and corresponding C r The value matches the gain magnitude with the corresponding value in the chromaticity LUT.

[0024] This gain magnitude can be applied to each of the luminance and chrominance values ​​of the corresponding pixel. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of the specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments.

[0026] Figure 1 A schematic diagram of a medical system according to an embodiment is shown;

[0027] Figure 2 yes Figure 1 A block diagram of a controller with inputs and outputs for a medical system;

[0028] Figure 3 It is an enhancement Figure 1 A flowchart of a method for creating images in a medical system;

[0029] Figure 4A It shows the result of Figure 1 An example of an image obtained by a medical system without performing modifications;

[0030] Figure 4B It shows the result of Figure 1 An example of an image obtained by a medical system after modification; and

[0031] Figure 5 It is possible to base on Figure 1 The technology described in the medical system is used in an example system. Detailed Implementation

[0032] This disclosure is described with reference to exemplary medical systems and imaging apparatuses for imaging target sites and improving the overall brightness of one or more images produced by the imager of the target site. However, it should be noted that references to any particular procedure are provided for convenience only and are not intended to limit this disclosure. Those skilled in the art will recognize that the basic concepts of the disclosed apparatus and application methods can be used in any suitable medical or other procedure. This disclosure can be understood with reference to the following description and accompanying drawings, wherein like elements are referred to by like reference numerals.

[0033] For ease of description, a portion of the device and / or its components are referred to as the proximal portion and the distal portion. It should be noted that the term "proximal" refers to the portion closer to the user of the device, while the term "distal" in this document refers to the portion further away from the user. Similarly, "extending distally" indicates that the component extends in a distal direction, while "extending proximally" indicates that the component extends in a proximal direction. As used herein, the terms "about," "approximately," and "substantially" indicate a range of values ​​within + / - 10% of the calibrated or implied value. Furthermore, terms indicating component / surface geometry refer only to approximate shapes.

[0034] Figure 1A medical system 10 is schematically illustrated, comprising a handle 20, an access sheath 30 (shown in cross-section), a controller 50 located in or on the handle 20 (position not limited thereto), a display 60, and a user input device (e.g., a graphical user interface (GUI) 70). The medical system 10 may be an endoscope system for imaging and / or providing access to target sites within the body. For example, the medical system 10 may be Boston Scientific's SpyGlass. TM DM system, ureteroscope, renal sheath, hysteroscopic sheath, cystoscopic sheath, maneuverable sheath or other endoscopes.

[0035] Access sheath 30 can be, for example, a ureteral access sheath (e.g., Boston Scientific's Navigator). TM Ureteral access sheath, renal sheath, hysteroscopic sheath, cystoscopic sheath, maneuverable sheath, or other suitable access sheath. The access sheath 30 may be formed from an extrusion having an outer sheath 32, and may have different diameters and lengths depending on the medical procedure and the body cavity accessed during the procedure. The access sheath 30 may be inserted into a body opening (e.g., a natural orifice or incision) and advanced to the target site to perform the medical procedure.

[0036] Access sheath 30 may include an imaging cavity 34 and an optical cavity 36. A sensor or imaging device 40 (e.g., a camera) may be fixed at the distal end of the imaging cavity 34. An imaging cable 42 may extend from the imaging device 40 proximally and may connect the imaging device 40 to a display 60, a controller 50, and / or any other associated device. A light-emitting device 44 (e.g., an LED) may be fixed at the distal end of the optical cavity 36. An optical cable 46 may extend from the light-emitting device 44 proximally to the controller 50, a user input device 70, and / or any other associated device. Figure 1 As shown, the access sheath 30 may include a working channel 38 through which medical instruments or other tools may be inserted. It will be understood that the imaging device 40 and / or the light-emitting device 44 do not need to be fixed at the distal end of the access sheath 30, and each of the imaging device 40 and / or the light-emitting device 44 can move within the respective cavities 34 and 36. Alternatively, the light-emitting device 44 and the imaging device 40 may be formed as a single unit and may be disposed within and / or distal to a single cavity.

[0037] The controller 50 can control the display of images on the imaging device 40, the light-emitting device 44, and / or the display 60. The controller 50 may be located on or in the handle 20, or may be located near the handle 20, and may include one or more processors and / or memories (e.g., ...). Figure 5The memory 80 in the memory). The controller 50 can be configured to implement software and / or instructions (e.g., memory 80). Figure 5 The controller 50 may also receive instructions (110) from the user input device 70 to perform methods for modifying one or more images, as described herein. Figure 2 As shown, controller 50 may include a first color converter 52, a gamma corrector 54, and a second color converter 56. Controller 50 may receive input from imaging device 40. For example, controller 50 may receive one or more images in RGB format from imaging device 40. As will be explained herein, the first color converter 52 may convert the RGB format image to different multichannel color spaces or color formats (e.g., YC). b C r The image (in format). Gamma corrector 54 then applied to YC. b C r The image undergoes an image adjustment process, as will be explained herein. The processed image is then converted back to RGB format by a second color converter 56. The controller 50 can output the processed RGB format image to a display 60. One or more processors of the controller 50 can execute the processes of the first color converter 52, the gamma corrector 54, and the second color converter 56.

[0038] The controller 50 modifies or adjusts an image by changing both the chroma (e.g., color) and luminance or lightness values ​​of pixels in the image using a gamma corrector 54 through a non-linear gain change. That is, the luminance or lightness value of a pixel can be adjusted separately from the chroma, hue, or saturation values ​​of the same pixel. For example, the controller 50 calculates the amplitude of a pixel. The amplitude value M is calculated based on a formula, for example, the following formula: In this way, during the conversion from RGB format input, the amplitude value M takes into account the Y and C values ​​of the pixel. b and C r The negative value. Although the square root formula has been described, it will be understood that the amplitude value can be calculated or determined by the controller 50 using another formula.

[0039] The controller 50 also uses a corresponding amplitude M to determine the luminance (luma) gain and chrominance (e.g., color) gain for each pixel. The controller 50 includes an associated storage device (e.g., Figure 5 Storage device 80), on which multiple lookup tables (LUTs) are stored (e.g., Figure 5(LUT 120 in the example). It will be understood that a LUT can be an equation, a mapping, etc. According to the example, the first LUT includes multiple luminance gain values ​​associated with multiple amplitude values ​​M. The second LUT includes multiple chrominance gain values ​​associated with multiple amplitude values ​​M. Once the luminance and chrominance gains are determined, the controller 50 applies the gains to the luminance values. For example, the original luminance value can be multiplied by the gain from the luminance LUT corresponding to the amplitude value M. The controller 50 also applies the gains to each corresponding chrominance value. For example, the original chrominance value can be multiplied by the gain from the chrominance LUT corresponding to the amplitude value M. In this way, the luminance and chrominance gains for each pixel are varied using the same amplitude value M. The amplitude value M can be associated with the same or different gain values ​​in each of the luminance and chrominance LUTs.

[0040] In some examples, the LUT may provide almost no modification to the original luminance or chrominance values ​​(e.g., the gain value may be small or zero), based on the brightness of the pixels associated with those original values. In other examples, the maximum luminance gain may be approximately 4.5 times the original luminance value, and the maximum chrominance gain may be approximately 6 times the original chrominance value. That is, the brightness of the original values ​​can be appropriate, and the luminance and chrominance values ​​may not be modified. Alternatively, the luminance and chrominance values ​​may be such that one or both of the luminance and chrominance values ​​can be multiplied by a multiplication factor to correct the overall brightness of the pixels. It will be understood that the first and second LUTs may be generated during development and may be stored in the memory of the controller 50. Alternatively or additionally, the LUTs may be updated via software or hardware updates. It will also be understood that the maximum gain value is merely an example and may be modified based on medical systems, imaging equipment, and / or other parameters.

[0041] According to the example, controller 50 is a field-programmable gate array (FGPA). Controller 50 is positioned on medical system 10 (e.g., in or on handpiece 20) such that image data generated by imaging device 40 is processed by controller 50, rather than by a general processing unit (GPU) (such as a server located remotely from medical system 10). Processing the image data by controller 50 minimizes the transmission time between imaging device 40, controller 50, and display 60, allowing the image data to be modified within a latency requirement of approximately 150 ms or less. If the image data were transmitted to the GPU, the image data might not be able to be modified within the latency requirements of medical system 10.

[0042] Figure 3This is a flowchart of a method 300 for enhancing medical images according to various aspects of this disclosure. At step 302, image data generated by imaging device 40 may be received by controller 50. The data generated by imaging device 40 may be provided in RGB format. Subsequently, in step 304, controller 50 converts the image data, including data with multiple individual pixel values, from RGB pixel format to YC format via a first color converter 52. b C r Pixel format (e.g., YC) b C r data).

[0043] In step 306, for each pixel, the pixel amplitude is calculated. As mentioned above, the amplitude M is based on the formula... Perform the calculation.

[0044] In step 308, the luminance gain and chrominance gain are calculated for each pixel using the corresponding amplitude value M calculated in step 308. The storage device of the controller 50 includes multiple LUTs. The first LUT includes multiple luminance gain values ​​associated with multiple amplitude values ​​M. The second LUT includes multiple chrominance gain values ​​associated with multiple amplitude values ​​M. In this way, the luminance gain and chrominance gain of each pixel are based on the same amplitude value M.

[0045] In step 310, a luminance gain value is applied to the original Y value of the pixel, and a chrominance gain value is applied to each original C value of the pixel. b and C r Values. That is, the original Y value and the original C value. b and C r Each value is multiplied (e.g., scaled) by the corresponding gain value. As previously mentioned, applying the gain value involves multiplying the original Y value by the gain value associated with the amplitude value M in the luminance LUT. Applying the gain value also includes multiplying the original C value by the gain value associated with the amplitude value M in the luminance LUT. b and the original C r The value is multiplied by the gain value associated with the amplitude value M in the chromaticity LUT. This step provides the modified or adjusted Y, C... b and C r value.

[0046] After applying the gain value, the modified or adjusted YC b C r Pixel values ​​are obtained from YC in step 312 b C rThe format is converted to RGB format. Subsequently, controller 50 controls the image to be output and displayed on display 60 based on the modified pixel values. It will be understood that steps 306, 308, 310, and 312 can be repeated for each pixel in the image before the image is displayed on display 60. Alternatively, less than all image data received from imaging device 40 can be modified by method 300. Although the conversion of RGB format pixels to YC format is described... b C r While the format is not explicitly stated, it will be understood that other multichannel color formats can be used. For example, RGB format can be converted to Hue, Saturation, and Lightness (HSV) or Hue, Saturation, and Lightness (HSL) format, and the gain can be calculated based on each value of the pixel in the converted format. In other words, each channel in a multichannel color format can be adjusted using the same amplitude based on the values ​​of all channels. In this way, the lightness or brightness gain and chroma or color gain of each pixel can be determined based on each channel of the pixel.

[0047] Figure 4A and Figure 4B This is an example image generated by imaging device 40. Figure 4A An image 400 showing the target area is provided, including a target 420, a "hot zone" 430, and a darker background object 410. When image 400 is generated, the hot zone 430 and the background object 410 may be due to differences in illumination at the target area. For example, the light-emitting device 44 ( Figure 4A The position of the object (not shown) relative to the target 420 may cause a greater light intensity on the target 420 to create a hot zone 430 and prevent sufficient light from reaching the background object 410, resulting in the background object 410 being darker. Figure 4B An image 400' modified in the manner described herein is shown. As described herein, pixels in region 430' on target 420' and background object 410' have been subjected to a non-linear gain. The non-linear gain has reduced the brightness of hot area 430, making hot area 430 more visible as region 430'. Similarly, pixels in background object 410 have been subjected to a non-linear gain to increase brightness and provide a modified background object 410'. As described herein, image 400' can be displayed on display 60.

[0048] Figure 5 It is shown that various aspects according to this disclosure can be used as follows: Figure 1 The exemplary system used in the technology discussed in section -4. Figure 5This is a simplified functional block diagram of a computer that can be configured as a medical system 10, an imaging device 40, and / or a user input device 70 according to exemplary embodiments of the present disclosure. Specifically, in one embodiment, any of the user devices discussed herein may be components of hardware 100, including, for example, a data communication interface for packet data communication. The platform may also include a controller 50 in the form of one or more processors for executing program instructions 110. The platform may include storage units 90 (such as ROM, HDD, SDD, etc.) that may store data on computer-readable medium 130, although the medical system 10 may receive programming and data via network communication. The medical system 10 may also have memory 80 (such as RAM) storing instructions 110 for performing the techniques proposed herein, although the instructions 110 may be temporarily or permanently stored in other modules of system 10 (e.g., controller 50 and / or computer-readable medium 130). System 10 may also include input and output ports (e.g., user input device 70, imaging device 42, and / or light-emitting device 44) and / or display 60 for connection to input and output devices such as keyboards, mice, touchscreens, monitors, displays, etc. These systems can be implemented through appropriate programming of a computer hardware platform. As discussed herein, it will be understood that processing of modified pixels must be performed locally, for example, on a processor associated with medical system 10, so that image data does not have to be sent to a remote processing unit (e.g., a server). In this way, the latency of pixel modification can be improved for the system.

[0049] The methods described herein and the associated imaging systems can improve image generation in medical systems. For example, the amplitude or gain value depends on the overall intensity (both luminance and chrominance values) of each pixel to determine the appropriate gain value to be applied, thus providing non-linear gain. By relying on non-linear gain, the method can increase the visibility of darker and hotter areas (e.g., stones in the body) based on the overall pixel intensity. This preserves the sharpness of darker colors and prevents a “washed-out” appearance in the image. Furthermore, since this method is a post-processing technique and operates automatically within the medical system, it can be implemented with any user-selected luminance without any automatic light control feedback loop. Moreover, the method provides modified images without illumination or exposure control and does not require changes to the amount of light at the target site. Furthermore, the low complexity of the mathematical equations (e.g., square root functions) allows the method to be executed efficiently using FPGA logic, which can provide a piecewise design without requiring a GPU. Operation using FPGA logic (e.g., a controller associated with the medical system) allows the image to be modified within the time delay parameters of the endoscopic imaging device. Operating within system latency requirements reduces noticeable jitter in the output images and allows the method to be implemented with most (if not all) imaging devices associated with endoscopes (e.g., Boston Scientific's SpyGlass). TM DM system).

[0050] For those skilled in the art, various modifications and variations can be made to the disclosed apparatus without departing from the scope of this disclosure. For example, methods for enhancing images obtained using a camera are not limited to a single camera or medical system. Various imaging and / or luminescent devices, or different medical systems, can be used in conjunction with a controller to modify the image. It will be understood that the controllers and methods described for enhancing images can be applied to any range. Other embodiments of this disclosure will be apparent to those skilled in the art in light of the specification and practice of the invention disclosed herein. The specification and examples are intended to be illustrative only, and the true scope and spirit of the invention are indicated by the appended claims.

Claims

1. A medical system comprising: A shaft having a proximal end and a distal end; An imaging device located at the distal end of the axis; as well as Controller, wherein the controller is configured to: Receive image data including pixel data from the imaging device, the pixel data including a plurality of individual pixel values; Convert the pixel data from RGB format to YC format. b C r Format; and Adjusted pixel data is formed by applying a gain to the pixel data, wherein the gain is based on the Y value, C value, and C value of the individual pixel values ​​of the transformed pixel data. b Value and C r value, The controller is further configured to calculate the magnitude of each pixel value, wherein the magnitude of each pixel value is equal to... And wherein the gain is a function of the amplitude.

2. The medical system according to claim 1, wherein, The gain includes a first gain and a second gain, and the first gain is applied to the Y value, while the second gain is applied to the C value corresponding to the pixel value. b Value and C r value.

3. The medical system according to any one of the preceding claims further includes a memory, wherein, The memory includes lookup tables (LUTs) for each of the luminance and chrominance values.

4. The medical system according to claim 1 or 2, wherein, The controller is also configured to transfer adjusted pixel data from YC b C r Convert the format to RGB format.

5. The medical system of claim 4, further comprising a display, and wherein, The controller is also configured to generate images based on adjusted and transformed pixel data.

6. The medical system according to claim 1 or 2 further includes a light-emitting device located at the distal end of the axis.

7. The medical system of claim 1 or 2 further includes a user input device configured to receive at least one user input for controlling the medical system.

8. The medical system according to claim 1 or 2, wherein, The controller includes a field-programmable gate array (FPGA).

9. The medical system according to claim 1 or 2, wherein, The medical system also includes a handle, wherein the controller is disposed on or in the handle.

10. A method for controlling an imaging device for a medical system, the method comprising: Receive an image comprising pixel data from the imaging device of the medical system, the pixel data including a plurality of individual pixel values; The pixel data is converted from RGB format to a second multi-channel color format, which has lightness and color values. Calculate the magnitude of each pixel value, where the magnitude of each pixel value is equal to ; Gain is applied to adjust the corresponding brightness or corresponding color of each pixel, wherein the gain is based on the Y value, C value, and C value of the individual pixel values ​​of the converted pixel data. b Value and C r Value; and Each pixel in the multiple pixels is converted from the second multi-channel color format to RGB format.

11. The method of claim 10, further comprising applying a gain to the lightness value that is different from the color value of the corresponding pixel value.

12. The method of claim 10 or 11, further comprising generating an image on a display of the medical system based on adjusted and transformed pixel data.

13. The method according to claim 10 or 11, wherein, Applying the gain to each pixel value includes: matching the magnitude of the gain with the corresponding value in a luminance lookup table (LUT) for the corresponding luminance value, and matching the magnitude of the gain with the corresponding value in a color lookup table (LUT) for the corresponding color value.

14. The method according to claim 10 or 11, wherein, The magnitude of the gain is applied to each of the brightness and color values ​​of the corresponding pixel.

Images