System and method for color calibration and correction of medical images

Abstract

A system and method for color-correcting images are described. An exemplary method includes the steps of receiving, at first and second time points, first and second image data captured by an imaging device of a medical imaging system during a medical procedure, and first and second values ​​of operating parameters of a light source of the medical imaging system, wherein the color of the light emitted from the light source shifts based on the values ​​of the operating parameters. A first color shift is determined based on a first value, and a color correction function is applied to the first image data to compensate for the first color shift to produce a first color-corrected image data. A second color shift is determined based on a second value, and a color correction function is applied to the second image data to compensate for the second color shift to produce a second color-corrected image data.

Description

Technical Field

[0001] The present disclosure generally relates to medical imaging systems and related methods of use. More specifically, aspects of the present disclosure relate to color calibration and correction processes for endoscopic imaging systems.

Background Art

[0002] An endoscopic imaging system configured to capture images of a target site during a medical procedure includes at least one light source and at least one imaging device, among other components. Some endoscopic imaging systems may drive the light source at a constant current throughout the duration of a medical procedure, and the gain and exposure time of the imaging device may be adjusted to achieve acceptable pixel saturation. To improve illumination control, other endoscopic imaging systems may first adjust the light intensity of the light source and only adjust the gain and exposure time of the imaging device when the dynamic range of the light source reaches its limit. However, when the light intensity is adjusted, the amount of current flowing through the light source changes, and when the current changes, the spectrum of the light emitted from the light source changes. Variations in other operating parameters of the light source, such as temperature, may also cause variations in the spectrum of the light emitted from the light source. This spectral variation causes a color shift that affects the way the endoscopic imaging system captures and reproduces the features of the target site.

Summary of the Invention

[0003] An exemplary method for color-correcting an image performed by a computing system may include, at a first time, receiving first image data captured by an imaging device of a medical imaging system during a medical procedure, and a first value of an operating parameter of a light source in the medical imaging system, wherein the color of the light emitted from the light source may shift based on the value of the operating parameter. The exemplary method may also include, determining a first color shift based on the first value; applying a color correction function to the first image data to compensate for the first color shift to generate first color-corrected image data; and providing the first color-corrected image data to a display associated with the computing system for display. The exemplary method may also include, at a second time different from the first time, receiving second image data captured by an imaging device during a medical procedure, and a second value of an operating parameter of a light source, wherein the second value may differ from the first value. The method may further include, determining a second color shift based on the second value; applying a color correction function to the second image data to compensate for the second color shift to generate second color-corrected image data; and providing the second color-corrected image data to a display for display.

[0004] In any of the exemplary methods disclosed herein, multiple color shifts of light emitted from one or more light sources of one or more medical imaging systems, including at least the light sources of the medical imaging system, are characterized by multiple values ​​of an operating parameter, and color shift data can be acquired and stored. To determine a first color shift, a first color shift data corresponding to a first value in the stored color shift data can be identified. The first color shift data may indicate a measured change in the pixel intensity value of one or more image data components based on a first color shift of light emitted by the light sources when the operating parameter has a first value. To apply a color correction function to the first image data, one or more first compensation coefficients of a color correction function can be determined based on the first color shift data to compensate for the measured change in the pixel intensity value of one or more image data components. The first color shift data may further indicate timing associated with the measured change in the pixel intensity value of one or more image data components when the operating parameter has a first value, and the color correction function can be applied to the first image data based on timing.

[0005] In an additional embodiment, to determine a second color shift, a second color shift data corresponding to a second value in the stored color shift data may be identified. The second color shift data may indicate a measured change in the pixel intensity value of one or more image data components based on a second color shift of the light emitted from the light source when the operating parameter has a second value. To apply a color correction function to the second image data, one or more second compensation coefficients of the color correction function may be determined based on the second color shift data to compensate for the measured change in the pixel intensity value of one or more image data components. The second color shift data may further indicate the timing associated with the measured change in the pixel intensity value of one or more image data components when the operating parameter has a second value, and the color correction function may be applied to the second image data based on the timing.

[0006] In some embodiments, it may be determined that the difference between a second value and a first value exceeds a predetermined threshold. The operating parameter may be the current of the light source, and the first and second values ​​of the current may be received from the light source at a first time and a second time, respectively. The operating parameter may be the temperature of the light source, and the first and second values ​​of the temperature may be received from a temperature sensor placed adjacent to the light source at a first time and a second time, respectively. The operating parameter may be the temperature of the light source, and the first and second values ​​of the temperature may be estimated based on a known ambient temperature, a known junction thermal resistance of the light source, and the current values ​​of the light source received at a first time and a second time, respectively.

[0007] In other embodiments, a first timing associated with a first color shift may be determined. The first timing may include a measured period from the detection of the operating parameter of a light source operating at a first value to the observation of the first color shift. A color correction function may be applied to the first image data based on the first timing. A second timing associated with a second color shift may be determined. The second timing may include a measured period from the detection of the operating parameter of a light source operating at a second value to the observation of the second color shift. A color correction function may be applied to the second image data based on the second timing. The operating parameter may be a first operating parameter, and at a first time, a first value of the first operating parameter, in addition to a first value of the second operating parameter of the light source, may be received. It may be determined that the first operating parameter being at a first value has a more dominant effect on the color of the light emitted from the light source than the second operating parameter being at a first value, and this determination may allow the first color shift and the timing associated with the first color shift to be determined based on the first value of the first operating parameter.

[0008] A computing system for color-correcting images, which is communicably connectable to a medical imaging system, may include a data store, at least one memory, and one or more processors, including an image processor. The data store may store color shift data obtained by characterizing multiple color shifts of light emitted by one or more light sources of one or more medical imaging systems, including the light sources of the medical imaging system, with multiple values ​​of the operating parameters of one or more light sources. At least one memory may store instruction sets, which may cause the computing system to perform operations when the instruction sets are executed by one or more processors. An operation may include, at a first time, receiving first image data captured by the imaging device of the medical imaging system during a medical procedure, and first values ​​of the operating parameters of the light sources of the medical imaging system. The operation may also include identifying first color shift data from color shift data stored in a data store that corresponds to a first value, the first color shift data which may include measured changes in the pixel intensity values ​​of one or more image components based on a first color shift of light emitted by a light source when the operating parameter is a first value; determining one or more first compensation coefficients of a color correction function based on the first color shift data to compensate for the measured changes in the pixel intensity values ​​of one or more image components; applying the color correction function to first image data based on one or more first compensation coefficients to generate first color-corrected image data; and providing the first color-corrected image data for display to a display associated with a computing system. The operation may further include receiving second image data captured by an imaging device during a medical procedure and a second value of the operating parameter of a light source at a second time different from the first time, the second value being different from the first value.The operation may further include identifying a second color shift data from color shift data stored in a data store that corresponds to a second value, the second color shift data which may indicate a measured change in the pixel intensity value of one or more image components based on a second color shift of light emitted by a light source when the operation parameter is a second value; determining one or more second compensation coefficients of a color correction function based on the second color shift data to compensate for the measured change in the pixel intensity value of one or more image components; applying the color correction function to the second image data based on one or more second compensation coefficients to compensate for the second color shift to generate a second color-corrected image data; and providing the second color-corrected image data to a display for display.

[0009] In any of the exemplary computing systems disclosed herein, the operation may also include determining whether the difference between a second value and a first value exceeds a predetermined threshold. The operating parameter may be the current of a light source or the temperature of a light source.

[0010] A method for color-correcting an image performed by a computing system may include, at a first time, receiving first image data captured by an imaging device of a medical imaging system during a medical procedure, and a first value of an operating parameter of a light source of the medical imaging system, wherein the color of the light emitted from the light source may shift based on the value of the operating parameter, and the operating parameter may be the current of the light source or the temperature of the light source. The method may also include, determining a first color shift based on the first value, applying a color correction function to the first image data to compensate for the first color shift to generate first color-corrected image data, and providing the first color-corrected image data for display to a display associated with the computing system. The method may further include, at a second time different from the first time, receiving second image data captured by an imaging device during a medical procedure, and a second value of an operating parameter of a light source, wherein the second value may be different from the first value. The method may further include the steps of determining whether the difference between a second value and a first value exceeds a predetermined threshold; determining a second color shift based on the first value based on the difference exceeding the predetermined threshold; applying a color correction function to the second image data to compensate for the second color shift to generate a second color-corrected image data; and providing the second color-corrected image data to a display for display.

[0011] In any of the exemplary methods disclosed herein, a predetermined threshold may be based on the type of operating parameter. A first predetermined threshold associated with the current of the light source may be smaller than a second predetermined threshold associated with the temperature of the light source.

[0012] It should be understood that both the general description above and the detailed description below are illustrative and descriptive only and do not limit the claimed invention. Where used herein, the terms “comprises,” “comprising,” “including,” or other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a list of elements does not necessarily consist only of those elements, but may include other elements not expressly enumerated or inherently present in such a process, method, article, or apparatus. The term “exemplary” is used to mean “example” rather than “ideal.” The term “distal” refers to a direction away from the operator / towards the treatment site, and the term “proximal” refers to a direction towards the operator. The term “about” or similar terms (e.g., “substantially”) include values ​​within + / - 10% of the stated values. [Brief explanation of the drawing]

[0013] The accompanying drawings incorporated herein and constituting part of herein illustrate examples of the present disclosure and, together with the description, help to illustrate the principles of the present disclosure. [Figure 1] This figure shows an exemplary environment in which an image color calibration and / or correction process may be implemented. [Figure 2] This figure shows an example of an image color calibration method. [Figure 3] This diagram shows an example of an image color correction method. [Figure 4] This is an illustrative system flow chart for color correction of images. [Figure 5] This figure shows an example of a computing device. [Modes for carrying out the invention]

[0014] As briefly described in the background technology, an endoscopic imaging system configured to capture images of a target site during a medical procedure includes, along with other devices, at least one light source and at least one imaging device. Some endoscopic imaging systems can drive the light source with a constant current throughout the entire duration of the medical procedure, and the gain and exposure time of the imaging device can be adjusted to achieve acceptable pixel saturation. To improve illumination control, other endoscopic imaging systems can first adjust the light intensity of the light source and only adjust the gain and exposure time of the imaging device when the dynamic range of the light source reaches its limit. However, adjusting the light intensity changes the amount of current flowing through the light source, and a change in current changes the spectrum of the light source. This spectral variation results in a color shift that affects how the endoscopic imaging system captures and reproduces image data of the target site.

[0015] For example, the light source could be a light-emitting diode (LED). An LED is a semiconductor device that emits light when an electric current flows through it. Electrons in a semiconductor can recombine with holes and emit energy in the form of photons, and the color of the light (corresponding to the energy of the photons) is determined by the energy required for the electrons to cross the band gap of the semiconductor. White light can be obtained by using multiple semiconductors or layers of light-emitting phosphors on a semiconductor. For example, typically a white LED uses a blue LED photon pump coupled to a yttrium aluminum garnet (YAG) phosphor. The first portion of the photon from the blue LED passes through the phosphor unaffected, while the second portion of the photon is absorbed by the phosphor material and can be emitted from the green portion of the spectrum to the yellow, orange, and red portions of the spectrum. The human eye can perceive this combination as white light.

[0016] When the LED current is low, the light emitted by the LED may have less blue component (e.g., it may be warmer in color). In contrast, when the LED current is high, the phosphor becomes less efficient, blue emission becomes more dominant, and the light emitted by the LED has more blue component (e.g., it becomes cooler in color). In other words, a high LED current increases the proportion of high-energy photons (e.g., high-energy photons with short wavelengths), which shifts the spectrum of the emitted light toward the blue region. Therefore, the pixel intensity of the blue component in the image data of the target area may increase as the LED current increases.

[0017] Fluctuations in other operating parameters of the light source, such as temperature, can also cause variations in the spectrum of light emitted from the light source. For example, when the temperature of the LED substrate increases, the semiconductor's band gap changes (e.g., decreases), which alone causes the emitted photons to shift towards the red region of the spectrum. However, the rise in temperature can also be due to an increase in the LED current, and since current has a more dominant effect on color than temperature, the spectrum of the emitted light may still shift overall towards the blue region of the spectrum.

[0018] Aspects of this disclosure relate to image color calibration and correction techniques that enable real-time image color correction during medical procedures and compensate for light source spectral variations and resulting color shifts in captured image data based on one or more light source operating parameters. For example, in a final inspection calibration process for a medical imaging system, multiple color shifts of light emitted from the light sources of the medical imaging system may be characterized by multiple values ​​of one or more light source operating parameters (e.g., characterization is performed for each light source) to obtain color shift data specific to the light source. As another example, characterization may be performed across light sources of multiple medical imaging systems throughout the product development lifecycle to obtain, for example, more generalized color shift data (e.g., averaged color shift data across multiple light sources).

[0019] Color shift data may indicate a measured change (e.g., an increase or decrease in quantity or percentage) in the pixel intensity value of one or more components of image data (e.g., red, green, and / or blue components) when given operating parameters of the light source are at a given value. Operating parameters may include the current of the light source, the temperature of the light source, and / or any other operating parameters of the light source that cause a color shift in the light emitted from the light source. Color shift data may be stored by a computing system that is communicably connected to a medical imaging system and / or otherwise accessible from the computing system. Color shift data may be used in an image color correction process performed by the computing system during a medical procedure.

[0020] To provide an exemplary image color correction process, image data from an imaging device and a first value of the operating parameter of a light source may be received. The color shift of the light emitted from the light source may be determined based on the first value of the operating parameter, and a color correction function may be applied to the image data to compensate for the color shift to produce color-corrected image data. For example, first color shift data corresponding to the first value may be identified using stored color shift data. The first color shift data may indicate a measured change in the pixel intensity value of one or more components of the image data based on the color shift of the light emitted from the light source when the operating parameter has the first value. Subsequently, one or more compensation coefficients of the color correction function may be determined based on the first color shift data to compensate for the measured change in the pixel intensity value of one or more components of the image data. By applying the color correction function based on the compensation coefficients, color differences in the image data caused by the color shift can be eliminated or at least minimized. The color-corrected image data may then be provided for display on one or more user interfaces, displays, and / or other external devices.

[0021] In some examples, a first value of a plurality of operating parameters of a light source may be received (e.g., both a first current value and a first temperature value may be received). In such examples, a color shift is determined using an operating parameter that has a dominant influence on the color shift (e.g., causes a greater measured change in pixel intensity values of one or more image components), and for example, a correction function may be applied accordingly.

[0022] The above-described image color correction process may be repeated one or more times over the course of a medical procedure, and for example, as the value of one or more operating parameters changes, real-time image color correction may be enabled. For example, as described above, the intensity of light emitted from a light source may be adjusted during a medical procedure to improve illumination control, which in turn changes the current and / or temperature, which further affects the color of the light emitted from the light source. Thus, based on the receipt of new values (e.g., a second value, a third value, etc., different from the first value) for one or more operating parameters, the above process is repeated, and a subsequent color shift and a corresponding compensation factor for the color correction function to be applied to the image data are determined, and color-corrected image data may be generated.

[0023] In some examples, a lag or delay in the timing of a color shift due to a light source in response to a change in an operating parameter value may be taken into account when applying a color correction function to image data. In further examples, to prevent the compensation factor of the color correction function from always changing for small changes in the operating parameters, the process may be repeated when the difference between the new value and the previously received value exceeds a predetermined threshold. Such threshold processing may help reduce and / or eliminate flicker or other variations in the color-corrected image data provided for display. Additionally, such threshold processing may help conserve the processing resources of a computing system. In further examples, when the value received for one or more operating parameters is not included in the stored color shift data, interpolation techniques using neighboring values may be applied.

[0024] FIG. 1 shows an exemplary environment 100 in which an image color calibration and / or correction process can be implemented. The environment 100 can include one or more of a medical device 102, a computing system 104, one or more external devices 106, one or more optional server-side systems 130, and / or a network 140.

[0025] The medical device 102 can be used to perform diagnostic and / or interventional medical procedures (hereinafter referred to as medical procedures for simplicity) on a patient. The medical device 102 can be an endoscope or other type of scope such as, among other examples, a bronchoscope, a ureteroscope, a duodenoscope, a gastroscope, an endoscopic ultrasound (「EUS」) scope, a colonoscope, a laparoscope, an arthroscope, a cystoscope, a suction scope, a sheath, or a catheter.

[0026] The medical device 102 can include an imaging system 108. The imaging system 108 can include at least one imaging device 110 and at least one light source 112. The imaging device 110 and / or the light source 112 can be disposed at the distal end of the medical device 102 (e.g., the distal tip of the medical device 102). The imaging device 110 can be configured to capture an image signal during a medical procedure when the distal end of the medical device 102 is inserted into a body lumen of a patient and guided through the body lumen of the patient to a target site. The imaging device 110 can include, among other similar devices, one or more cameras, one or more image sensors, an endoscope viewing element, or an optical assembly including one or more image sensors and one or more lenses. The light source 112 can be configured to illuminate a region of the patient's body (e.g., the target site) during a medical procedure to assist in imaging the target site by the imaging device 110. The light source 112 can include one or more LEDs, an incandescent light source, an optical fiber, and / or other illuminators.

[0027] In some cases, to improve lighting control, the intensity of light emitted from the light source 112 may be adjusted throughout the medical procedure. Adjusting the light intensity causes a change in the amount of current flowing through the light source 112, and as the current changes, the spectrum of the light emitted from the light source 112 may change. Spectral variations can result in a color shift that affects how the imaging device 110 captures features of the target site. Throughout the medical procedure, the light source 112 may be configured to detect and / or measure the amount of current flowing through it and convert it into an electrical signal (e.g., a current signal) indicating the current value. The current value may be provided to a computing system 104 for use in one or more image color correction processes to compensate for the color shift.

[0028] Fluctuations in other operating parameters of the light source 112, such as temperature, can also cause fluctuations in the spectrum of light emitted from the light source 112, either alone or in combination with current, resulting in a color shift. Therefore, the medical device 102 may also include an optional temperature sensor 114. The optional temperature sensor 114 may be positioned adjacent to the light source 112. The optional temperature sensor 114 may be configured to measure the temperature of the light source 112. For example, the optional temperature sensor 114 may detect and / or measure coldness and / or heat and convert it into an electrical signal (e.g., a temperature signal) indicating a temperature value. In an example where the light source 112 includes one or more LEDs, the optional temperature sensor 114 may be configured to measure the temperature of the substrate of one or more LEDs. The temperature values ​​may be provided to a computing system 104 for use in one or more image color correction processes to compensate for the color shift.

[0029] One or more components of the medical device 102, including the imaging system 108 and its components, may be communicably connected to the computing system 104 via wired and / or wireless connections (e.g., via network 140) to enable communication of various signals between the medical device 102 and the computing system 104. For example, image signals captured by the imaging device 110 may be received by the computing system 104. Furthermore, current signals from the light source 112 and / or temperature signals from an optional temperature sensor 114 may be received by the computing system 104. In some examples, the computing system 104 may provide the medical device 102 with illumination signals to cause the light source 112 to emit light and / or emit light at a given intensity.

[0030] In some examples, the computing system 104 is a separate computing device, controller, or other similar standalone processing unit from the medical device 102. In other examples, the computing system 104 may be integrated with the medical device 102. For example, the computing system 104 may be located within the handle of the medical device 102. In other examples, the computing system 104 may be located at the distal end of the medical device 102.

[0031] The computing system 104 may include a memory 116 and one or more processors 118. The memory 116 may store a set of instructions to be executed by one or more processors 118 to cause the computing system 104 to perform the corresponding operations. At least a portion of the instructions stored in the memory 116 may include an image color correction process. The memory 116 may also include one or more data stores. Additionally or alternatively, the computing system 104 may also include one or more data stores separate from the memory 116. The processors 118 may include at least one image processor 120. The image processor 120 may be configured to process image signals captured by the imaging device 110 and received by the computing system 104 to generate image data. Furthermore, the image processor 120 may be configured to perform an image color correction process to generate color-corrected image data. As will be described in more detail below, color-corrected image data may be generated based on the image signal and one or more other signals (e.g., current signals and / or temperature signals) related to one or more operating parameters of the light source 112, to compensate for color shifts resulting from the corresponding states of one or more operating parameters, as will be described later. In some examples, the image processor 120 may be a field-programmable gate array (FPGA), a digital signal processing (DSP) processor, a graphics processing unit (GPU), or other similar. Depending on the type of image processor (e.g., basic or more advanced), the image processor 120 may be capable of performing a variety of additional image processing operations, including more complex artificial intelligence (AI) or machine learning-based techniques.

[0032] The computing system 104 may further include an optional communication interface 122 for providing connectivity to the network 140. The optional communication interface 122 may also provide connectivity to the medical device 102 and / or one or more external devices 106. In some examples, communication connections between the computing system 104 and the medical device 102 (or its components), and / or between the computing system 104 and one or more external devices 106 may be supported at least partially via the network 140.

[0033] At least one of the external devices 106 may be a display device configured to display image data, such as color-corrected image data generated by the computing system 104. The display device may be a monitor, a screen of the computing device, a touchscreen display device, and the like. In some examples, the display device may be a separate device from the computing system 104, and may be connected to the computing system 104 via wired and / or wireless connections. In other examples, the display device may be the display of the computing system 104 itself.

[0034] In some examples, the computing system 104 can generate or cause to generate one or more graphical user interfaces based on a set of instructions or information stored in memory 116, a set of instructions or information received from one or more of an optional server-side systems 130, and / or similar, and can display the graphical user interfaces via a display device. The graphical user interfaces may be, for example, application interfaces or browser user interfaces and may include text, selection controls, etc., in addition to the displayed color-corrected image data. The display device may include a touchscreen or a display with other input systems (e.g., mouse, keyboard, voice, etc.) for an operator of the computing system 104 to control one or more functions of the computing system 104, the medical device 102 (or its components) via the computing system 104 and / or the display device. As an example, the operator may select one or more control elements displayed on the graphical user interface of the display device to adjust the intensity of light emitted from the light source 112. The selection is received by the computing system 104, and a corresponding illumination signal is sent from the computing system 104 to the light source 112.

[0035] In some examples, the external device(s) 106 may further include one or more third-party processing systems (e.g., AI processing systems) that can be connected to the computing system 104. The exemplary AI processing system may be configured to receive image data and / or color-corrected image data generated by the computing system 104 as input, and may be configured to process the image data (and optionally other input data) to generate augmented image data.

[0036] One or more components of environment 100, such as the medical device 102, the computing system 104, and / or one or more external devices 106, are network-connectable and can communicate with each other via a wired or wireless network such as network 140. Network 140 may be an electronic network. Network 140 may include a wide area network ("WAN"), a local area network ("LAN"), a personal area network ("PAN"), a cellular network (e.g., a 3G network, a 4G network, a 5G network, etc.), and one or more other wired and / or wireless networks. In other examples, components of environment 100 may communicate with and / or connect to network 140 via a universal serial bus (USB) or other similar local low-latency connection or direct wireless protocol.

[0037] In some embodiments, network 140 includes the Internet, and information and data provided between various systems occur online. "Online" may mean connecting to or accessing source data or information located remotely from other devices or networks connected to the Internet. Alternatively, "online" may mean connecting to or accessing an electronic network (wired or wireless) via a mobile communication network or device. The Internet is a global system of computer networks, a network of networks where users of one computer or other device connected to the network can obtain information from any other computer and communicate with users of other computers or devices. Components of environment 100 may be connected via network 140 using one or more standard communication protocols, and components may send and receive communications with each other via network 140.

[0038] In some examples, if one or more components of environment 100 are connectable to network 140, environment 100 may also include one or more optional server-side systems 130. The optional server-side systems 130 may include one or more remote image processing systems configured to perform at least some image processing (for example, to conserve local resources of computing system 104 when network connectivity is available). Additionally, or alternatively, the server-side systems 130 may include a data storage system for storing image data and / or color-corrected image data generated by computing system 104, and / or enhanced image data generated by third-party processing systems (e.g., one or more of external devices 106). In some examples, at least one data storage system may include a picture archiving and communication system (PACS) that stores image data, color-corrected image data, and / or augmented image data together with other types of imaging data from various imaging modalities (e.g., ultrasound, magnetic resonance imaging, nuclear medicine imaging, positron emission tomography, computed tomography, mammography, digital radiography, histopathology, etc.). Furthermore, an optional server-side system 130 may include an endoscopic reporting system configured to enable the generation of reports based on the image data, color-corrected image data, and / or augmented data.

[0039] In Figure 1, various components within Environment 100 are shown as separate components; however, it should be understood that in some embodiments, components or parts of components within Environment 100 may be integrated with or incorporated into one or more other components. In some embodiments, the operation or manner of one or more of the above-described components may be distributed among one or more other components. Any suitable arrangement and / or integration of the various systems and devices of Environment 100 may be used.

[0040] Certain examples included throughout this disclosure implement an endoscopic imaging system configured to perform a real-time image color correction process during a medical procedure based on LED operating parameters that affect color shift, such as current and / or temperature. However, it should be understood that the technology described herein can be adapted to other medical imaging systems with various types of light sources, and therefore the image color correction process may be further based on different types and / or additional types of light source operating parameters that affect color shift (e.g., depending on the type of light source), such as the lifespan of an incandescent light source. It should also be understood that the above examples are merely illustrative. The technology described herein can be adapted to any appropriate activity.

[0041] Figure 2 shows an exemplary image color calibration method 200 (hereinafter referred to as Method 200) for determining color shift data. In some examples, one or more steps of Method 200 may be performed by the computing system 104. In other examples, one or more steps of Method 200 may be performed by another computing device or system, and the determined color shift data may be provided to the computing system 104 for storage.

[0042] In step 202, multiple color shifts in the light emitted from a light source of a medical imaging system may be characterized by multiple values ​​of one or more operating parameters of the light source. In one example, the light source to be characterized may be the light source 112 of a medical device 102 during the final line calibration process, in which case the multiple color shifts are specific to the light source 112. That is, the characterization is performed for each medical device 102. In another example, the color shifts may be characterized more generally across the product development lifecycle of multiple medical devices 102. That is, the characterization may be performed across multiple light sources 112 of multiple medical devices 102.

[0043] Exemplary operating parameters may include current, temperature, and / or any other operating parameters that affect the color shift based on the type of light source 112 (e.g., lifetime if light source 112 is an incandescent light source). For each operating parameter, parameter-specific color shift data may be determined, including a range of operating parameter values ​​and the corresponding color shift measured at each value within that range. The color shift measured at a given value may indicate a measured change (e.g., an increase or decrease in quantity or percentage) in the pixel intensity value of one or more components (e.g., red, green, and / or blue components) of the image data captured by the imaging device 110 when the operating parameter of light source 112 is at a given value. Timing associated with the color shift may also be characterized and stored in relation to the measured change at a given value. Timing may include a measured period from the detection of the operating parameter of light source 112 being at a first value to the observation of the color shift. As an example, timing may include a specific period (e.g., a specific number of microseconds) from the detection of the current at a given value until the measurement change is observed. As another example, timing may include a specific period (e.g., a certain number of seconds) from when the temperature is detected at a given value until a change in the measurement is observed. In some examples, a calibration mapping may be generated for each operating parameter from parameter-specific color shift data.

[0044] For example, the current flowing through the light source 112 can be adjusted within a range of current values, for example, by adjusting the intensity of the light emitted from the light source 112 (for example, by sending one or more signals from the computing system 104 to the light source 112 to emit light at a given intensity). The color shift, including the associated timing of the color shift, can be determined by individual current values ​​within a range of operating parameter values, based on image data captured by the imaging device 110 when the light source 112 is operating at individual current values. The determined color shift can be stored as color shift data associated with the current (e.g., current-specific color shift data). For example, current-specific color shift data may indicate that, assuming a current of x milliamperes flows through the light source 112, the pixel intensity value of a first component of the image data captured by the imaging device 110 increases by y%. The first component may be the primary component affected (e.g., the spectrum eventually shifts towards the first component). However, when the light source 112 is operating at a current of x milliamperes, it may also affect, at least minimally, the remaining components of the image data. For example, if the first component is the blue component of the image data, the red and / or green components of the image data may also be affected. Therefore, current-specific color shift data may also show the effect on the pixel intensity values ​​of the remaining components. Current calibration mapping can be generated based on current-specific color shift data.

[0045] As another example, the temperature of the light source 112 can be adjusted within a range of temperature values. The color shift, including the associated timing of the color shift, can be determined at individual temperature values ​​within the range of temperature values, based on image data captured by the imaging device 110 at individual temperature values. The determined color shift can be stored as temperature-associated color shift data (e.g., temperature-specific color shift data). For example, the temperature-specific color shift data may indicate that, assuming the light source 112 has a temperature of α degrees, the pixel intensity value of a first component of the image data captured by the imaging device 110 increases by β%. The first component may be the primary component affected (e.g., the spectrum eventually shifts towards the first component). However, when the light source 112 is operating at a temperature of α degrees, it may also affect, at least minimally, the remaining components of the image data. Therefore, the temperature-specific color shift data may also indicate the effect on the pixel intensity values ​​of the remaining components. A temperature calibration mapping can be generated based on the temperature-specific color shift data. In some examples, the temperature value can be measured by a sensor, such as an optional temperature sensor 114 placed adjacent to the light source 112. In other examples, the temperature value may be estimated based on the current value of the light source 112 (e.g., over time) and one or more other known values ​​related to the light source (e.g., a known ambient temperature and a known junction thermal resistance of the light source 112).

[0046] In some examples, the color shift and the associated timing of the color shift may be determined for each potential or tolerance value of the operating parameter (e.g., each allowable current value and / or each allowable temperature value based on the specifications of the light source 112). In other examples, the color shift may be determined for a subset of the tolerance values ​​of the operating parameter, and interpolation techniques may be applied to obtain color shift data for values ​​between the subset of tolerance values.

[0047] In step 204, multiple characterized color shifts corresponding to multiple values ​​of one or more operating parameters may be stored as color shift data (e.g., current-specific color shift data, temperature-specific color shift data, etc.). If a mapping is generated from the color shift data, that mapping may be stored. In some examples, the color shift data may be stored in one or more data stores of memory 116 and / or in other local data storage of computing system 104 (e.g., in one or more data stores of computing system 104 separate from memory 116). Additionally or alternatively, the color shift data may be stored in a data storage system associated with an optional server-side system 130.

[0048] If the spectrum of light emitted from the light source 112 shifts due to a change in one or more operating parameters of the light source 112 during a medical procedure, different compensation coefficients of the color correction function may be appropriate for producing an image with minimized color difference (e.g., producing a color-corrected image that eliminates and / or at least mitigates color shifts caused by spectral shifts). Therefore, as will be described in detail below, the stored color shift data may then be retrieved to determine one or more compensation coefficients of the color correction function to be applied in real time to image data captured by the imaging device 110 during the medical procedure, and to compensate for color shifts based on one or more current operating parameters of the light source 112.

[0049] Therefore, certain embodiments may be performed to determine the color shift data. The method 200 described above is provided merely as an example and may include additional, fewer, different, or differently arranged steps compared to the one shown in Figure 2.

[0050] Figure 3 shows an exemplary image color correction method 300 (hereinafter referred to as Method 300). Figure 4 shows an exemplary system flow diagram 400 for color correcting an image by performing, for example, one or more steps of Method 300 in Figure 3. Referring together to Figures 3 and 4, in some examples, one or more steps of Method 300 may be performed by a computing system 104, more specifically by the image processor 120 of the computing system 104. Method 300 may be performed during a medical procedure utilizing a medical device 102. For example, when the medical device 102 is inserted into a patient's body lumen during a medical procedure and guided through the patient's body lumen to a target site, the imaging device 110 may capture an image of the area of ​​the patient's body while a light source 112 illuminates the area of ​​the patient's body (e.g., the target site). Method 300 described below is provided merely as an example and may include additional, fewer, different, or differently arranged steps compared to that shown in Figure 3.

[0051] In step 302, image data 412 may be received from the imaging device 110 of the medical imaging system (e.g., the imaging system 108 of the medical device 102). As described above, image data 412 may be received (e.g., from the imaging system 108) during the insertion and / or guidance of the medical device 102 into the target site.

[0052] In step 304, one or more values ​​of one or more operating parameters of the light source 112 of the medical imaging system may be received. For example, the current value 406 of the current of the light source 112 may be received from the light source 112 (for example, as a current signal from the light source 112). Additionally or alternatively, a temperature value 408 may be received from an optional temperature sensor 114 (for example, as a temperature signal from the light source 112). In other examples, such as when the medical device 102 does not include an optional temperature sensor 114, the temperature value 408 may be estimated based on a plurality of current values ​​over time, including the current value 406, and one or more other known values ​​associated with the light source 112. For example, the image processor 120 may determine the temperature value 408 based on a known ambient temperature, a known junction thermal resistance of the light source 112, and the current value 406 over time. Other exemplary operating parameters may include any other operating parameters of the light source 112 that affect color shift depending on the type of light source 112. For example, if the light source 112 is an incandescent light source, the lifetime of the light source 112 may be received and / or determined for use in step 306.

[0053] In step 306, the color shift of the light emitted from the light source 112 may be determined based on one or more values ​​received in step 304 (e.g., via a color shift compensation determination process 402). For example, the image processor 120 may receive color shift data that has been determined and stored locally in one or more data stores of the computing system 104 (e.g., the data store of memory 116 and / or a data store separate from memory 116) and / or remotely stored in the data storage system of an optional server-side system 130, as described with reference to method 200 in Figure 2. In some examples, the received color shift data may be a calibration mapping. The color shift data may indicate how the pixel intensity values ​​of each of the components of the image data captured by the imaging device 110 (e.g., each of the red, green, and / or blue components) are affected when one or more operating parameters are detected to be at one or more values ​​received in step 304. Color shift data may also indicate the associated timing of the effect of one or more components on each pixel intensity value after one or more operating parameters have been detected to be at one or more values.

[0054] The image processor 120 may use color shift data and / or calibration mapping for a given operating parameter to identify the color shift of light (e.g., a measured change in the pixel intensity value of one or more components of the image data) corresponding to the value of a given operating parameter received in step 304. For example, if a current value of x milliamperes 406 is received, the change in y1 in the red component of the image data 412, the change in y2 in the green component of the image data 412, and / or the change in y3 in the blue component of the image data 412 corresponding to x milliamperes may be identified within the current-specific color shift data and / or current calibration mapping. Furthermore, the associated timing from the detection that the light source 112 is operating at x milliamperes to the observation of the change may be identified. As another example, if a temperature value of α degrees 408 is received, the change in β1 in the red component of the image data 412, the change in β2 in the green component of the image data 412, and / or the change in β3 in the blue component of the image data 412 corresponding to α degrees may be identified within the temperature-specific color shift data and / or temperature calibration mapping. In addition, the associated timing from the detection that light source 112 is operating at alpha to the observation of the change can be identified. In some examples, if the value of a given parameter is not included in the color shift data and / or calibration mapping, interpolation techniques using neighboring values ​​can be applied to identify the corresponding change in one or more image components.

[0055] In addition, in further examples, if multiple operating parameters (e.g., both current and temperature) can affect the color shift by individual given values, the dominant operating parameter may be used to characterize the color shift. The dominant operating parameter may be the one that causes the largest measured change in the pixel value intensity of the image component (e.g., the largest increase or decrease in amount or percentage).

[0056] Furthermore, as part of the color shift compensation determination process 402, the determined color shift may then be used to determine one or more coefficients (e.g., compensation coefficients 410) of a color correction function 404 to compensate for the color shift. In some examples, the color correction function 404 may be a 3x3 color correction matrix applied to the image data 414 to obtain pixel-wise color-corrected image data 412 and to minimize the color difference resulting from the color shift (e.g., reduce or eliminate its effect). As previously mentioned with reference to Figure 2, one of the red, green, and / or blue components of the image data 412 may be the principal component affected by the color shift (e.g., the spectrum shifts towards the principal component). However, the remaining components may also be affected, at least minimally. Therefore, the compensation coefficients 410 may also be determined for the off-diagonal components of the color correction matrix (e.g., components corresponding to the remaining components). For example, a 3x3 color correction matrix may consist of three rows. Each row may include three determined compensation coefficients 410 to be multiplied by the red, green, and blue pixel value intensities detected by the imaging device 110 in one of the red, green, or blue channels.

[0057] The determined compensation coefficient(s) 410 may result in a reciprocal of the change identified in the pixel intensity value of one or more image components. For example, based on the current value 406, if the color shift is a change in y1 in the red component of the image data 412, a change in y2 in the green component of the image data 412, and / or a change in y3 in the blue component of the image data 412, then compensation coefficients 410 to cancel out the changes in y1, y2, and / or y3 in the red, green, and blue components, respectively, may be determined to minimize the color difference caused by the color shift. As another example, based on the temperature value 408, if the color shift is a change in β1 in the red component of the image data 412, a change in β2 in the green component of the image data 412, and / or a change in β3 in the blue component of the image data 412, then compensation coefficients 410 to cancel out the changes in β1, β2, and / or β3 in the red, green, and blue components, respectively, may be determined to minimize the color difference caused by the color shift. The determined compensation coefficient 410 may be provided for use in the color correction function 404.

[0058] In step 308, a color correction function 404 may be applied to the image data 412 to compensate for the color shift, thereby generating color-corrected image data 414. For example, the color correction function 404 may include one or more compensation coefficients 410 for compensating for the color shift. The timing of the application of the color correction function 404 may be based on the associated timing of the color shift indicated by the color shift data and / or calibration mapping.

[0059] In step 310, the color-corrected image data 414 may be provided for display on one or more of the external devices 106. For example, the external devices 106 may include a display device communicatively connected to the computing system 104, and / or a display device integrated into the computing system 104. Displaying the color-corrected image data 414 can help the operator of the medical device 102 to visually guide to features within the target site related to the medical procedure and to more clearly identify those features.

[0060] In other examples, the color-corrected image data 414 may be provided to one or more other external devices 106, such as a third-party AI processing system connected to the computing system 104. The exemplary AI processing system may be configured to receive the color-corrected image data 414 generated by the computing system 104 as input, and to process the color-corrected image data 414 to generate augmented image data. The augmented image data may then be provided for display via one or more of the display devices described above.

[0061] Steps 302–310 of Method 300 may be repeated periodically throughout a medical procedure and / or as changes in one or more operating parameters of the light source 112 occur in real time or near real time to account for changes, enabling “on-the-fly” color correction of image data received from the imaging device 110. For example, the image data 412 received in step 302 may be first image data received from the imaging device 110 at a first time, and one or more values ​​received in step 304 may be first values ​​of the operating parameters at the first time used to determine the first color shift. The color correction function 404 that compensates for the first color shift may continue to be applied to the first image data received from the imaging device 110 until a change in the operating parameter(s) is detected (e.g., at a second time different from the first time).

[0062] For example, at a second time during a medical procedure, a second image data may be received from the imaging device 110 in step 302, and one or more second values ​​different from a first value of one or more operating parameters of the light source 112 may be received in step 304. Accordingly, in step 306, a second color shift of the light emitted from the light source 112 may be determined based on one or more second values, and in step 308, a color correction function 404 may be applied to the second image data to compensate for the second color shift to generate a second color-corrected image data. For example, one or more different (e.g., a second) compensation coefficients 410 of the color correction function 404 may be determined via a color shift compensation determination process 402 to compensate for the second color shift. The second color-corrected image data may then be provided for display on one or more external devices in step 310. The color correction function 404 for compensating for the second color shift may continue to be applied to the second image data as it is received from the imaging device 110 until the next change in the operating parameter(s) is detected (for example, at a third time during the medical procedure).

[0063] When applying method 300 in real time during a medical procedure, the image processor 120 may apply hysteresis. For example, a timing lag or delay in color shifts due to changes in operating parameter values ​​may be considered. For instance, assuming the temperature rises to α degrees, a lag or delay of z milliseconds may be expected before a color shift occurs from the light source 112. Thus, a color correction function having a compensation coefficient determined to compensate for the color shift when the temperature is α degrees may be applied to image data received after z milliseconds.

[0064] Furthermore, as the values ​​of the operating parameters change, one or more predetermined thresholds may be applied before changing the color correction function 404 (for example, before changing the compensation coefficients 410 of the color correction function 404). For example, each predetermined threshold may represent the minimum change (in absolute value) with respect to a given operating parameter that should be exceeded before changing the compensation coefficients (one or more) 410 in the color correction function 404. In other words, before changing the color correction function 404, it may be determined that the absolute difference between the current value and the previous value (for example, between a first value and a second value) exceeds a predetermined threshold.

[0065] Each predetermined threshold may be based on the type of operating parameter and / or the sensitivity of the color shift to changes in the value of that type of operating parameter. For example, a first predetermined threshold for current may be lower than a second predetermined threshold for temperature (e.g., to allow for lower minimum changes). This thresholding may help prevent the color correction function 404 from constantly changing over small changes in the operating parameters (e.g., to prevent the image processor 120 from overreacting). Such thresholding may help reduce and / or eliminate flicker in the color-corrected image data 414 provided to one or more external devices 106 for display. In addition, such thresholding may help conserve the processing resources of the computing system 104, for example, by reducing the number of iterations or instances in which at least steps 306 and 308 are performed.

[0066] Figure 5 shows an example of computer 500. Figure 5 is a simplified functional block diagram of computer 500, which may be configured as a device for performing processes, steps, or operations shown in or described with respect to Figures 1 to 4, according to exemplary embodiments of the present disclosure. For example, computer 500 may be configured as one or more of the medical device 102, computing system 104, external device 106, server-side system 130, and / or other devices or components, according to exemplary embodiments of the present disclosure. In various embodiments, any of the systems of this specification may be or include computer 500, for example, a data communication interface 520 for packet data communication. Computer 500 may communicate with one or more other computers, for example, using an electronic network 525 (for example, via the data communication interface 520). The electronic network 525 may include, for example, a wired network or a wireless network similar to network 140 shown in Figure 1.

[0067] Computer 500 may also include a central processing unit ("CPU") in the form of one or more processors 502 for executing a set of program instructions 524. The set of program instructions 524 may include at least a set of instructions for performing image processing, including image color correction (for example, if computer 500 is a computing system 104).

[0068] Computer 500 may include an internal communication bus 508. Computer 500 may also include a drive unit 506 (such as read-only memory (ROM), a hard disk drive (HDD), or a solid-state disk drive (SDD)) that can store data in a computer-readable medium 522 (e.g., a non-temporary computer-readable medium), but computer 500 may receive programming and data via network communication. Computer 500 may also have memory 504 (such as random-access memory (RAM)) that stores instruction sets 524 for performing the techniques presented herein. However, it should be noted that in some embodiments, instruction sets 524 may be temporarily or permanently stored in other modules of computer 500 (e.g., a processor 502 and / or computer-readable medium 522). Computer 500 may also include user input and output devices 512 and / or displays 510 for connecting to input and / or output devices such as a keyboard, mouse, touchscreen, monitor, or display. Various system functions may be implemented in a distributed manner across multiple similar platforms to distribute the processing load. Alternatively, the system can be implemented through appropriate programming on a single computer hardware platform.

[0069] The program aspects of this technology can typically be considered as “products” or “manufactured items” in the form of executable code and / or related data, carried on or embodied within a certain type of machine-readable medium. “Storage” types of media include tangible memory (various semiconductor memories, tape drives, disk drives, etc.) of any or all of a computer, processor, etc., or their related modules, which can provide non-temporary storage at any given time for software programming. All or part of the software may sometimes be transmitted through the Internet or various other telecommunication networks. Such communications can, for example, enable the loading of software from one computer or processor to another. Therefore, other types of media that can carry software elements include light waves, radio waves, and electromagnetic waves, which are used across physical interfaces between local devices, through wired and optical fixed-line networks, and via various wireless links. Physical elements that carry such waves, such as wired or wireless links and optical links, can also be considered media that carry software. As used herein, unless limited to non-temporary, tangible “storage” media, terms such as computer or machine “readable media” refer to any medium involved in providing a set of instructions to a processor for execution.

[0070] The principles of this disclosure are described herein with reference to exemplary examples for specific applications, but it should be understood that this disclosure is not limited thereto. Those skilled in the art and those with access to the teachings provided herein will recognize that all additional modifications, applications, and substitutions of equivalents fall within the scope of the examples described herein. Accordingly, the present invention should not be considered limited by the foregoing description.

Claims

1. A method for color correcting images performed by a computing system, At the first time point, First image data captured by the imaging device of the medical imaging system during a medical procedure, A step of receiving a first value of the operating parameter of the light source of the medical imaging system, wherein the color of the light emitted from the light source shifts based on the value of the operating parameter. A step of determining a first color shift based on the first value, The steps include: applying a color correction function to the first image data to compensate for the first color shift to generate a first color-corrected image data; The steps include providing the first color-corrected image data for display on a display associated with the computing system, At a second time different from the first time, The second image data captured by the imaging device during the aforementioned medical procedure, A step of receiving a second value of the operating parameter of the light source, which is different from the first value. A step of determining a second color shift based on the second value, The steps include applying the color correction function to the second image data to compensate for the second color shift and generating a second color-corrected image data, A method comprising the step of providing the second color-corrected image data to the display for display.

2. The method according to claim 1, wherein multiple color shifts of light emitted from one or more light sources of one or more medical imaging systems, which include at least the light source of the medical imaging system, are characterized by multiple values ​​of the operating parameter, and color shift data is acquired and stored.

3. The step of determining the first color shift is: The method according to claim 2, comprising identifying a first color shift data corresponding to the first value in the stored color shift data, wherein the first color shift data indicates a measured change in the pixel intensity value of one or more image data components based on the first color shift of the light emitted by the light source, when the operating parameter has the first value.

4. Applying the color correction function to the first image data means that The method according to claim 3, comprising determining one or more first compensation coefficients of the color correction function based on the first color shift data in order to compensate for the measured change in the pixel intensity value of one or more image data components.

5. The first color shift data further indicates the timing associated with the measured change in the pixel intensity value of one or more image data components when the operating parameter has the first value, and applying the color correction function to the first image data is: The method according to claim 3 or claim 4, comprising applying the color correction function to the first image data based on the timing.

6. Determining the second color shift is The method according to claim 2, comprising identifying a second color shift data corresponding to the second value in the stored color shift data, wherein the second color shift data indicates a measured change in the pixel intensity value of one or more image data components based on the second color shift of the light emitted by the light source, when the operating parameter has the second value.

7. Applying the color correction function to the second image data means that The method according to claim 6, comprising determining one or more second compensation coefficients of the color correction function based on the second color shift data in order to compensate for the measured change in the pixel intensity value of one or more image data components.

8. The second color shift data further indicates the timing associated with the measured change in the pixel intensity value of one or more image data components when the operating parameter has the second value, and applying the color correction function to the second image data is: The method according to claim 6 or 7, comprising applying the color correction function to the second image data based on the timing.

9. The method according to any one of claims 1 to 8, further comprising the step of determining whether the difference between the second value and the first value exceeds a predetermined threshold.

10. The aforementioned operating parameter is the current of the light source, and receiving the first value and the second value of the aforementioned operating parameter is The method according to any one of claims 1 to 9, comprising receiving the first value and the second value of the current from the light source.

11. The aforementioned operating parameter is the temperature of the light source, and receiving the first and second values ​​of the aforementioned operating parameter is The method according to any one of claims 1 to 10, comprising receiving a first value of the temperature and a second value of the temperature from a temperature sensor positioned adjacent to the light source at the first time and the second time, respectively.

12. The aforementioned operating parameter is the temperature of the light source, and receiving the first and second values ​​of the aforementioned operating parameter is The method according to any one of claims 1 to 10, comprising estimating a first value of the temperature and a second value of the temperature based on a known ambient temperature, a known junction thermal resistance of the light source, and current values ​​of the light source received at a first time and a second time, respectively.

13. Applying the color correction function to the first image data means that Determining a first timing associated with the first color shift, wherein the first timing includes a measured period from the detection of the operating parameter of the light source operating at a first value to the observation of the first color shift. The method according to any one of claims 1 to 12, comprising applying the color correction function to the first image data based on the first timing.

14. Applying the color correction function to the second image data means that Determining a second timing associated with the second color shift, wherein the second timing includes a measured period from the detection of the operating parameter of the light source operating at the second value to the observation of the second color shift. The method according to any one of claims 1 to 13, comprising applying the color correction function to the second image data based on the second timing.

15. The aforementioned operating parameter is a first operating parameter, and the method is The steps include receiving, at the first time, the first value of the second operating parameter of the light source in addition to the first value of the first operating parameter, The method according to any one of claims 1 to 14, further comprising the step of determining that the first operating parameter being at a first value has a more dominant effect on the color of light emitted from the light source than the second operating parameter being at a first value, wherein the determination determines the first color shift and the associated timing of the first color shift based on the first value of the first operating parameter.

Images