Method and apparatus for color correction of a system of more than two self-illuminating cameras

Abstract

An apparatus and method for color correcting two camera systems, each having an imaging device and an illumination source. The camera systems are each white balanced with the imaging device outside of a scene. Their white balance gains and color correction matrices are saved. Based on the measurements, a combined white balance gain and a combined color correction matrix are calculated and saved. Thereafter, white balancing is performed by measuring the R, G, and B values of both imaging devices together with the imaging device in the scene, and then by measuring the R, G, and B values of each individual imaging device with the lights of the other camera system turned off. The scene measurements for each camera are compared to the combined scene measurements. If they differ significantly, a combined light set of white balance gains and color correction matrices is applied to the digital signal processing path of each camera system.

Description

Technical Field

[0001] The present invention relates to methods, systems, and apparatus for imaging structures using endoscopic camera systems, and more particularly, to methods and apparatus for providing color correction for two or more self-illuminating camera systems. Background Art

[0002] Endoscopic medical devices (also referred to herein as "endoscopes") that include endoscopic camera systems provide the surgeon with a view of the area of the patient's body being treated during endoscopic surgery without having to completely open the area, thereby facilitating minimally invasive surgery. In endoscopic surgery, illumination is typically directed into very small, tightly enclosed cavities and spaces in the body. Such illumination can be provided by optical fibers or other light channels that carry light from a distant light source. An illumination source, such as an LED, can also be located on the endoscope and focused directly outside the endoscope. Some endoscopes use light channels (e.g., optical fibers or translucent plastic light pipes) to direct light from an onboard LED to a point outside the endoscope. The illumination source can also be one or a combination of xenon lamps, halogen lamps, LED lamps, and laser lamps.

[0003] The surgical field during endoscopic surgery can present unique lighting and imaging challenges. When viewing an object under illumination using a camera system, unrealistic color casts often occur in the image, which can make visualization of the surgical field difficult. This is particularly true within the tight confines of endoscopic surgery, where the distance between the light source and the illuminated tissue can vary and where reflected light can be affected by the color of the illuminated tissue and the shadows cast.

[0004] White balancing (WB) is the process of removing unrealistic color casts, performing color correction, and establishing color fidelity so that objects that appear white by themselves appear white in photographs or visual displays such as computer screens. This is critical because practicing physicians often rely on color when making diagnoses and other medical decisions. Proper camera white balancing must take into account the color temperature of the light source. Color temperature refers to the relative warmth or coolness of white light. While the human eye is good at judging what is white under different light sources, digital cameras can struggle with automatic white balancing (AWB) and, as a result, can produce unsightly blue, orange, or even green color casts that can interfere with optimal use of the endoscope.

[0005] Endoscopic camera systems may include two or more cameras and light sources arranged at different angles, allowing for a magnified field of view (e.g., by stitching together images from different cameras) and / or allowing for different views without having to reposition an endoscope. Unlike single-camera systems, the process of white balancing becomes increasingly complex when dealing with multi-camera endoscopes. When a white-balanced, self-illuminating camera system (CS2) is used as a secondary (or tertiary, quadruple, etc.) view for another white-balanced, self-illuminating camera system (CS1), the scene is being illuminated by two (or more) independent light sources. The color temperature of the scene illumination (CS1 and CS2) will vary depending on the position of each system in the scene and the intensity of each system's illumination. The color temperature may also be affected by what tissue / structure is being illuminated by a particular light source (e.g., white tissue versus red tissue), the distance between the camera lens and the illuminated tissue, and the type of light source used. This variation in scene illumination color temperature can result in erroneous color reproduction in one or more of the camera systems involved.

[0006] Current systems for white balancing multi-endoscopic camera systems rely on continuously white balancing each of the multiple cameras against a white background before the procedure begins. These prior art systems are unable to achieve white balance after the procedure has begun, and in fact, if white balancing is accidentally attempted, the endoscope must often be removed and its white balance recalibrated.

[0007] Therefore, there remains a need for a system to correct the color reproduction errors inherent in systems with multiple self-illuminating cameras. Summary of the Invention

[0008] The present invention provides a color correction system for at least two self-illuminated endoscopic camera systems, the color correction system comprising: a central processing unit; an input / output interface for a camera control unit; and a memory for retaining collected data and software or firmware for operating the color correction system; wherein each of the at least two self-illuminated endoscopic camera systems comprises an imaging device and an illumination source.

[0009] In another embodiment, the present invention provides a color correction system for at least two self-illuminated endoscopic camera systems, the color correction system comprising: a central processing unit; an input / output interface for a camera control unit; and a memory for retaining collected data and software or firmware operating the color correction system; wherein each of the at least two self-illuminated endoscopic camera systems comprises an imaging device and an illumination source; and wherein one of the at least two self-illuminated endoscopic camera systems can act as a master device, and another of the at least two self-illuminated endoscopic camera systems can act as a slave device. In another embodiment, the present invention provides a color correction system for at least two self-illuminated endoscopic camera systems, the color correction system comprising: a central processing unit; an input / output interface for a camera control unit; and a memory for retaining collected data and software or firmware for operating the color correction system; wherein each of the at least two self-illuminated endoscopic camera systems comprises an imaging device and an illumination source; and wherein when the at least two self-illuminated endoscopic camera systems are outside a scene, the color correction system calculates and saves white balance information and a color correction matrix for each of the at least two self-illuminated endoscopic camera systems, and the color correction system further calculates and saves combined white balance information and color correction matrices for the at least two self-illuminated endoscopic camera systems; and when the at least two self-illuminated endoscopic camera systems are placed in a scene and their illumination sources are turned on, the color correction system measures The color correction system measures individual values of R (red), G (green), and B (blue) for each of the at least two self-illuminated endoscopic camera systems and stores the individual values in a memory as a combined scene measurement value, and, when the illumination source of one of the at least two endoscopic camera systems is turned on and the illumination source of the other of the at least two endoscopic camera systems is turned off, measures an average value of R (red), G (green), and B (blue) for the endoscopic camera system with the illumination source turned on and stores the average value in a memory, and repeats these measuring and storing steps except that the illumination source of the other of the at least two endoscopic camera systems is switched from off to on and the illumination source of the original endoscopic camera system is switched from on to off, compares the scene measurement values when the at least two endoscopic camera systems are turned on with the combined scene measurement value, and corrects the white balance of the at least two endoscopic camera systems, if necessary.

[0010] In another embodiment, the present invention provides a color correction system for at least two self-illuminating endoscopic camera systems, the color correction system comprising: a central processing unit; an input / output interface for a camera control unit; and a memory for retaining collected data and software or firmware operating the color correction system; wherein each of the at least two self-illuminating endoscopic cameras comprises an imaging device and an illumination source; and wherein illumination of any one of the at least two self-illuminating endoscopic camera systems can be deactivated or interleaved with frames of another of the at least two self-illuminating endoscopic camera systems.

[0011] In another embodiment, the present invention provides a color correction system for at least two self-illuminating endoscopic camera systems, the color correction system comprising: a central processing unit; an input / output interface for a camera control unit; and a memory for retaining collected data and software or firmware operating the color correction system; wherein each of the at least two self-illuminating endoscopic cameras comprises an imaging device and an illumination source; and wherein the at least two imaging devices and two illumination sources are connected to a single camera control unit.

[0012] In another embodiment, the present invention provides a color correction system for at least two self-illuminating endoscopic camera systems, the color correction system comprising: a central processing unit; an input / output interface for a camera control unit; and a memory for retaining collected data and software or firmware operating the color correction system; wherein each of the at least two self-illuminating endoscopic cameras comprises an imaging device and an illumination source; and wherein the color correction system is a separate unit interconnected to the camera control unit of each of the at least two self-illuminating endoscopic camera systems.

[0013] In another embodiment, the present invention provides an apparatus for color correction of at least two self-illuminated endoscopic camera systems, each self-illuminated endoscopic camera system having at least one imaging device and one illumination source, the apparatus for color correction calculating and storing white balance information and a color correction matrix for each of the at least two self-illuminated endoscopic camera systems, and further calculating and storing combined white balance information and a color correction matrix for the at least two self-illuminated endoscopic camera systems; and comprising: a central processing unit; an input / output interface for a camera control unit; a memory for retaining collected data and software or firmware for operating the apparatus for color correction; and a control component for controlling the operation of the color correction apparatus.

[0014] In another embodiment, the present invention provides an apparatus for color correction of at least two self-illuminating endoscopic camera systems, each self-illuminating endoscopic camera system having at least one imaging device and one illumination source, the apparatus comprising: a central processing unit; an input / output interface for a camera control unit; a memory for retaining collected data and software or firmware for operating the apparatus for color correction; and a control member for controlling the operation of the color correction apparatus; wherein when the color correction apparatus is activated, white balancing of the at least two self-illuminating endoscopic camera systems is performed when the at least two self-illuminating endoscopic camera systems are outside a scene, and then the white balancing is performed when the two self-illuminating endoscopic camera systems are placed in a scene and used for the scene at the same time.

[0015] The present invention further provides a method for color correction of at least two self-illuminating endoscopic camera systems, each of which has at least one imaging device and one illuminator, the method comprising the following steps: (A) with the at least two imaging devices oriented outside a scene; measuring and saving white balance information and a color correction matrix for each of the at least two self-illuminating endoscopic camera systems; and calculating and saving combined white balance information and color correction matrices for the at least two self-illuminating endoscopic camera systems; and (B) with the at least two imaging devices positioned in the scene; and with the illumination sources of the at least two self-illuminating endoscopic cameras turned on, measuring R (red), G (green), and and storing the individual values of R (red), G (green), and B (blue) of the endoscopic camera system with the illumination source turned on in a memory as a combined scene; with the illumination source of one of the at least two endoscopic camera systems turned on and the illumination source of the other of the at least two endoscopic camera systems turned off, measuring the average values of R (red), G (green), and B (blue) of the endoscopic camera system with the illumination source turned on and storing the average values in a memory, repeating these measuring and storing steps except that the illumination source of the other of the at least two endoscopic camera systems is switched from off to on and the illumination source of the original endoscopic camera system is switched from on to off, performing a comparison between the scene measurement values when the at least two endoscopic camera systems are turned on and the combined scene measurement value, and correcting the white balance of the endoscopic camera system if necessary.

[0016] In yet another embodiment, the present invention provides a method for color correcting at least two self-illuminating endoscopic camera systems, each self-illuminating endoscopic camera system having an imaging device, an illumination light, and a camera control unit, the method comprising the steps of: with the at least two imaging devices oriented outside of a scene; (a) white balancing the at least two self-illuminating camera systems to determine a white balance gain for each of the at least two self-illuminating camera systems, and storing the white balance gains in a memory together with a color correction matrix for each of the at least two self-illuminating camera systems; and (b) calculating a combined white balance gain and an accompanying combined color correction matrix based on each of the individual white balance gains and the color correction matrix. (c) with illumination of the at least two endoscopic camera systems turned on, respectively measuring the average values of R (red), G (green), and B (blue) within a predetermined window of the captured scene of each of the at least two endoscopic camera systems and storing the combined white balance gain and the accompanying combined color correction matrix set in a memory; and with at least two imaging devices positioned in the scene; (d) with illumination of a first of the at least two endoscopic camera systems turned on and illumination of a second of the at least two endoscopic camera systems turned off, measuring the average values of R (red), G (green), and B (blue) within a predetermined window of the captured scene of each of the at least two endoscopic camera systems and storing the measured values in a memory as a combined scene measurement value; (e) measuring an average of R, G, and B within a predetermined window of the captured scene for a second of the at least two endoscopic camera systems with illumination of the second of the at least two endoscopic camera systems turned on and illumination of the first of the at least two endoscopic camera systems turned off, and saving the average of R, G, and B for the second of the at least two endoscopic camera systems in a memory as a second scene measurement; (f) performing a first comparison between the first scene measurement and the combined scene measurement, and If the first scene measurement value is perceived to be significantly different from the combined scene measurement value upon close observation, applying the combined light set of white balance gains and color correction matrices to the digital signal processing path of the first of the at least two endoscopic camera systems; (g) performing a second comparison between the second scene measurement value and the combined scene measurement value, and if the second scene measurement value is perceived to be significantly different from the combined scene measurement value upon close observation, applying the combined light set of white balance gains and color correction matrices to the digital signal processing path of the second of the at least two endoscopic camera systems; and (h) repeating steps (f) and (g) as needed to correct the color balance of the at least two self-illuminating endoscopic camera systems.

[0017] In yet another embodiment, the present invention provides a method for color correcting at least two self-illuminated endoscopic camera systems, each having at least one imaging device and one illuminator, the method comprising the steps of: (a) first white balancing the camera systems individually outside a scene and saving their white balance gains and color correction matrices; (b) based on the measurements, calculating and saving a combined white balance gain and a combined color correction matrix; and (c) performing white balancing by measuring the R, G, and B values of the two cameras together with the two camera systems in the scene, and then performing white balancing by measuring the R, G, and B values of each individual imaging device with the light of the other camera system turned off; and (d) comparing the scene measurements of each imaging device to the combined scene measurements, and if they are significantly different, applying the combined light set of white balance gains and color correction matrices to the digital signal processing path of each camera system.

[0018] These and other features of the invention are described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a diagram depicting two endoscopic camera systems with their tips pointing into overlapping surgical fields, wherein the two endoscopic camera systems are connected to two separate camera control units.

[0020] Figure 2 is a diagram depicting two endoscopic camera systems with their tips pointing into overlapping surgical fields, where the two endoscopic camera systems are connected to a single camera control unit.

[0021] Figure 3 is an exemplary block diagram illustrating an exemplary embodiment of a portion of the color correction system of the present invention.

[0022] Figure 4 is a flow chart illustrating an exemplary embodiment of a method for color correction of two or more self-illuminating camera systems of the present invention.

[0023] Figure 5 is a schematic diagram depicting the fields of view of two highly overlapping camera systems, where the corresponding color correction windows of the fields of view are completely within the overlapping region.

[0024] Figure 6 is a schematic diagram depicting the fields of view of two camera systems with little overlap, where the corresponding color correction windows of the fields of view are only partially in the overlapping region. DETAILED DESCRIPTION

[0025] In view of the limitations on white balance of multi-camera endoscopes described above, the present inventors have developed a system that addresses these shortcomings.

[0026] First turn Figure 1 , a diagrammatic view illustrating a first exemplary arrangement of two separate endoscopic camera systems 10A and 10B (where an "endoscopic camera system" is sometimes referred to herein as an "endoscope") is shown. Each endoscopic camera system 10A and 10B generally includes a handle 12 and an endoscope 14 extending from the handle, wherein each endoscope 14 has a distal end 16. An imaging device 18, such as a charge coupled device (CCD), can be located at the distal end 16. The imaging device can also include optics and electromechanical components. Alternatively, the imaging device 18 can be located, for example, in the handle 12, or even external to the endoscopic camera system, where an image guide is used to transmit the image being viewed to the imaging device 18 (not shown). At least one illumination light source 20, such as an LED light, can be positioned at the distal end 16 of the endoscope 14. Alternatively, the illumination light source 20 can be positioned remote from the distal end 16, where light is transmitted out of the distal end via a connecting light pipe (not shown). Alternatively, the illumination light source 20 for the camera system may be located elsewhere, with the light transmitted to the surgical site.

[0027] like Figure 1 As further shown in FIG, each handle 12 is connected to a camera control unit A 30A and a camera control unit B 30B via input plugs 32A and 32B via data and power cables 22A and 22B, respectively. The endoscope can be a laparoscope, an arthroscope, or the like. If either or both of the endoscopic camera systems 10A and 10B are wireless devices, then instead of physical data and power cables 22A and 22B, a wireless data link (not shown) exists between the handle 12 and the camera control units 30A and 30B. As will be understood by those skilled in the art based on this disclosure, three or more endoscopic camera systems can also be used together. However, for the remainder of the discussion, two endoscopic systems are described.

[0028] Camera control unit A 30A and camera control unit B 30B can be connected to displays A 34A and B 34B, respectively, to display images from their respective endoscope camera systems 10A and 10B. For user convenience, in some cases, it may be possible to have camera control unit A 30A and / or camera control unit B 30B display a composite image of the scene being viewed on one or more of displays A 34A or B 34B. Alternatively, a single display with a split-screen function (not shown) can be used in place of displays A 34A and B 34B.

[0029] exist Figure 1, two endoscopes 14 of two endoscopic camera systems 10A and 10B are shown as being placed in a surgical field SF and their corresponding illumination sources 20, which, when turned on simultaneously, illuminate the surgical field SF and affect the overall light color, intensity, and quality in the surgical field SF. Although the term surgical field is used herein, the term may refer to any location or "scene" in which two or more endoscopic camera systems 10A and 10B are used. Although the illumination source 20 is shown as being approximately in the same plane as the imaging device 18 at the distal tip 16, if desired, the illumination source 20 can be positioned so that its light is not directly aimed in the field of view of the imaging device 18. The illumination sources 20 in the endoscopic camera systems 10A and 10B can have the same or different desired illumination characteristics, such as light frequency, including light color, intensity, illumination focus (e.g., narrow spot light versus wide divergent light), and direction.

[0030] In general, at least two endoscopic camera systems 10A and 10B may be identical or may differ in design and operation, including the make, model, and type of their cameras and the number and characteristics of their light sources. Thus, when in operation, image data collected by endoscopic camera systems 10A and 10B is transmitted to camera control unit A 30A and camera control unit B 30B, respectively, via data / power cables 22A and 22B. Camera control unit A 30A and camera control unit B 30B may be interconnected via a data link 36, which connects camera control unit A 30A and camera control unit B 30B via a color correction unit 24. The color correction unit may correlate the data collected by the two endoscopic camera systems 10A and 10B and their respective connected control units A 30A and B 30B, and may contain the hardware, firmware, and software necessary to implement the color correction system of the present invention. Alternatively, the camera control units may be connected to each other, with one of the camera control units containing the color correction unit 24.

[0031] Each control unit A 30A and camera control unit B 30B may optionally include white balance reset and correction buttons 38A and 38B, respectively, to allow an operator to manually reset and correct white balance. Alternatively, color correction unit 24 may include white balance reset and color correction buttons 26. As an alternative to having a dedicated color correction unit 24, a general-purpose computer loaded with software may instead be connected to camera control units 30A and 30B and used to perform the steps of the present invention as further described below. In the case of a general-purpose computer, there would be no separate and dedicated white balance reset and color correction buttons 26, 38A, or 38B, as such a computer program may include predetermined keyboard buttons to click or software icons to select.

[0032] Furthermore, the system of the present invention can be configured so that initial white balancing of two or more endoscopes can be activated before the endoscopes are inserted into the surgical field by launching a computer program in a dedicated color correction unit 24 or a general-purpose computer. Thereafter, with the endoscopes placed in the surgical field, the color correction step can be automatically performed without the operator having to push any buttons, for example, if the system detects that one or more of the endoscopes has been moved or that the lighting conditions in the surgical field have changed. Further details of the operation of the white balance reset process are discussed further below.

[0033] Steering Figure 2 , shows a difference in a first exemplary arrangement of two separate endoscope camera systems 10A and 10B, wherein instead of Figure 1 In the case of an arrangement in which each endoscope camera system 10A and 10B is connected to its own dedicated camera control unit A 30A and camera control unit B 30B, at least two endoscope camera systems 10A and 10B (as shown in FIG. Figure 1 42B).

[0034] Camera control unit AB 40 may optionally be equipped with a white balance reset and correction button 44 to allow an operator to manually reset and correct the white balance of endoscopic camera systems 10A and 10B. A color correction unit 46 may be interconnected to camera control unit AB 40 via a data link 50 and correlate the data collected by the two endoscopic camera systems 10A and 10B. It may also include the hardware, firmware, and / or software necessary to perform the color correction process of the present invention. The color correction unit 46 may include a white balance reset and color correction button 48, thereby eliminating the need for the white balance reset and correction button 44 on camera control unit AB 40.

[0035] and Figure 1 As an alternative to a dedicated color correction unit 46, a general-purpose computer loaded with software may instead be connected to the camera control unit 40 and used to perform the steps of the present invention as described further below. If so, there would be no separate white balance reset and color correction buttons 44 or 48, as such a computer program may include predetermined keyboard buttons for clicking or software icons for selecting.

[0036] Furthermore, the system of the present invention can be configured so that initial white balancing of two or more endoscopes can be activated before the endoscopes are inserted into the surgical field by launching a computer program in a dedicated color correction unit 46 or a general-purpose computer. Thereafter, with the endoscopes placed in the surgical field, the color correction step can be performed automatically without the operator having to push any buttons, for example, if the system detects that one or more of the endoscopes has been moved or that the lighting conditions in the surgical field have changed. Further details of the operation of the white balance correction process will be discussed further below.

[0037] Having described the layout of a typical physical hardware of an endoscope, including an endoscope camera system, a camera control unit, a display, and their interconnections, we now turn to a description of an exemplary embodiment of the color balance error correction system of the present invention. Figure 3 is an exemplary block diagram illustrating an exemplary embodiment of the color correction unit 24 / 46 of the present invention. The color correction unit 24 / 46 includes a CPU 60, a memory unit 62, an input / output interface 64 for the camera control unit with a data link 36 / 50, a power supply / power input 66, and white balance reset and color correction buttons 26 / 48 to initiate the process as will be further described below. As noted above, instead of a dedicated color correction unit 24 / 46, the functionality and operation of the color correction unit 24 / 46 can be replicated in a general-purpose computer that can be connected to both endoscopes and / or to the camera control unit.

[0038] As described above, the color balance error correction system of the present invention may reside in a separate module or unit, such as the one described above in Figure 1 and 2 The color correction unit 24 / 46 described in

[15] is also described above. The possibility of replicating the functionality in a general-purpose computer is also discussed above. Another possibility is to integrate the required features of the color correction unit directly into one or more of the camera control unit A30A, camera control unit B 30B, or camera control unit AB 40. As long as the functionality is provided, the hardware, firmware, and software can reside anywhere in the setup.

[0039] Therefore, regardless of the physical location of the necessary electronics, firmware, and / or software in the setup, the present invention follows these steps to correct color balance errors when two or more endoscopic camera systems are used during an endoscopic procedure and their lights are simultaneously on. The imaging device can be located at the distal end of the endoscope, such as a camera chip on the tip, in the proximal end of the endoscope, or elsewhere as needed. Furthermore, the imaging device can be reusable or disposable. The camera systems can function so that one camera system acts as a master and the other as a slave. Finally, the illumination of either camera system can be deactivated or interleaved with the frames of the other camera system.

[0040] Figure 4 is a flow chart illustrating an exemplary embodiment of a method and system for color correction of two or more self-illuminating camera systems of the present invention.

[0041] In a first step 100 of the method and system in the flowchart, a white balance is typically pre-formed for each camera system using its respective light source. In an embodiment, this is achieved by the imaging devices of the endoscopic camera systems A and B being outside the surgical field and directed at a length of pure white material, such as a pure white card or a piece of white gauze. The camera systems are cycled through and individual white balance gains WB1 and WB2 are stored in separate white balance gain memories for each individual digital signal processing path of each camera. Each signal processing path also has its own color correction matrix, CC1 and CC2, respectively, which is also stored in memory along with WB1 and WB2, for example, in Figure 3 The color correction unit 24 / 46 is stored in the memory unit 62. As noted above, instead of having a dedicated color correction unit, the functions and steps of the present invention can be performed by a general-purpose computer connected to the camera control unit, or even performed in the camera control unit. However, for the purpose of discussing this embodiment of the steps and method, it will be assumed that a dedicated color correction unit is present.

[0042] Next, in step 102, a set of combined light sets of white balance gains WB1′ and WB2′ and accompanying sets of combined light sets of color correction matrices CC1′ and CC2′ are calculated based on the individual white balance gains WB1 and WB2 and the individual color correction matrices CC1 and CC2 for each camera determined from step 100. The calculation of the WB1′ and WB2′ and CC1′ and CC2′ values may be calculated by the color correction unit 24 / 46 and stored in its memory unit 62.

[0043] The next set of steps occurs during surgery with the endoscopic camera systems 10A and 10B located within the body (in the surgical field SF) or "scene" and generally within Figure 4In step 114, camera system 1 (CS1) and camera system 2 (CS2) (which may correspond to endoscope camera systems 10A and 10B, respectively, as shown in FIG. Figure 1 and 2

[0014] The endoscope (shown in FIG. 1 ) is physically positioned within a body part (surgical field SF) for use. In the system of the present invention, the following steps are performed periodically. The cycles of the steps can be at predetermined intervals, or can be triggered or increased whenever movement is detected in one or more of the endoscopes or in some other manner. Exemplary formulas used in this process are discussed further below.

[0044] In step 116, with both CS1 and CS2 light sources on (e.g., illumination light source 20 in each of endoscopic camera systems 10A and 10B), the system measures the average values of R (red), G (green), and B (blue) within a predetermined window of the captured scene for CS1 and CS2, respectively. The predetermined window of the captured scene will be associated with the view that each imaging device of each endoscopic camera system will capture at any moment. When both endoscopes have their imaging devices pointing toward the majority of the same area in the surgical field, the captured images of CS1 and CS2 will be in an area of significant overlapping illumination from both endoscopes, and the illumination from each endoscope's illumination light source will have a greater impact on the overall quality, color, and degree of illumination picked up by each respective camera system CS1 and CS2. In contrast, when the two endoscopes have their imaging devices pointing in different directions, the area of overlapping illumination from the two endoscopes will be smaller, and the image captured by each individual camera system CS1 or CS2 will be less affected by the illumination from the other camera system CS1 or CS2.

[0045] For example, Figure 5 As shown in , the endoscopes (CS1 and CS2) are (mostly) pointed at the same area in the surgical field, and the image frames captured by CS1 will be in the illumination overlap region 120 of the light projection fields 122A and 122B from the respective light sources of CS1 and CS2. A small segment of each image frame, namely the CS1 window 124A for CS1 and the CS2 window 124B for CS2, is the area in each camera system where light is sampled for white balance purposes. Figure 5 , CS1 window 124A and CS2 window 124B are completely within the illumination overlap region 120. However, referring to the schematic Figure 6If the endoscopes (CS1 and CS2) are not pointed at the same area in the surgical field, the illumination overlap area 120' of the light projection fields 122A' and 122B' from the respective light sources of CS1 and CS2 will be smaller, and the white balance segments of the image frames captured by CS1 and CS2, as sampled by the CS1 window 124A' and the CS2 window 124B', respectively, will not be completely within the illumination overlap area 120'. In this case, the images captured by each individual camera system CS1 or CS2 and their white balance segments CS1 window 124A' and CS2 window 124B' will be less affected by the illumination from the other camera system CS1 or CS2.

[0046] The predetermined windows CS1 window 124A / 124A' and CS2 window 124B / 124B' may be conventional exposure control or white balance windows. If each predetermined window is smaller, it is more likely to be in the overlapping region of the two camera images CS1 and CS2 and should result in a more appropriate color correction (see Figure 5 ). However, if Figure 6 As shown in , if each window (CS1 window 124A' and CS2 window 124B') does not reside completely within or outside of illumination overlap region 120', the correction will be less effective, but still in the right direction and can be used. Regardless of the degree of overlap, the average values of R, G, and B within a predetermined window of the captured scene for the CS1 window and the CS2 window are collected in a memory, such as in memory unit 62 of color correction unit 24 / 46, as measurement value SceneAB for further processing.

[0047] In step 118, with only CS1 lighting on, the system measures the average values of R, G, and B within a predetermined window of the captured scene for CS1. These average values of R, G, and B are stored in the memory unit 62 of the color correction unit 24 / 46 as the measured values SceneA for further processing. The color correction unit 24 / 46 can control CS1 and CS2 to automatically achieve the lighting conditions of CS1 and CS2.

[0048] In step 120, with only CS2 lighting on, the system measures the average values of R, G, and B within a predetermined window of the captured scene for CS2. These average values of R, G, and B are stored in a memory, such as in the color correction unit 24 / 46, as measurement SceneB for further processing. The color correction unit 24 / 46 can control CS1 and CS2 to automatically achieve the lighting conditions of CS1 and CS2.

[0049] Next, in step 122, a first comparison is performed. If the detected SceneA measurement is perceived to differ significantly from the detected SceneAB measurement upon close observation, WB1' and CC1' are applied to the CS1 digital signal processing path of the first camera system. Scene measurement can be performed, for example, using the Commission Internationale de l'Eclairage (CIE) distance metric ΔE*ab (also known as ΔE*, or inaccurately as dE*, dE, or "ΔE"), where Δ is the Greek letter commonly used to represent difference and E stands for Empfindung; German for "feeling." Alternative formulas include CIE76, CIE94, and CIEDE2000, as well as CMC1:c (1984). See Backhaus, W.; Kliegl, R.; Werner, JS (1998), Color Vision: Perspectives from Different Disciplines, Walter de Gruyter, p. 188, ISBN 9783110154313; Valberg, A. (2005), Light Vision Color, Wiley, p. 278, ISBN 9780470849026; Sharma, Gaurav (2003), Digital Color Imaging Handbook (ed. 1.7.2.), CRC Press, ISBN 0-8493-0900-X and http: / / zschuessler.github.io / DeltaE / learn / / . However, regardless of the mathematical approach, the present inventors describe this herein as "perceivable as significantly different upon close observation."

[0050] Likewise, a second comparison is performed in step 124. If the detected SceneB measurement is perceived to be significantly different from the detected SceneAB measurement by close observation, WB2' and CC2' are applied to the CS2 digital signal processing path of the second camera system.

[0051] In step 126, steps 122 and 124 are typically repeated as needed to correct color balance as the CS1 and CS2 images change using the formula below. As noted above, movement of CS1 or CS2 can trigger the operation of steps 114-126, or if the system detects that white balance correction is required at a predetermined interval between steps, the cycle can be increased to ensure that the image being viewed is correctly white balanced.

[0052] Individual steps 118 and 120, in which the selected light sources are turned off, may last for a single video frame or for several video frames. When one of the illumination sources is operating alone, such as CS1 illumination, the image from CS2, which has its illumination briefly turned off, may be dark. That is, the color correction step may disrupt acceptable quality video frames from one or both imaging devices. In situations of this nature, previous CS1 video frames may be inserted until CS1 light returns, as shown in optional step 128, and previous CS2 video frames may be inserted until CS2 light returns, as shown in optional step 130. Previously saved frames should be replaced as needed to conceal missing or poor quality frames.

[0053] For example, the group of steps 112 (steps 114-130) may be initiated when an operator pushes the white balance reset and correction button 26, 38A, 38B, and / or 44 on the applicable camera control unit 30A, 30B, 40 or color correction unit 24 or 46, or may be automatically and periodically implemented when two endoscopes are used as described above.

[0054] The calculations for correcting the white balance gain for red (R) are presented below. The calculations for green (G) and blue (B) follow the same formulas and are presented after the calculation of the white balance gain for red (R). These formulas, like other data utilized in this method, are stored, for example, in the memory unit 62 of the color correction unit 26 / 48. They can also reside in computer software on a general-purpose computer, a camera control unit, or can be included in one of the endoscopes.

[0055] Calculation of white balance gain for red (R) correction

[0056] R 增益 1 = Initial red white balance gain for CS1

[0057] R 增益 2 = Initial red white balance gain of CS2

[0058] R L1 = Red value measured under CS1 lighting

[0059] R L2 = Red value measured under CS2 lighting

[0060] R L1 +G L1 +B L1 = The sum of the red, green and blue values measured under CS1 illumination

[0061] R L2 +G L2 +B L2= The sum of the red, green and blue values measured under CS2 illumination

[0062] R L1L2 = Red value measured under CS1 and CS2 lighting conditions

[0063] R L1L2 +G L1L2 +B L1L2 = The sum of the red, green, and blue values measured under CS1 and CS2 illumination conditions

[0064] Equation 1R:

[0065] R 增益1已校正 =R 增益1 ×[R L1 / (R L1 +G L1 +B L1 )] / [R L1L2 / (R L1L2 +G L1L2 +B L1L2 )]

[0066] Equation 2R:

[0067] R 增益2已校正 =R 增益2 ×[R L2 / (R L2 +G L2 +B L2 )] / [R L1L2 / (R L1L2 +G L1L2 +B L1L2 )]

[0068] Calculation of white balance gain for green (G) correction

[0069] G 增益 1 = Initial green white balance gain for CS1

[0070] G 增益 2 = Initial green white balance gain of CS2

[0071] G L1 = Green value measured under CS1 lighting

[0072] G L2 = Green value measured under CS2 lighting

[0073] R L1 +G L1 +G L1 = The sum of the red, green and blue values measured under CS1 illumination

[0074] RL2 +G L2 +B L2 = The sum of the red, green and blue values measured under CS2 illumination

[0075] G L1L2 = Green value measured under CS1 and CS2 lighting conditions

[0076] R L1L2 +G L1L2 +B L1L2 = The sum of the red, green, and blue values measured under CS1 and CS2 illumination conditions

[0077] Equation 1G:

[0078] G 增益1已校正 =G 增益1 ×[G L1 / (R L1 +G L1 +B L1 )] / [G L1L2 / (R L1L2 +G L1L2 +B L1L2 )]

[0079] Equation 2G:

[0080] G 增益2已校正 =G 增益2 ×[G L2 / (R L2 +G L2 +B L2 )] / [G L1L2 / (R L1L2 +G L1L2 +B L1L2 )]

[0081] Calculation of white balance gain for blue (B) correction

[0082] B 增益 1 = Initial blue white balance gain of CS1

[0083] B 增益 2 = Initial blue white balance gain of CS2

[0084] B L1 = Blue value measured under CS1 lighting

[0085] B L2 = Blue value measured under CS2 lighting

[0086] R L1 +G L1 +B L1= The sum of the red, green and blue values measured under CS1 illumination

[0087] R L2 +G L2 +B L2 = The sum of the red, green and blue values measured under CS2 illumination

[0088] B L1L2 = Bl; value measured under CS1 and CS2 lighting conditions

[0089] R L1L2 +G L1L2 +B L1L2 = The sum of the red, green, and blue values measured under CS1 and CS2 illumination conditions

[0090] Equation 1B:

[0091] B 增益1已校正 =B 增益1 ×[G L1 / (R L1 +G L1 +B L1 )] / [B L1L2 / (R L1L2 +G L1L2 +B L1L2 )]

[0092] Equation 2B:

[0093] B 增益2已校正 =B 增益2 ×[B L2 / (R L2 +G L2 +B L2 )] / [B L1L2 / (R L1L2 +G L1L2 +B L1L2 )]

[0094] In the above article about R 增益 1 已校正 , G 增益 1 已校正 and B 增益 1 已校正 In the first equation, each numerator is the fraction of red (or green or blue) in the measurement window under CS1 illumination. The denominator is the fraction of red (or green or blue) in the measurement window under CS1 and CS2 illumination. The ratio of these measured red fractions (or green or blue) is used to correct the red white balance gain, green white balance gain, and blue white balance gain.

[0095] As noted above, the imaging devices in the individual endoscopes are connected to coordinate the timing of performing the noted steps of the procedure. This can be done, for example, via a data link 36 through the color correction unit 24 to the two camera control units A and B. Figure 1 As shown in FIG, the color correction unit 46 connected to the camera control unit AB is performed as shown in FIG. Figure 2 and / or via a general purpose computer in which the system's software is connected to the camera control unit and the endoscope, or via a color correction unit inside the camera control unit.

[0096] However, having disclosed a preferred embodiment of this invention, one of ordinary skill in this art would recognize that certain modifications would come within the scope of this invention.

Claims

1. A color correction system for at least two self-illuminating endoscopic camera systems, the color correction system comprising: Central processing unit; An input / output interface for a camera control unit; as well as a memory for retaining collected data and software or firmware for operating the color correction system; wherein each of the at least two self-illuminating endoscopic camera systems comprises an imaging device and an illumination source; wherein, when the at least two self-illuminating endoscopic camera systems are outside the scene, the color correction system calculates and stores white balance information and a color correction matrix for each of the at least two self-illuminating endoscopic camera systems, and the color correction system further calculates and stores combined white balance information and a color correction matrix for the at least two self-illuminating endoscopic camera systems; and With the at least two self-illuminating endoscopic camera systems placed in a scene and their illumination sources turned on, the color correction system measures individual values of R red, G green, and B blue for each of the at least two self-illuminating endoscopic camera systems and saves the individual values in the memory as combined scene measurements, and the color correction system measures average values of R red, G green, and B blue for the endoscopic camera system with the illumination source turned on and stores the average values in the memory, with the illumination source of one of the at least two endoscopic camera systems turned on and the illumination source of the other of the at least two endoscopic camera systems turned off, and repeats this measuring and storing step except that the illumination source of the other of the at least two endoscopic camera systems is switched from off to on and the illumination source of the original endoscopic camera system is switched from on to off, A comparison is performed between scene measurement values of the at least two endoscope camera systems when they are turned on and the combined scene measurement value, and if necessary, a white balance correction is performed on the at least two endoscope camera systems.

2. The color correction system of claim 1 , wherein the camera control units in each of the at least two self-illuminating endoscopic camera systems are connected via a data link to coordinate the timing of the measurement steps. 3 . The color correction system according to claim 1 , wherein one of the at least two self-illuminating endoscopic camera systems is capable of functioning as a master device, and the other one of the at least two self-illuminating endoscopic camera systems is capable of functioning as a slave device.

4. The color correction system of claim 1, wherein illumination of any one of the at least two self-illuminating endoscopic camera systems can be deactivated or interleaved with frames of another one of the at least two self-illuminating endoscopic camera systems.

5. The color correction system of claim 1, wherein at least two imaging devices and two illumination sources are connected to a single camera control unit.

6. The color correction system of claim 1, wherein the color correction system is a separate unit interconnected to a camera control unit of each of the at least two self-illuminating endoscopic camera systems.

7. The color correction system of claim 1, wherein the color correction system comprises hardware, firmware, or software on a camera control unit of a first endoscopic camera system interconnected to a second camera control unit of a second endoscopic camera system.

8. The color correction system of claim 1, wherein the color correction system comprises hardware, firmware, or software on at least one of the at least two self-illuminating endoscopic camera systems.

9. An apparatus for color correction of at least two self-illuminating endoscopic camera systems, each self-illuminating endoscopic camera system having at least one imaging device and one illumination source, the apparatus performing the steps of claim 1 and comprising: Central processing unit; An input / output interface for a camera control unit; a memory for retaining collected data and software or firmware for operating the apparatus for color correction; and A control member is used to control the operation of the color correction device.

10. A device for color correction, the device for color correction performing color correction on at least two self-illuminating endoscopic camera systems, each self-illuminating endoscopic camera system having at least one imaging device, an illumination source, and a camera control unit, the device for color correction comprising: Central processing unit; an input / output interface for the camera control unit; a memory for retaining collected data and software or firmware for operating the apparatus for color correction; and a control member for controlling the operation of the device for color correction; wherein when the device for color correction is activated, white balancing of the at least two self-illuminating endoscopic camera systems is performed when the at least two self-illuminating endoscopic camera systems are outside the scene, and then white balance correction of each of the at least two self-illuminating endoscopic camera systems is performed when the two self-illuminating endoscopic camera systems are placed in the scene and used for the scene at the same time; wherein when the at least two self-illuminating endoscopic camera systems are outside a scene, the means for color correction performs white balancing of the at least two self-illuminating camera systems to determine a white balance gain for each of the at least two self-illuminating camera systems, and the white balance gain is stored in the memory together with a color correction matrix for each of the at least two self-illuminating camera systems; and Based on each of the individual white balance gains and color correction matrices, the means for color correction calculates a combined white balance gain and an accompanying set of combined color correction matrices, and stores the combined white balance gain and the accompanying set of combined color correction matrices in the memory; With the at least two self-illuminating endoscopic camera systems positioned in a scene and illumination of the at least two endoscopic camera systems turned on, the means for color correction respectively measures an average value of R red, G green, and B blue within a predetermined window of a captured scene for each of the at least two endoscopic camera systems, and saves the average value in the memory as a combined scene measurement value; the means for color correction measuring, with illumination of a first one of the at least two endoscopic camera systems turned on and illumination of a second one of the at least two endoscopic camera systems turned off, an average value of the R, G, and B within the predetermined window of the captured scene of the one of the at least two endoscopic camera systems, and saving the average value of the R, G, and B of the first one of the at least two endoscopic camera systems in the memory as a first scene measurement value; the means for color correction measuring, with illumination of the second one of the at least two endoscopic camera systems turned on and illumination of the first one of the at least two endoscopic camera systems turned off, an average value of the R, G, and B within the predetermined window of the captured scene of the second one of the at least two endoscopic camera systems, and saving the average value of the R, G, and B of the second one of the at least two endoscopic camera systems in the memory as a second scene measurement value; the means for color correction performing a first comparison between the first scene measurement value and the combined scene measurement value and applying a combined light set of white balance gains and the color correction matrix to a digital signal processing path of the first of the at least two endoscopic camera systems if the first scene measurement value and the combined scene measurement value are perceived to be significantly different upon close observation; the means for color correction performing a second comparison between the second scene measurement value and the combined scene measurement value, and if the second scene measurement value is perceived to be significantly different from the combined scene measurement value upon close observation, the means for color correction applying a combined light set of white balance gains and the color correction matrix to a digital signal processing path of the second of the at least two endoscopic camera systems; and The device for color correction repeats the first comparison and the second comparison as needed to correct the color balance of at least two self-illuminating endoscope camera systems.

11. A method for color correction of at least two self-illuminating endoscopic camera systems, each of which has at least one imaging device and one illuminator, the method comprising the following steps: (A) When at least two imaging devices are directed outside the scene; measuring and saving white balance information and a color correction matrix for each of the at least two self-illuminating endoscopic camera systems; and calculating and storing combined white balance information and a color correction matrix for the at least two self-illuminating endoscopic camera systems; as well as (B) with the at least two imaging devices positioned in the scene; With the illumination source turned on, measuring individual values of R red, G green, and B blue for each of the at least two imaging devices and saving the individual values in a memory as a combined scene; measuring average values of R red, G green, and B blue of the endoscopic camera system with the illumination source turned on and saving the average values in the memory when the illumination source of one of the at least two endoscopic camera systems is turned on and the illumination source of the other of the at least two endoscopic camera systems is turned off, and repeating these measuring and saving steps except that the illumination source of the other of the at least two endoscopic camera systems is switched from off to on and the illumination source of the original endoscopic camera system is switched from on to off, A comparison is performed between scene measurement values of the at least two endoscope camera systems when they are turned on and the combined scene measurement value, and if necessary, a white balance of the endoscope camera systems is corrected.

12. The method for color correction of claim 11, further comprising repeating the step of performing the comparison as needed to correct the color balance of at least two self-illuminating endoscopic camera systems.

13. The method for color correction according to claim 11, wherein the at least two self-illuminating endoscopic camera systems each include a camera control unit, and the camera control units are connected via a data link to coordinate the timing of the measuring steps.

14. The method for color correction according to claim 11, wherein the color correction system comprises hardware, firmware or software on at least one camera control unit.

15. A method for color correction of at least two self-illuminating endoscopic camera systems, each self-illuminating endoscopic camera system having an imaging device, an illuminator, and a camera control unit, the method comprising the following steps: Where at least two imaging devices are oriented outside the scene; (a) white balancing the at least two self-illuminated camera systems to determine a white balance gain for each of the at least two self-illuminated camera systems and saving the white balance gains in a memory along with a color correction matrix for each of the at least two self-illuminated camera systems; and (b) calculating, based on each of the individual white balance gains and color correction matrices, a combined white balance gain and an accompanying set of combined color correction matrices, and storing the combined white balance gain and the accompanying set of combined color correction matrices in the memory; as well as In case the at least two imaging devices are positioned in the scene; (c) with illumination of the at least two endoscopic camera systems turned on, measuring the average values of R red, G green, and B blue within a predetermined window of a captured scene of each of the at least two endoscopic camera systems, respectively, and storing the values in a memory as a combined scene measurement value; (d) measuring an average value of the R, G, and B within the predetermined window of the captured scene of the one of the at least two endoscopic camera systems with illumination of the first one of the at least two endoscopic camera systems turned on and illumination of the second one of the at least two endoscopic camera systems turned off, and saving the average value of the R, G, and B of the first one of the at least two endoscopic camera systems in a memory as a first scene measurement value; (e) measuring an average value of the R, G, and B within the predetermined window of the captured scene of the second one of the at least two endoscopic camera systems with illumination of the second one of the at least two endoscopic camera systems turned on and illumination of the first one of the at least two endoscopic camera systems turned off, and saving the average value of the R, G, and B of the second one of the at least two endoscopic camera systems in a memory as a second scene measurement value; (f) performing a first comparison between the first scene measurement value and the combined scene measurement value, and applying a combined light set of white balance gains and the color correction matrix to a digital signal processing path of the first of the at least two endoscopic camera systems if the first scene measurement value and the combined scene measurement value are perceived to be significantly different upon close observation; (g) performing a second comparison between the second scene measurement value and the combined scene measurement value, and applying a combined light set of white balance gains and the color correction matrix to a digital signal processing path of the second of the at least two endoscopic camera systems if the second scene measurement value is perceived to be significantly different from the combined scene measurement value by close observation; and (h) Repeat steps (f) and (g) as needed to correct the color balance of at least two self-illuminating endoscopic camera systems.

16. The method for color correcting at least two self-illuminating endoscopic camera systems of claim 15 , wherein if the video quality from one of the at least two self-illuminating endoscopic camera systems is unacceptable due to missing video frames during a period when one of the at least two endoscopic camera systems was turned off, inserting previous video frames from the one of the at least two endoscopic camera systems to replace the missing video frames.

Images