Surgical system including a general illumination device with non-white light

Abstract

To provide a system and method of minimizing noise in the images of a surgical site by using non-white light illumination.SOLUTION: In a teleoperated surgical system 200, cameras 220L and 220R capture color images of a surgical site 203. An illumination control device 215 includes a non-white light module 217 connected between a system processing module 262 and a light source 211. The non-white light module 217 controls output illumination from an illumination device 210 so that the illumination device 210 outputs non-white light. The cameras 220L and 220R fetch images of a surgical site to display on a solid vision display 251 of a surgeon console 250. Non-white light illumination minimizes noise in the images of the surgical site displayed on the solid vision display in comparison to images using white light illumination.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] Related Applications This application claims priority to and the benefit of U.S. Patent Application No. 61 / 954,512 (filed March 17, 2014, by Jeffrey DiCarlo et al., disclosing “Surgical System Including a Non-White Light General Illuminator”), which is incorporated herein by reference.

[0002] Aspects of the invention relate to endoscopic imaging, and more particularly, to non-white light used for general illumination of teleoperated surgical systems. [Background technology]

[0003] The da Vinci® Surgical System, commercialized by Intuitive Surgical, Inc. of Sunnyvale, Calif., is a minimally invasive, teleoperated surgical system that offers many benefits to patients, including reduced trauma to the body, faster recovery, and shorter hospital stays. One feature of the da Vinci® Surgical System is its ability to provide two-channel (i.e., left and right) video capture and display visualization images to provide stereoscopic vision to the surgeon.

[0004] Such electronic stereoscopic imaging systems can output high resolution video images to the surgeon and enable features such as zoom that provide a "closed up" view that allows the surgeon to work with increased precision as well as identify specific tissue types and characteristics. However, in a typical surgical field, the image quality captured by the cameras of an electronic stereoscopic imaging system is limited by the signal-to-noise ratio of the cameras.

[0005] As the camera collects light, the captured light is converted to electrons and stored in wells of the image sensor. There is one well per pixel. Figure 1 is a schematic diagram of a well 101 for red pixels R, a well 102 for green pixels G, and a well 103 for blue pixels B. If the camera collects more electrons in the well, the signal is multiplied while the noise remains relatively constant, thereby increasing the signal-to-noise ratio, i.e., the signal captured in the well is increased relative to the noise.

[0006] The physics of light capture by a camera is that the more light a camera pixel captures, the better the camera can estimate the rate at which the light is being captured. However, if a camera pixel collects so much light that it overfills a well, the signal for that pixel is lost and is no longer valid. Thus, the camera exposure time is set to attempt to collect light to fill all of its electronic wells 101, 102, 103 as high as possible without overfilling any of its electronic wells 101, 102, 103.

[0007] In a typical surgical site scene illuminated by white light for general viewing, red is the dominant color in the scene captured by the camera because most of the reflected light is in the red spectrum compared to the blue and green spectrum.

[0008] Typically, color video cameras used in teleoperated surgical systems include a color filter array, which is a mosaic of different color filters. Ideally, each different color filter passes only a portion of the visible electromagnetic spectrum that corresponds to a particular color spectrum, for example, a first set of filters in the color filter array passes primarily red light, a second set of filters passes primarily green light, and a third set of filters passes primarily blue light.

[0009] The camera includes an image sensor with pixels that capture light that passes through a color filter array. Each pixel is a well that fills with electrons as it captures light. The set of camera pixels that capture light that passes through a first set of filters are included in the camera's first color channel. The set of camera pixels that capture light that passes through a second set of filters are included in the camera's second color channel. The set of camera pixels that capture light that passes through a third set of filters are included in the camera's third color channel.

[0010] As known to those skilled in the art, in one example, white light illumination is comprised of a combination of light in the red, green, and blue spectrums that appears white to the human eye with normal color perception. However, due to the predominant reflection of red spectrum light by the surgical site, the red pixel well 101 (FIG. 1) typically fills much faster than either the green pixel well 102 or the blue pixel well 103. To prevent the red pixel well 101 from overflowing, the camera exposure is set to limit the light it collects so that the red pixel well 101 does not overflow.

[0011] If light collection is stopped just as the well of the color channel that receives the most light is about to overflow, the result is that the wells of the other color channels may not be full, as shown in Figure 1. In the example of Figure 1, the green well 102 and the blue well 103 are less than 50% full when light collection is stopped. The signal-to-noise ratio of these less-than-full color channels is significantly less than the signal-to-noise ratio of the color channel(s) that are about to overflow. Again, for the example of Figure 1, the signal-to-noise ratio of the red channel is about 6, while the signal-to-noise ratio of each of the green and blue channels is about 3. Summary of the Invention [Problem to be solved by the invention]

[0012] The camera has a worse signal-to-noise performance when all of the camera color channel wells 101, 102, 103 are not filled. The signals from the less-than-perfect wells 102 and 103 need to have gain applied to them as part of the white balance stage of the surgical system's image processing to form an image for display. White balancing is necessary to ensure that when the camera captures an image of a white surface, this white surface appears white on the display monitor. White balancing consists of amplifying the less-than-perfect color channels (the blue and green color channels in FIG. 1), for example by applying a digital gain such that when the camera captures an image of the white surface, all color channels have the same value. Amplification of these less-than-perfect well signals increases the noise of these less-than-perfect well color signals relative to the other color signals, further increasing the noise of the final image. [Means for solving the problem]

[0013] In one aspect, non-white light from an endoscope of a teleoperated surgical system is used to illuminate the surgical site. A camera captures an image of the surgical site, which is displayed on a monitor. The non-white light illumination minimizes noise in the image of the surgical site presented on the monitor relative to an image captured and displayed on the monitor using white light illumination.

[0014] The color of light used to illuminate the surgical site is non-white light, e.g., light having a purple tint, but the image displayed on the monitor does not include this tint. To the viewer, the light illuminating the surgical site appears white when viewed on the monitor. Only when the endoscope is removed from the patient is the light emitted from the endoscope viewed directly as if the viewer were looking at non-white light. Non-white light is different from the combination of, e.g., only two narrow-spectrum light sources used for general illumination and for highlighting specific anatomical structures.

[0015] In one aspect, the apparatus includes a camera and an illumination device. The camera is configured to separate light incident on the camera into a set of pixels, each set of pixels being in a different color channel of the camera. In one aspect, the camera includes a color filter array. The color filter array is configured to separate light incident on the camera into a set of pixels.

[0016] The illuminator is configured such that the illuminator outputs non-white light such that each camera color channel has an approximately equal response to non-white light reflected from a perfect reflective surface. As used herein, a perfect reflective surface is a surface that has a response to an illumination spectrum that is spectrally uniform and equal attenuation across the entire illumination spectrum. As used herein, "approximately equal" or "substantially equal" means that the responses may not be exactly equal due to differences in the reflective properties of the reflective surface (e.g., a reflective surface may not be exactly perfectly reflective everywhere on the surface) and due to normal differences in the response of the electronic wells of the image sensor, but the responses are equal within the combined tolerance of the image sensor and surface.

[0017] In one aspect, the lighting device includes a multiple color component illumination source configured such that the lighting device outputs non-white light.

[0018] In another aspect, the apparatus also includes a controller coupled to the multiple color component illumination sources, the controller configured to weight the outputs of each of the multiple color component illumination sources such that a combination of the outputs of the multiple color component illumination sources is non-white light.

[0019] In yet another aspect, an apparatus includes a camera, an illumination device, and a control device. The camera is configured to separate light incident on the camera into color components. The color components are captured by the camera as sets of pixels. Each set of the sets of pixels is in a different camera color channel. In one aspect, the camera includes a color filter array. The color filter array is configured to separate light incident on the camera into sets of pixels.

[0020] A controller is coupled to the illuminator and is configured to adjust characteristics of the light output by the illuminator to increase a signal-to-noise ratio of pixels of one camera color channel for a color image captured by the camera.

[0021] In one aspect, the illumination device includes a plurality of color component illumination sources, and a controller coupled to the plurality of color component illumination sources, the controller configured to adjust a characteristic of at least one of the plurality of color component illumination sources to increase a signal-to-noise ratio of pixels of one camera color channel.

[0022] In one aspect, the multiple color component illumination sources are multiple light emitting diodes. In another aspect, the multiple color component illumination sources are multiple laser diodes.

[0023] In one aspect, the controller is configured to control the output of the illumination device's multiple color component illumination sources such that the illumination device outputs a non-white color, whereby each camera color channel has a substantially equal response to non-white light reflected from a perfect reflective surface. In another aspect, the controller is configured to vary the illuminance level of at least one of the illumination device's multiple color component illumination sources such that the illumination device outputs a non-white light. In another aspect, a fixed filter is used to vary the illuminance level of at least one of the illumination device's multiple color component illumination sources such that the illumination device outputs a non-white light. In yet another aspect, the variation in illuminance level that produces the non-white light is controlled to adjust for unequal power loss due to life-induced aging of the multiple color component illumination sources.

[0024] In another aspect, the device includes an image processing pipeline configured to form a high dynamic range image from a monochromatic image captured by the camera. In this aspect, the controller is configured to vary the illuminance level of at least one of the illumination sources of the multiple color components of the illumination device. Varying the illuminance level can be implemented, for example, using a rotating wheel or a liquid crystal device.

[0025] The rotating wheel has a plurality of sections, each of which has a color of one of a plurality of color lighting components, and each of which has a different light attenuation level.

[0026] In one aspect, the liquid crystal device is configured in an on / off pulse width modulated shutter mode with a variable ratio of on / off times including one or more on / off cycles per image frame capture of the camera. In another aspect, the liquid crystal device is configured as an adjustable attenuator. In yet another aspect, the liquid crystal device is configured as a tunable filter.

[0027] The method includes illuminating the scene with a non-white light configured such that each camera color channel of the camera has a substantially equal response to the non-white light reflected from a perfectly reflective surface. The method also includes capturing an image of the scene with the camera and outputting an image for display based on the captured image without white balancing the captured image.

[0028] Another method includes capturing a color image, the captured color image including sets of pixels, each set of pixels being in a different color channel of the camera, and constructing a high dynamic range image from the sets of pixels.

[0029] Yet another method includes illuminating the site with a non-white light configured such that each camera color channel of the camera has a substantially equal response to the non-white light reflected from the site, capturing an image of the site with the camera, and outputting an image for display based on the captured images. [Brief description of the drawings]

[0030] [Figure 1] FIG. 2 illustrates the prior art filling of an electronic well of an image sensor for color components of an image captured of a scene illuminated by white light. [Diagram 2] FIG. 1 is a block diagram of a portion of a teleoperated surgical system including an illumination device that outputs non-white light. [Diagram 3] FIG. 3 is a diagram illustrating an example of a response function of the camera in FIG. 2. [Figure 4] 3 is a graph of the power spectrum of each light emitting diode in one embodiment of the lighting device of FIG. 2. [Diagram 5] 3 is a graph showing a comparison of white light and non-white light from the lighting device of FIG. 2. [Figure 6A] FIG. 13 is a schematic diagram of a time sequence for generating dynamic color channel lighting control factors. [Figure 6B] FIG. 1 illustrates the filling of the electronic wells of an image sensor for color components of an image captured from a scene illuminated by non-white light. [Figure 6C] FIG. 1 illustrates the filling of the electronic wells of an image sensor for color components of an image captured from a scene illuminated by non-white light. [Figure 6D] FIG. 1 illustrates the filling of the electronic wells of an image sensor for color components of an image captured from a scene illuminated by non-white light. [Figure 7] 3A-3C are diagrams illustrating a method of how non-white light can be produced by varying the relative on / off times over the period of the illumination source output of various color components from the lighting device of FIG. 2. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] In the drawings, the first digit of a reference number identifies the figure in which the element containing that reference number first appears.

[0032] As used herein, electronic stereoscopic imaging involves the use of two imaging channels (ie, a channel for left and right images).

[0033] As used herein, a stereoscopic optical path includes two channels (e.g., channels for left and right images) within an endoscope to carry light from tissue. The light carried in each channel represents a different view of the tissue. The light can form one or more images. Without loss of generality or applicability, the aspects described more fully below can also be used in the context of a field sequential stereoscopic acquisition system and / or a field sequential display system.

[0034] As used herein, an illumination pathway includes a pathway within an endoscope that provides illumination to tissue.

[0035] As used herein, white light is visible white light that is composed of three (or more) visible color components, e.g., a red visible color component, a green visible color component, and a blue visible color component. White light can also refer to multiple continuous spectrums within the visible spectrum, such as what humans can see from, for example, a heated tungsten filament.

[0036] As used herein, non-white light is visible light composed of three (or more) visible color components, e.g., a red visible color component, a green visible color component, and a blue visible color component in a combination different from the combination used to form white light. Non-white light can also refer to multiple continuous spectra of the visible electromagnetic spectrum, e.g., a broad spectrum of wavelengths of the visible electromagnetic spectrum that is not visible to a human observer as white light but includes multiple colors of the visible spectrum. Non-white light does not include a combination of two narrow-spectrum only light sources, such as a combination of two different narrow-spectrum blue light sources, or a combination of a narrow-spectrum blue light source and a narrow-spectrum green light source used to distinguish certain tissues.

[0037] As used herein, a color component has a spectrum of wavelengths within the visible electromagnetic spectrum.

[0038] As used herein, the visible electromagnetic spectrum ranges from wavelengths of approximately 400 nanometers (nm) to 700 nm.

[0039] As used herein, a color image includes a combination of all color components of a color model, as opposed to a monochromatic color image, or a color image that includes only a combination of a subset of the color components of a color model. For example, for a color model that includes red, green, and blue color components, a color image includes a combination of the red, green, and blue color components. A red image, a green image, a blue image, a blue and green image, etc. are not included in the definition of a color image, since such images do not include a combination of all color components of the color model.

[0040] In one embodiment, light from an endoscope 201 that is part of the teleoperated surgical system 200 is used to illuminate the surgical site 203. The illumination is a non-white light, e.g., the light appears purplish when viewed directly by a human. The use of non-white illumination minimizes noise in the image of the surgical site 203 presented on a stereoscopic display 251, sometimes referred to as display 251, at the surgeon console 250. The surgeon console 250 is sometimes referred to as the console 250.

[0041] The color of light used to illuminate the surgical site 203 is non-white light, for example light having a purple tint, but the image displayed on the stereoscopic display 251 of the surgeon console 250 does not include this tint. To a viewer, when viewed through the surgeon console 250, the light illuminating the surgical site 203 appears white.

[0042] As described more fully below, the cameras 220L, 220R and image processing pipeline 240 of the teleoperated surgical system 200 modify the captured images to remove a purple tint in the surgical images displayed on the stereoscopic display 251. Only when the surgeon steps out of the surgeon console 250, pulls the endoscope 201 away from the patient, and looks directly at the light emitted from the endoscope 201 will the surgeon see the non-white light.

[0043] Aspects of the present invention facilitate illuminating the surgical site 203 with non-white illumination and facilitating acquisition of color images of the surgical site 203 by the cameras 220L, 220R (FIG. 2) of the teleoperated surgical system 200 with improved signal-to-noise as compared to images captured of the surgical site 203 with white light illumination. One example of a teleoperated surgical system 200 is the da Vinci® minimally invasive teleoperated surgical system commercialized by Intuitive Surgical, Inc. of Sunnyvale, Calif. The teleoperated surgical system 200 is merely exemplary and is not limited to the application of non-white illumination to improve the signal-to-noise ratio of images to this particular teleoperated surgical system. In light of the present disclosure, non-white illumination may be used in any surgical system that utilizes color camera(s) to improve the signal-to-noise ratio of color images captured by those color cameras.

[0044] In this example, a surgeon at a surgeon console 250 remotely operates an endoscope 201 attached to a robotic manipulator arm (not shown). Other components, cables, etc., such as those associated with a da Vinci® surgical system are present, but are not shown in FIG. 2 to avoid distracting from this disclosure. Further information regarding remotely operated minimally invasive surgical systems can be found, for example, in U.S. patent application Ser. No. 11 / 762,165 (filed Jun. 13, 2007, disclosing “Minimally Invasive Surgical System”) and U.S. Patent No. 6,331,181 (filed Dec. 18, 2001, disclosing “Surgical Robotic Tools, Data Architecture, and Use”), both of which are incorporated herein by reference.

[0045] An illumination system, such as an illumination device 210, is coupled to the endoscope 201. In one embodiment, the illumination device 210 includes a light source 211 and an illumination control device 215. The illumination control device 215 is coupled to the light source 211 and an optional variable non-white light device 218.

[0046] The illumination controller 215 includes a non-white light module 217 connected between the system processing module 262 and the light source 211. The non-white light module controls the output illumination from the illumination device 210, in one aspect, such that the illumination device 210 outputs non-white light that is used for general illumination of the surgical site 203.

[0047] In one embodiment, the light source 211 includes multiple color component illumination sources 212. In the embodiment shown in FIG. 2, the multiple color component illumination sources include P color component illumination sources, where P is a non-zero positive integer. In one embodiment, the number P is selected such that the combination of the color component illumination sources provides the broad spectrum white light of the prior art. Also, to form the non-white light, the optical power output of at least one of the multiple color component illumination sources 212 is altered to be increased or decreased compared to the condition used to generate the broad spectrum white light of the prior art.

[0048] In one embodiment, the multiple color component illumination source 212 includes multiple light emitting diodes (LEDs). The use of LEDs is merely exemplary and not limiting. The multiple color component illumination source 212 could also be implemented with multiple laser light sources instead of LEDs, for example.

[0049] In this embodiment, the illumination device 210 is used in conjunction with at least one illumination path of the stereoscopic endoscope 201 to illuminate the surgical site 203. The non-white light from the illumination device 210 is directed to the connector 216. The connector 216 provides the non-white light to the illumination path of the stereoscopic endoscope 201, which then directs the light to the surgical site 203. The connector 216 and each of the illumination paths in the stereoscopic endoscope 201 can be implemented with, for example, a fiber optic bundle, a single rigid or flexible rod, or optical fibers. In one embodiment, the endoscope 201 also includes two optical channels, i.e., stereoscopic optical paths, that pass light from the surgical site 203, such as reflected non-white light. However, the use of a stereoscopic endoscope is merely exemplary and not limiting. In light of the present disclosure, an endoscope including a single optical channel that passes light from the surgical site 203 can be used.

[0050] Non-white light from the surgical site 203 (FIG. 2) is passed by the stereoscopic optical channel of the endoscope 201 to the cameras 220L, 220R. As described more fully below, in one embodiment, the left camera 220L includes a color filter array and a left image sensor 221L. The left image sensor 221L captures light received from the left channel of the stereoscopic endoscope 201 as a left image 222L. Similarly, in this embodiment, the right camera 220R includes a color filter array and a right image sensor 221R. The right image sensor 221R captures light received from the right channel of the stereoscopic endoscope 201 as a right image 222R. Thus, the cameras 220L, 220R are color cameras using color filter arrays. However, this is merely an example and is not limiting.

[0051] Here, the cameras are configured to separate light incident on the cameras into N color components, which are captured by the cameras as N sets of pixels, each set of pixels being in a different camera color channel. Thus, each camera 220L, 220R includes multiple color channels. In one aspect, the multiple color channels are N color component channels, where N is a non-zero positive integer.

[0052] Camera 220L is coupled to a stereoscopic display 251 of the surgeon console 250 by a left camera control unit 230L and an image processing pipeline 240. Camera 220R is coupled to a stereoscopic display 251 of the surgeon console 250 by a right camera control unit 230R and an image processing pipeline 240. The camera control units 230L and 230R receive signals from a system process 262. The system process 262 represents various control devices within the system 200.

[0053] The display mode selection switch 252 provides a signal to the user interface 261, which then passes the selected display mode to the system process 262. Various controllers in the system process 262 configure the non-white light module 217 in the illumination controller 215, configure the left and right camera control units 230L and 230R to acquire the desired images, and configure other elements in the image processing pipeline 240 necessary to process the requested images so as to present the images requested by the surgeon on the display 250. The image processing pipeline 240 is equivalent to known image processing pipelines, except for the details provided herein.

[0054] Here, the capture, processing, and display of images captured by camera 220L is the same as the capture, processing, and display of images captured by camera 220R. Accordingly, in the following description, only images associated with camera 220L are discussed below. The description is directly applicable to images associated with camera 220R, and therefore the description will not be repeated for camera 220R. The description is also directly applicable to systems utilizing only a single camera, and a single image processing pipeline with an endoscope having a single optical channel.

[0055] As indicated above, typically, illumination components of three (or more) visible color components are combined to form white light, i.e., the white light includes a combination of a first visible color component, a second visible color component, and a third visible color component. Each of the three visible color components is a different visible color component, e.g., a red component, a green component, and a blue component. Illumination components of more visible colors, such as a cyan color component together with the red, green, and blue components, can also be used to generate white light.

[0056] Also, as mentioned above, in one embodiment, the light source 211 includes multiple color component illumination sources 212. In one embodiment, to generate the non-white light illumination, the non-white light module 217 varies the optical power output of at least one of the multiple color component illumination sources 212 compared to that required to form the white light. The illumination output of the (non)illumination device 210 is non-white light. The non-white light has a color tint compared to white light.

[0057] 2, the cameras 220L, 220R and the light source 212 are shown as being external to the endoscope 201. However, in one aspect, the cameras 220L, 220R and the light source 212 are included at the distal tip of the endoscope 201 adjacent the tissue 203.

[0058] There are various ways to configure the illuminator 210 so that it outputs non-white light illumination. The first method is based on white surface calibration, and the second method is based on image calibration of the surgical site. We also describe the generation of non-white light by varying the illumination time of at least one of the illumination sources of multiple color components. Each of these processes will be considered in turn.

[0059] Calibrating a White Surface for Non-White Light In this embodiment, the non-white light mode operates by having the illumination controller 215 vary the illumination intensity from the illumination source 212 of the multiple color components in the light source 211 such that all of the multiple color channels of the camera 220L receive the same amount of reflected light when viewing a white (fully reflective) surface. When imaging a white surface, there are two beneficial effects of the camera 220L responding to light evenly across the color channels. First, all color channels can fully utilize their well capacity, i.e., one channel is not limited to a percentage of its well capacity by another channel filling its well first. This increases the signal-to-noise ratio of all pixels. Second, the camera does not need to apply digital white balance gain to some color channels, which would eliminate noise amplification. Both of these effects improve the final surgical image displayed on the stereoscopic display 251.

[0060] Here, the noise characteristics of the camera 220L are interpreted as being entirely dependent on how highly filled the electron well of the image sensor 221L is. As the camera 220L collects light, the captured light is converted into electrons and stored in these electron wells. There is one electron well per pixel. As the image sensor 221L, e.g., camera 220L, collects more electrons in its electron well, the signal-to-noise ratio of the camera increases, i.e., the signal to noise ratio improves.

[0061] As shown above, if the camera 220L collects too much light and overfills an electron well, the signal for that electron well will be lost and will no longer be valid. Thus, the exposure time of the camera 220L is set to collect light to fill all of its electron wells as high as possible without overfilling any of the electron wells. If the color channel that receives the most light stops collecting light just as it is about to overflow its electron well, the result is that the other color channels do not have enough electron wells. The electron wells of the other color channels may only be 50% full. The signal-to-noise ratio of the color channels with these less-than-full electron wells is significantly less than the signal-to-noise ratio of color channels whose electron wells are about to overflow.

[0062] As explained earlier, for normal white light illumination, not only will the camera have poor signal-to-noise performance when all color channels of the camera do not have perfect or near-perfect wells, but the signal from the non-perfect wells will be amplified as part of the white balance stage in the image processing stages of prior art cameras.

[0063] In contrast to normal white light illumination, one aspect of the non-white light illumination mode operates by varying the illumination intensity of the illumination sources 212 of the multiple color components in the light source 211 by the illumination controller 215 so that all of the multiple color channels of the camera 220L receive the same amount of light when pointed at a white (fully reflective) surface. Because each of the multiple color channels of the camera 220L receives approximately the same amount of light, all of the color channels of the camera can be fully utilized in the non-white illumination mode. This is because the color channels do not reach their well capacity first. Instead, the wells of the multiple color channels reach their well capacity at approximately the same time. When photographing a white surface, the wells of the color channels will not be at 50% of their well capacity.

[0064] Also, because the non-white illumination from illuminator 210 is designed so that all three camera channel responses are equal to the light when directed at a white surface, the white balance stage in image processing pipeline 240 using white light illumination does not need to amplify any of the signals in the multiple color channels; the signals in the multiple color channels are already equal. The overall effect is that by fully utilizing the wells, there is very little noise during capture, so the resulting image is significantly less noisy and does not need to amplify any color channel with respect to the other color channels.

[0065] The characteristics of the camera 220L are considered, and the control of the light source 211 is considered, to determine how to increase or decrease the intensity of the different color component illumination sources in the multiple color component illumination source 212 so that the color channels of the camera 220L respond evenly when the camera 220L is pointed at a white surface. As mentioned above, the considerations for the camera 220R are the same as those for the camera 220L, so the description will not be repeated for the camera 220R. The following description also applies to an endoscope system that utilizes only a single signal channel and a single camera.

[0066] Additionally, in the following description, color spaces utilizing red, green, and blue color components are discussed. Typically, color cameras include a color filter array, such as a Bayer color filter array. Regardless of the color camera configuration, in the following description, there is a first set of pixels captured by the image capture sensor 221L associated with the red color channel of the camera. There is a second set of pixels captured by the image capture sensor 221L associated with the green color channel of the camera, and there is a third set of pixels captured by the image capture sensor 221L associated with the blue color channel of the camera. The use of three color channels to represent the multiple N color channels of the camera 220L is merely exemplary and not limiting. Additionally, the use of red, green, and blue color channels as three color channels is merely exemplary and not limiting. In light of this disclosure, one skilled in the art can define both the number of channels associated with the camera 220L and the specific colors associated with the number of channels based on the color space of interest and the color filter array of interest.

[0067] The optics of the camera 220L, the color filter array used in the camera 220L, and the quantum efficiency of the camera 220L (collectively referred to as the camera's sensitivity function) determine how the camera 220L responds to different wavelengths of incident light, i.e., how it responds to light from the surgical site 203. Figure 3 shows an example of a response function for the camera 220L. There is a response function for each of the multiple N color channels of the camera 220L.

[0068] Thus, in this example, there are three response functions, one for the blue channel 301, one for the green channel 302, and one for the red channel 303. In Figure 3, each of the response functions 301-303 is represented by a curved line.

[0069] A larger value of a particular response function indicates that the color channel of camera 220L is more responsive to that particular wavelength of light compared to other wavelengths of light having lower values ​​for that particular response function. For example, red response function 303 indicates that the red channel of camera 220L passes more light in the 600-650 nm wavelength range than the 450-500 nm wavelength range.

[0070] A lower value of a particular response function indicates that the color channel of camera 220L is less responsive to that particular wavelength of light compared to other wavelengths of light having higher values ​​for that particular response function. For example, blue response function 301 indicates that the blue channel of camera 220L passes less light in the 600-650 nm wavelength range than it passes light in the 440-470 nm wavelength range. A zero value of the response function indicates that the color channel of camera 220L cannot recognize that wavelength of light.

[0071] In one embodiment, the response functions of the multiple color channels of the camera 220L are converted into a matrix representation as three column vectors that make up the matrix R. The matrix R is an M×N matrix. Specifically, in one embodiment, a set of M equally spaced wavelengths, for example 400 nm to 700 nm spaced 1 nm apart, are selected, and then the response function values ​​are read at each of these selected wavelengths. In this example, 301 values ​​are generated for each response function. Thus, in this example, M is equal to 301 and N is equal to 3 for the red, green, and blue response functions. As long as the range of 400 nm to 700 nm encompasses all significant non-zero portions of the response function, and the interval spacing is small enough (1 nm), then the vector form corresponds to a complete curve. The sampling range and spacing will typically vary based on the application.

[0072] As mentioned above, in one embodiment, light source 211 includes a plurality of P color component illumination sources 212. As an example, consider an implementation where P is four, such that light source 211 includes four different color component illumination sources, e.g., four individual LEDs that can be adjusted to emit light of different intensities. An example of an illumination device including four individual LEDs is shown in U.S. Patent Application Publication No. 2012 / 0004508 (filed Aug. 13, 2010, disclosing “Surgical Illuminator With Dual Spectrum Fluorescence”), which is incorporated herein by reference.

[0073] Examples of illumination spectra for four different LEDs are shown in Figure 4. Spectrum 401 is a blue color component illumination. Spectrum 402 is a cyan color component illumination. Spectrum 403 is a green color component illumination, and spectrum 404 is a red color component illumination. In this embodiment, each of the multiple color component illumination sources 212 is a different color component illumination source.

[0074] The spectra 401-404 can also be represented in matrix notation. Each spectrum is a column of the matrix E. The matrix E is an M by P matrix. Specifically, in one embodiment, a set of M equally spaced wavelengths, for example 400 nm to 700 nm spaced at 1 nm intervals, are selected, and then the LED illumination spectrum values ​​are read at each of these M selected wavelengths.

[0075] Also, because the illumination control device 215 controls the output intensity of each LED in the light source 211, and because the illumination outputs of the LEDs are optically linearly combined together with their mixed output, the spectrum of light emitted from the tip of the endoscope 201 e out can be expressed as follows:

number

[0076] To achieve non-white light, a weight vector w is determined such that the responses from each of the N camera color channels are equally constrained such that if the LED cannot emit negative light, then the elements of the weight vector w must be positive. This can be written in matrix form as follows:

number

[0077] In other words, the above equation determines the P color channel illumination control factors CCIW (elements of weight vector w) such that when the non-white light illumination module 217 applies a different one of the P color channel illumination control factors CCIW to each of the P color component illumination sources in the light source 211, the illumination device 210 outputs non-white light, and when the non-white light emitted by the endoscope 201 is incident on a perfect reflective surface, i.e., a white surface, each electronic well of the camera 220L is filled to a 100% full level.

[0078] Thus, the problem is to find a weight vector w that contains only positive components. If the number of controllable LEDs P is equal to the number of camera color channels N, a simple solution is to find the matrix R T When *diag(t)*E is a square matrix, we take the inverse matrix. In this case, we get:

number

[0079] The weight vector w is all positive or zero (and R T A solution exists as long as *diag(t)*E has an inverse. If the weight vector w has negative components, the LEDs cannot be controlled to equalize the camera color channels. However, in this situation, the negative components of the weight vector w can be clipped to zero to get the closest solution.

[0080] However, if the number of controllable LEDs, P, is not equal to the number of camera color channels, N, then the matrix R T A simple solution is not possible because there is no inverse of *diag(t)*E. For example, for camera 220L, which has four controllable LEDs and only three color channels, the simple solution is not available.

[0081] In this particular case (i.e., whenever the number of illumination sources of controllable color components, P, is greater than the number of color channels, N), there exist many solutions for the components of the weight vector w, i.e., multiple different solutions for the weight vector w that will result in uniform filling of the camera color channel wells for non-white light incident on a perfectly reflective surface. Specifically, the set of solutions for w can be expressed as follows:

number

[0082] Possible constraints include, but are not limited to, maximizing radiant power, minimizing radiant power, maximizing drive current, minimizing drive current, or changing the color of the light viewed outside the surgeon console to appear whiter. A constraint to maximize the minimum LED brightness was implemented due to the constraints of the LED drive electronics. Essentially, what is desired is to have the minimum value of the weight vector w as high as possible, since the lighting control device 215 cannot reliably control the light at low brightness.

[0083] There is a minimax optimization to solve these constraints, but since there is only one extra degree of freedom in the solution (PN=1), the illumination from the green and cyan LEDs is always lowest for camera 220L, and the null space vector V n It was empirically noticed that the weights w have opposite signs in the green and cyan channels. The value of the weight vector w was determined by sweeping out the component α in the null space until the green and cyan channels were equal.

[0084] Table 1 is an example of MatLab computer code compiled by the MatLab® compiler that is then executed on a processor to determine a weight vector w by sweeping out the null space component α until the green and cyan lighting channels are equal. (MatLab is a U.S. registered trademark of The Mathworks, Inc., 3 Apple Hill Drive, Natick, Massachusetts, 01760, USA.) [Table 1] where svd(X) is the singular value decomposition of X, diag(x) is the diagonal matrix of x, The 2 and 3 indexes correspond to the green and cyan LEDs, respectively. Here, U,S,V are the outputs of singular value decomposition (svd), which is a common method for splitting a matrix into three separate matrices: X = U, S, V T , where U and V are orthonormal matrices and S is a diagonal matrix. This decomposition of X into U, S, and V allows us to find the pseudoinverse and the components of the null space.

[0085] 5 shows the difference between white light illumination 502 and non-white light illumination 501 using the above procedure. As can be seen, the blue and red LEDs output much more light than the green and cyan LEDs in non-white light mode compared to white light mode. In one embodiment, the response of the camera 220L to white light when reflected from a white surface is red (R): 61%, green (G): 100%, blue (B) 67%, while the camera response using non-white light is red (R): 100%, green (G): 100%, blue (B): 100% as the response was designed for.

[0086] This technique for generating non-white light demonstrates that noise reduction in the displayed surgical scene can be achieved by adjusting the color of the light from the endoscope. Specifically, if all color channels of the camera(s) capture the scene response equally when pointed at a white surface, the signal-to-noise ratio of each pixel can be increased in the displayed surgical scene. Increasing the signal-to-noise ratio reduces the perceived noise in the final surgical image.

[0087] Non-white light scene-based calibration Conventionally, the non-white light module 217 was configured to drive the light source 211 such that the illumination device 210 illuminated the surgical site 203 with non-white light, so that color balancing of the white light in the color image captured by the camera 220L was not necessary when the non-white light was reflected off a white surface. This improved the signal-to-noise ratio of the image displayed on the display 251 compared to an image captured and displayed on the display 251 using normal white light illumination.

[0088] However, non-white light can be generated in other ways to improve the signal-to-noise ratio of the image presented on the display 251. For example, the control device 215 is configured to vary the output illuminance level, e.g., optical power output, of at least one of the multiple color component illumination sources 212 such that the illumination device 210 outputs non-white light. The variation in the output illuminance level of at least one of the multiple color component illumination sources 212 is based on color component characteristics of a color scene captured by the camera 220L. The non-white light generated in this regard results in an improved signal-to-noise ratio of the scene displayed on the display 251 compared to a scene captured and displayed on the display 251 using normal white light illumination.

[0089] In one embodiment, the color channel lighting control factors CCIW are generated in various ways: First, captured scene information is used to generate the color channel lighting control factors CCIW used by the lighting controller 215.

[0090] In one aspect, a dynamic approach determines the color channel lighting control factors CCIW used to generate non-white light. For example, frame 610 (FIG. 6A) is captured and used to generate a first set of color channel lighting control factors CCIW 620. The first set of color channel lighting control factors CCIW 620 is used for the next Z consecutive frames captured, where Z is a non-zero positive integer. The Z consecutive frames are captured, processed, and displayed, with one of the frames 621 in the sequence being used to generate a second set of color channel lighting control factors CCIW 621. The second set of color channel lighting control factors CCIW 621 is used for the next Z consecutive frames captured, and the processing continues.

[0091] In one aspect, a time-weighted moving average is used to generate the color channel lighting control factor CCIW 621. In this aspect, a frame channel lighting control factor CCIW_frame is generated for each frame and, as described above, weighted in a decreasing manner over the next fixed number of frames and summed with other weighted frame channel lighting control factors CCIW_frame, averaged such that the frame channel lighting control factor CCIW_frame of the most recent frame dominates, but a decreasing (e.g., linear) fraction of the frame channel lighting control factor CCIW_frame of the previous frame is added to provide a currently applied weighted time-moving average channel lighting control factor CCIW 621 that is updated, for example, at the frame rate.

[0092] In one embodiment, the number Z is empirically determined as a time average of a number of frames to maintain the stability of the system 200. In another embodiment, the number Z is not constant. Rather, the average luminance of the color channels is monitored, and if the average luminance of any color channel changes by more than a predetermined percentage, e.g., 5%, a new set of color channel illumination control factors CCIW is generated. This approach adaptively compensates for changes in the fullness of the wells of the camera's color channels during the surgical procedure and adjusts the color channel illumination control factors CCIW so that the electronic wells remain near optimal filling when the characteristics of the surgical site within the camera's field of view change.

[0093] In another embodiment, a histogram of pixel brightness for each color channel of the camera 220L is created for the captured scene. A histogram of pixel brightness for each of the multiple N color channels of the camera 220L is created. As known to those skilled in the art, each pixel has a pixel value, which is a single number that represents the brightness of the pixel. The pixel value is an indication of the fill of the pixel well for that pixel. Thus, N brightness histograms are created for the camera 220L, one for each color channel.

[0094] In each of the N intensity histograms, the intensity value is plotted on the x-axis. The height of the bar for each intensity value represents the number of pixels in the color channel that have that intensity value. The intensity histograms can be based on the entire captured image or on a region of interest in the captured image. For example, the region of interest can be defined as the area within the fovea of ​​the person using the teleoperated surgical system 200. An indication of the fill of the pixel wells of the image sensor 221L where that color channel is derived from the intensity histograms of each color channel. The indication can be the average value or a value that corresponds to the 90th percentile of all values.

[0095] Matrix B is defined as the coupling between the lighting control and the color channel response of the camera sensor. This matrix transforms the lighting control, a P element vector, into the color channel response of the camera sensor, an N element vector. Matrix B can be measured at the reference level by rotating over the lighting channels one by one. Thus, we have:

number

[0096] The color channel illumination control factor CCIW is defined as follows:

number

[0097] Determining the color channel illumination control factors CCIW is straightforward if P is equal to N and the inverse of B exists. If P is greater than N, the pseudo-inverse of B is used.

[0098] Pseudoinverses of matrices are well known to those skilled in the art. A pseudoinverse suitable for use herein is called the Moore-Penrose pseudoinverse. A common use of the Moore-Penrose pseudoinverse is to calculate an "optimal" least squares solution to a system of linear equations that lacks a unique solution. Another use of the Moore-Penrose pseudoinverse is to find a minimum (Euclidean) norm solution to a system of linear equations. In one embodiment, the optimal least squares solution is used.

[0099] The color channel illumination control factors CCIW thus generated may be used in either a static or dynamic approach, as described above, in the teleoperated surgical system 200. Regardless of the technique used to generate the color channel illumination control factors CCIW, the use of non-white light illumination improves the image quality displayed on the display 251 by reducing the noise contribution to the displayed image as compared to a displayed image formed from an image captured using white light illumination.

[0100] As an example, assume that the camera 220L has red, green and blue color channels, e.g., N is 3, and the desired fill WFull is 75% for each color channel. With white light illumination, the red channel R603B is 90% filled (FIG. 6B), the green channel G602B is 50%, and the blue channel B601B is 37.5%. For non-white illumination, the red channel R603C is about 75% filled (FIG. 6C), the green channel G602C is 75%, and the blue channel B601C is about 75% filled. The noise floor for both white and non-white illumination was set to 15%. Thus, the signal-to-noise ratio of the blue and green components is improved for non-white light.

[0101] However, the electronic well may not be exactly 75% filled because illumination from each of the multiple color component illumination sources 212 in light source 211 is absorbed and reflected in different directions by the surgical site 203. Thus, in this embodiment, when the responses of the camera channels to non-white light are approximately equal, it means that the responses of the portions of the camera's image sensor in the different color channels are equal when differences in absorption and reflection of non-white light by the surgical site are taken into account. Nevertheless, the signal-to-noise ratio has been improved, at least for the blue and green pixels.

[0102] In a color image of the surgical site 203, the green and blue pixels provide most of the detail because the green and blue illumination components are less scattered and less penetrating than the red illumination component. As a result, improving the signal-to-noise ratio of the blue and green pixels improves the image quality provided to the display 251.

[0103] If a pixel having a filled well as shown in FIG. 6C were directly processed for display on the display 251, the displayed image would not be properly color balanced due to the amount of electronic well filling of the color channel associated with the non-white light illumination. Thus, as each color channel pixel is acquired from the image sensor, the color channel pixel value is corrected to compensate for the color channel illumination control factor CCIW. Assuming that illumination control L0 produces a white image when viewing a scene with a neutral color object (e.g., a white balance target),

number

[0104] Later, additional gain A=[ccw1,...,ccwP] T=CCW T is applied to the lighting control on top of L0, resulting in L, where L=[L1,...,LP] T =[ccw1*L01,...,ccwP*L0P] T The color response from a camera with illumination L is I=[r,g,b] T It is. K=B*A=[kr,kg,kb]

[0105] The adjusted pixel color I'=[r / kr, g / kg, b / kb] is presented on the display to achieve accurate color. This results in a typical pixel value as shown in FIG. 6D. The noise in the red channel has increased slightly, but the noise in the green and blue channels has disappeared. Thus, when the signals in the green and blue channels are amplified in the white balance adjustment process of the image processing pipeline 240, the signal-to-noise ratio is better than that with white light illumination.

[0106] High dynamic range images using non-white light The standard method for forming a high dynamic range image using a video camera is to have the video camera capture successive images at different exposure levels. The differently exposed images are then merged into a single high resolution image. This technique requires the video camera to have the ability to switch exposure settings on a frame-by-frame basis. This effectively results in a reduced frame rate compared to a video camera that generates a display image from each captured frame. Also, if there is motion in the scene, motion artifacts can be observed when images captured over time are merged into a single high resolution image.

[0107] Another approach used in imaging to create high dynamic range images is to capture images with graduated neutral density filters. The neutral density filters are stepped so that bright areas of the scene are attenuated by the filter more than dim areas of the scene. This works well for scenes with known areas of different brightness, such as sunrise and sunset scenes, but the use of graduated neutral density filters in a teleoperated surgical system is not practical because the same part of the scene at the surgical site is not always the brightest area of ​​the scene.

[0108] In applications that do not require color images, monochromatic high dynamic range images can be generated using camera 220L and non-white light illuminating surgical site 203. For example, unlike diagnosis, color images may not be important during navigation of the surgical instrument(s), such as during guidance of lung navigation.

[0109] In applications where monochrome images are acceptable, high dynamic range images can be formed using non-white light illumination and a camera without different exposure settings or neutral density filters. The non-white light module 217 is configured to drive the multiple color component illumination sources 212 such that the multiple color component illumination sources 212 have intensities other than the normal intensities used to form the white light illumination.

[0110] Thus, camera 220L captures N images, each of the camera's N color channels being effectively exposed differently due to the different weights used for the color component illumination sources 212. These N images are used to generate a high dynamic range image in a manner equivalent to that used for a camera with neutral density filters. Thus, a high dynamic range image is obtained without the need for any special filters or cameras with variable exposure settings. The use of non-white light illumination allows the generation of high dynamic range images using conventional cameras with color filter arrays used in teleoperated surgical systems.

[0111] For purposes of example, it is assumed that the number P of color component illumination sources in the multiple color component illumination sources 212 is three, and the three color component illumination sources are a red light source, a green light source, and a blue light source. It is also assumed that for normal white light illumination, the non-white light module 217 weights each of the three light sources equally, e.g., the weights (color channel illumination control factors CCIW) of red, green, and blue are 1:1:1. In one aspect, the weights (color channel illumination control factors CCIW) are changed such that the illumination device 210 provides non-white light, e.g., red, green, and blue light sources, with weights of 0.5:1.0:2.0.

[0112] In general, the weights of the color components are determined by taking the ratio of the dynamic range of the pixels of the color channels of the camera 220L (pixel dynamic range) to the dynamic range of the reflected light from the surgical scene (scene dynamic range). The ratio of pixel dynamic range to scene dynamic range is 1:DR, where DR is a non-zero positive integer. In this example, the maximum weight of the first illumination color component is DR times the weight of the Nth illumination color component. The weights of the second through (N-1) illumination color components are uniformly spaced between the minimum and maximum weights by powers of 2. For example, if the pixel dynamic range is 1:256 and the scene dynamic range is 1:1024, then the ratio between the two is 1:4 (2 2 ) In the above example, the minimum weight was 1, so the maximum weight was 4. The powers of 2 between 1 and 4 are 2 1 = 2, so the third weight is 2.

[0113] As another example, consider a 1:16 ratio and a three color component illumination source. The minimum weight is 1 and the maximum weight is 16. The other weight is 4. If the ratio is 1:16 and there is a four color component illumination source, the weights are 1:2. (4 / 3) :2 (8 / 3) It will be :16.

[0114] When a scene is captured from the reflected non-white light at camera 220L, each of the N color channels captures a grayscale scene with a different reflected light intensity due to the difference in the optical power output of the illumination source of the P color components at light source 212. The N grayscale scenes are processed to generate a high dynamic range grayscale image. One technique for performing this processing is described in Nayar and Mitunga, “High Dynamic Range Imaging: Spatially Varying Pixel Exposures,” IEEE Conference on Computer Vision and Pattern Recognition, Vol.1, pp.472-479(2000), which is incorporated herein by reference.

[0115] Rather than using images obtained with different camera exposure settings and neutral density filters, we use a fixed-exposure camera to acquire different exposure images using non-white light illumination that contains a combination of color lighting components of different intensities. Since the N different exposure images are captured simultaneously, there are no motion artifacts.

[0116] In the previous examples, different intensities of color components of non-white light were used to capture differently exposed images in a single frame. A similar effect can be obtained by varying the length of time that each color component illumination source is output within the exposure time of a frame. In this embodiment, the weights applied by the non-white light module 217 are the same as those used for white light, but a switching element(s) is added to the non-white light module 217.

[0117] Consider the same example where the ratio of optical power output of the red, green, and blue components is 0.5:1:2, and the exposure time is time t. In this example, the blue light source is output for time (2 / 3.5)*t. The green light source is output for time (1 / 3.5)*t, and the red light source is output for time (0.5 / 3.5)*t. Thus, each of the color component light sources is pulse width modulated for a particular percentage of the exposure time.

[0118] 7 illustrates an example of varying the output from the red, blue, and green component illumination sources of the illumination device 210 to control the output light from the illumination device 210. The particular order in which the illumination sources of each color component are turned on and off is not important, so long as the blue light source is on for 58% of the exposure time, the green light source is on for 28% of the exposure time, and the red light source is on for 14% of the exposure time. Of course, instead of switching the light sources on and off, the light sources can be kept on and their output can be directed away from or prevented from reaching the output of the illumination device 210 during the exposure time, as shown in the off state in FIG.

[0119] The particular number of light component illumination sources and light component illumination source weights used in the above examples are merely illustrative and are not intended to be limiting to the particular number of light component illumination sources and weights used in the embodiments.

[0120] In another embodiment, the output of the color component illumination sources is varied as shown in Figure 7, but different frames are captured for each color component illumination source. If a fixed frame shutter is used, the illumination switching is synchronized to the image acquisition. If a rolling shutter is used, the illumination switching must be at a frequency such that switching the illumination does not introduce flicker into the captured image. As used herein, rolling shutter means that information is read out from each row of the frame in sequence, e.g., from top to bottom, rather than capturing the entire frame at once.

[0121] Thus, in this example, the exposure time of the camera is fixed, but the illumination is varied to obtain different exposure images, in which case a high dynamic range image is formed in a manner that corresponds to known techniques for obtaining high dynamic range images for images taken at different exposures.

[0122] Varying the output capabilities of the lighting device 210 can be accomplished in many ways. The output of the multiple component light source can be directly controlled as described above. The element 218 can be a rotating wheel having multiple sections disposed in the path of the light output from the lighting device 210. Each of the multiple sections has one of the multiple color components and each of the multiple sections has a different light attenuation level. In another embodiment, the element 218 is an acousto-optic light modulator coupled to a controller.

[0123] Alternatively, element 218 can be a liquid crystal device. In one embodiment, the liquid crystal device is configured in an on / off pulse width modulated shutter mode, which has a varying ratio of on / off times including one or more on / off cycles per capture of a camera image frame. In another embodiment, the liquid crystal device is configured as an adjustable attenuator, such as an entrance polarizer, a compensated liquid crystal variable retarder, and an exit polarizer, where the entrance and exit polarizers are crossed with a linear polarizer. In yet another embodiment, the liquid crystal device is configured as a tunable filter. The use and operation of liquid crystal tunable filters are well known to those skilled in the art.

[0124] Thus, in some embodiments, a fixed filter is used to modify the illuminance level of at least one of the illumination sources of the multiple color components such that the lighting device outputs non-white light.

[0125] Also, in some aspects, the variation in illuminance levels producing non-white light is controlled to accommodate uneven power losses induced by life-time aging of the multiple color component illumination sources 212. Thus, in cases where the optical power output of light emitting diodes or laser diodes, etc., decreases over time due to aging-induced power losses, the color channel illumination control factor CCIW can be statically or dynamically adjusted to compensate for uneven power losses induced by aging of each of the multiple color component illumination sources 212.

[0126] The various modules described herein may be implemented by software running on a processor, hardware, firmware, or any combination of the three. When a module is implemented as software running on a processor, the software is stored in memory as computer readable instructions, and the computer readable instructions are executed on the processor. All or a portion of the memory may be in a different physical location than the processor, so long as the processor is coupled to the memory. Memory may refer to volatile memory, non-volatile memory, or any combination of the two.

[0127] Also, as described herein, the functionality of the various modules may be performed by one unit or may be divided among different components or modules, each function may in turn be implemented by any combination of hardware, software running on a processor, and firmware. When divided among different components or modules, the components or modules may be centralized in one location or distributed throughout the system 200 for distributed processing. Execution of the various modules results in methods that perform the processing described above for the various modules and the controller 260.

[0128] Thus, the processor is coupled to a memory containing instructions that are executed by the processor. This can be accomplished within a computer system or through a connection to another computer via a modem and analog lines or a digital interface and digital carrier lines.

[0129] Here, a computer program product includes a computer readable medium configured to store or on which computer readable code for some or all of the processes described herein is stored. Some examples of computer program products are CD-ROM disks, DVD disks, flash memory, ROM cards, floppy disks, magnetic tapes, computer hard drives, servers on a network, and signals transmitted over a network representing computer readable program code. A non-transitory tangible computer program product includes a tangible computer readable medium configured to store or on which computer readable instructions for some or all of the processes are stored. A non-transitory tangible computer program product is a CD-ROM disk, DVD disk, flash memory, ROM cards, floppy disks, magnetic tapes, computer hard drives, and other physical storage media.

[0130] In view of this disclosure, the instructions used in some or all of the processes described herein may be implemented in a wide variety of computer system configurations using operating systems and computer programming languages ​​of interest to the user.

[0131] The above detailed description and the accompanying drawings illustrating aspects and embodiments of the present invention should not be construed as limiting, and the claims define the protected invention. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of the present specification and claims. In some instances, well-known circuits, structures, and techniques have not been shown or described in detail to avoid obscuring the present invention.

[0132] Additionally, the terminology in this detailed description is not intended to be limiting of the invention. For example, spatial terms such as "beneath," "below," "lower," "above," "upper," "proximal," "distal," and the like are used to describe the relationship of one element or feature to another element or feature shown in the figures. These spatial terms are intended to encompass different positions (i.e., locations) and orientations (i.e., rotational positions) of the device during use or operation in addition to the positions and orientations shown in the figures.

[0133] For example, if a device in the figures is turned over, elements described as "below" or "beneath" another element or feature would then be "above" or "over" the other element or feature. Thus, the exemplary term "below" can encompass both "above" and "below" positions and orientations. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatial descriptions used herein interpreted accordingly.

[0134] Similarly, descriptions of movement along and about various axes include various particular positions and orientations of the device. The singular forms "a," "an," and "the" are intended to include the plural forms unless the context dictates otherwise. The terms "comprises," "comprising," "including," and the like specify the presence of described features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups.

[0135] Components described as "coupled" may be directly coupled electrically or mechanically, or indirectly coupled through one or more intermediate components. In view of this disclosure, the instructions used in any or any combination of operations described with respect to the improved display system may be implemented in a wide variety of computer system configurations using operating systems and computer programming languages ​​of interest to the user.

[0136] All examples and references to the description are non-limiting and should not be used to limit the scope of the claims to the specific implementations or embodiments described herein and their equivalents. Headings are for format only and should not be used to limit the subject matter in any manner, as the text under one heading may cross-reference, i.e., apply to the text under more than one heading. Finally, in light of this disclosure, a particular feature described in connection with one aspect or embodiment may be applicable to other disclosed aspects or embodiments of the invention, even though not specifically shown in the drawings or described in the text.

[0137] The contents of the claims as originally filed are described below as examples. [Example 1] A device, the device comprising: a camera configured to separate light entering the camera into sets of pixels, each set of pixels being in a different color channel of the camera; an illumination device configured to output non-white light such that each color channel of the camera has a substantially equal response to the non-white light reflected from a perfectly reflective surface; device. [Example 2] The device of example 1, wherein the lighting device has illumination sources of multiple color components, the multiple color components being configured such that the lighting device outputs the non-white light. [Example 3] The device of Example 2, further comprising a control device coupled to the multiple color component illumination sources, the control device configured to weight the output of each of the multiple color component illumination sources such that a combination of the outputs of the multiple color component illumination sources is non-white. [Example 4] A device, the device comprising: Camera and A lighting device; a control device coupled to the lighting device, the camera is configured to separate light incident on the camera into color components, the color components being captured by the camera as sets of pixels, each set of pixels being in a separate camera color channel; the control device is configured to adjust characteristics of the light output by the illumination device to increase a signal-to-noise ratio of pixels of one camera color channel for a color image captured by the camera; device. [Example 5] The apparatus of example 4, wherein the illumination device has illumination sources of multiple color components. [Example 6] The apparatus of Example 5, wherein the control device is coupled to the illumination sources of the multiple color components, and the control device is configured to adjust characteristics of at least one of the illumination sources of the multiple color components to increase a signal-to-noise ratio of the pixels of the one camera color channel for the color image captured by the camera. [Example 7] The device of Example 5, wherein the illumination sources of the multiple color components are multiple light emitting diodes. [Example 8] 6. The apparatus of claim 5, wherein the illumination sources of the multiple color components are multiple laser diodes. [Example 9] The apparatus of Example 4, wherein the control device is configured to control the output of multiple color component illumination sources of the illumination device such that the illumination device outputs non-white light, whereby each camera color channel has a substantially equal response to the non-white light reflected from a perfectly reflective surface. [Example 10] The device of Example 4, wherein the control device is configured to change the illuminance level of at least one of the illumination sources of multiple color components of the lighting device so that the lighting device outputs non-white light. [Example 11] 5. The apparatus of example 4, wherein the control device is further configured to form a high dynamic range image from a monochromatic image captured by the camera. [Example 12] The device of Example 11, wherein the control device is configured to change the illuminance level of at least one of the illumination sources of multiple color components of the lighting device. [Example 13] The device of example 12, further comprising a rotating wheel coupled to the control device and having a plurality of sections, each of the plurality of sections having one color of the plurality of color components, and each of the plurality of sections having a different level of light attenuation. [Example 14] 13. The apparatus of claim 12, further comprising a liquid crystal device coupled to the control device. [Example 15] The apparatus of Example 14, wherein the liquid crystal device is set to an on / off pulse width modulated shutter mode, the variation ratio of the on / off times including one or more on / off cycles per image frame capture of the camera. [Example 16] 15. The apparatus of claim 14, wherein the liquid crystal device is configured as an adjustable attenuator. [Example 17] 15. The apparatus of claim 14, wherein the liquid crystal device is configured as a tunable filter. [Example 18] 13. The apparatus of example 12, further comprising an acousto-optical light modulator coupled to the controller. [Example 19] 1. A method, comprising: illuminating the scene with a non-white light configured such that each camera color channel of the camera has a substantially equal response to light reflected from a perfectly reflective surface; capturing an image of the scene with the camera; and outputting an image for display based on the captured image without adjusting the white color balance of the captured image. method. [Example 20] 1. A method, comprising: capturing a color image comprising sets of pixels, each set of pixels being in a different color channel of a camera; and constructing a high dynamic range image from the set of pixels. method. [Example 21] 1. A method, comprising: illuminating the site with a non-white light configured such that each camera color channel of a camera has a substantially equal response to the non-white light reflected from the site; capturing an image of the site using the camera; and outputting an image for display based on the captured image. method.

Claims

1. 1. A system comprising: one or more processors; a memory for storing executable instructions; The instructions, when executed by the one or more processors, cause the device to: configuring an illumination device to illuminate a scene with non-white light, the illumination device including multiple color component illumination sources, the non-white light including light other than white light generated by a combination of light output by the multiple color component illumination sources; controlling a camera to capture frames of the scene illuminated with the non-white light in multiple color channels of the camera; and adjusting pixel values ​​of color channels of the camera in the frames of the scene to reduce noise in the color channels in the frames of the scene; configuring the illumination device to illuminate the scene with the non-white light is based on signal levels of the color channels of the camera in a previously captured frame of the scene. system.

2. Configuring the illumination device to illuminate the scene with the non-white light comprises weighting illuminance levels output by one or more of the color component illumination sources such that the combination of the light output by the color component illumination sources is the non-white light; The system of claim 1 , wherein adjusting the color channels of the camera in the frame of the scene is based on a weighting of the output illuminance levels of an illumination source of the color components corresponding to the color channels of the camera.

3. a video stream captured by the camera comprising a plurality of frames of the scene, the plurality of frames comprising a first set of frames and a second set of frames captured subsequent to the first set of frames; the second set of frames includes the frames of the scene, and the first set of frames includes the previously captured frames of the scene; Configuring the illumination device to illuminate the scene with the non-white light includes: determining a first set of color channel lighting control factors based on color component characteristics of light captured by the camera in one or more frames of the first set of frames, the first set of color channel lighting control factors including a color channel lighting control factor for each color component illumination source of the plurality of color component illumination sources; and 2. The system of claim 1, further comprising: weighting output illuminance levels of illumination sources of the plurality of color components for each frame in the second set of frames based on the first set of color channel illumination control factors.

4. 4. The system of claim 3, wherein the first set of color channel lighting control factors is determined based on signal levels of the color channels of the previously captured frames of the scene or a time-weighted moving average of color channel lighting control factors for each frame within the first set of frames.

5. The instructions, when executed by the one or more processors, cause the device to: monitoring the average luminance of the color channels of the camera; and 4. The system of claim 3, further comprising: determining said first set of color channel lighting control factors in response to detecting a predetermined change in said average luminance of said color channels.

6. The system of claim 3 , wherein adjusting the color channels of the camera in the frame of the scene to reduce the noise in the color channels is based on the first set of color channel lighting control factors.

7. The system of claim 1 , wherein the color channels of the camera include a blue channel or a green channel.

8. 1. A method, comprising: a computing device configuring an illumination device to illuminate a scene with non-white light, the illumination device including multiple color component illumination sources, the non-white light including light other than white light generated by a combination of light output by the multiple color component illumination sources; the computing device controlling a camera to capture frames of the scene illuminated with the non-white light in multiple color channels of the camera; and adjusting pixel values ​​of the color channels of the camera in the frames of the scene to reduce noise in the color channels in the frames of the scene, configuring the illumination device to illuminate the scene with the non-white light based on signal levels of the color channels of the camera in a previously captured frame of the scene; method.

9. configuring the illumination device to illuminate the scene with the non-white light comprises weighting illuminance levels output by one or more of the color component illumination sources such that the combination of the light output by the color component illumination sources is the non-white light; 9. The method of claim 8, wherein adjusting the color channels of the camera in the frame of the scene is based on a weighting of the output illuminance levels of an illumination source of the color components corresponding to the color channels of the camera.

10. a video stream captured by the camera comprising a plurality of frames of the scene, the plurality of frames comprising a first set of frames and a second set of frames captured subsequent to the first set of frames; the second set of frames includes frames of the scene, and the first set of frames includes the previously captured frames of the scene; Configuring the illumination device to illuminate the scene with the non-white light comprises: determining a first set of color channel lighting control factors based on signal levels of the plurality of color channels for the first set of frames in the video stream, the first set of color channel lighting control factors including a color channel lighting control factor for each color component illumination source of the plurality of color component illumination sources; and weighting output illuminance levels of illumination sources of the plurality of color components for each frame in the second set of frames based on the first set of color channel illumination control factors.

11. 11. The method of claim 10, wherein the first set of color channel lighting control factors is determined based on signal levels of the color channels of the previously captured frames of the scene or a time-weighted moving average of color channel lighting control factors for each frame of the first set of frames.

12. said computing device monitoring the average luminance of the color channels of said camera; 11. The method of claim 10, further comprising: in response to detecting a predetermined change in the average luminance of the color channels, the computing device determining the first set of color channel lighting control factors.

13. The method of claim 10 , wherein adjusting the pixel values ​​of the color channels of the camera in the frame of the scene is based on the first set of color channel lighting control factors.

Images