System for automatic exposure and lighting control to maintain a predetermined target brightness of a displayed scene - Patents.com

Abstract

To provide a system for auto-exposure and illumination control for maintaining a predetermined target brightness of a displayed scene.SOLUTION: Scenes 251 captured with an endoscope 201 of a teleoperated surgical system 200 and displayed on a display unit maintain a consistent brightness even though optical output power of an illuminator 210 changes, and working distance 204 between tissue 203 and the distal tip of the endoscope changes. The teleoperated surgical system 200 automatically detects when endoscope 201 contacts tissue 203 and adjusts the output optical power of the illuminator 210 so that tissue damage does not occur.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] Related Applications This application is related to U.S. Patent Application No. 61 / 954,336, filed March 17, 2014, and to a national phase application of International Application No. PCT / US2015 / 020892, which claims the benefit of U.S. Patent Application No. 61 / 954,381, filed March 17, 2014, each of which is incorporated herein by reference.

[0002] Aspects of the invention relate to endoscopic imaging, and more particularly to tissue contact detection, brightness control of a displayed scene, automatic exposure control of a camera, and lighting control in a teleoperated surgical system. [Background technology]

[0003] The da Vinci® Surgical System, commercialized by Intuitive Surgical, Inc. of Sunnyvale, Calif., is a minimally invasive teleoperated surgical system that offers many benefits to patients, including reduced trauma to the body, faster recovery, and shorter hospital stays. One feature of the Da Vinci® Surgical System is its ability to provide two-channel (i.e., left and right) video capture and display visualization images to provide stereoscopic vision to the surgeon. Such electronic stereoscopic imaging systems can output high resolution video images to the surgeon and enable features such as zooming to provide a "close-up" view that allows the surgeon to work with increased precision as well as identify specific tissue types and characteristics.

[0004] However, the illumination of the surgical site and the camera exposure time used during capture of the image of the surgical site are some of the factors that affect the image quality provided to the surgeon. For example, US Pat. No. 6,399,433 describes directing an endoscope into the body to provide a first level of illumination and directing an illumination device into the body to provide a second level of illumination. The second level of illumination is larger than the first level of illumination and is used to image a larger target area. A lower level of illumination from the endoscope is used to image a smaller target area.

[0005] However, potential problems with high intensity light sources have been recognized and a solution provided in U.S. Patent No. 5,399,633. According to this patent, the output from a high intensity light source is controlled to automatically reduce the output intensity of the light source to a safe level whenever the output is not directed at tissue. Light reflected from the tissue is monitored, and if the reflected light indicates that the light source is not directed at tissue, the light intensity is reduced to a safe level. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Patent No. 8,512,232 [Patent Document 2] U.S. Patent No. 6,511,422 Summary of the Invention [Problem to be solved by the invention]

[0007] In a teleoperated surgical system, when the controller detects contact between the tip of the endoscope and tissue, the controller attenuates the optical power output from the tip of the endoscope to prevent the tissue from being damaged by the contact.

[0008] In one aspect, the controller detects contact by monitoring the intensity reflected from the tissue and by monitoring the optical power output from the endoscope, and the controller determines that contact has occurred when a change from a first optical power output to a second optical power output does not result in a change in the reflected intensity for the second optical power output compared to the reflected intensity for the first optical power output.

[0009] The controller varies the output optical power from the tip of the endoscope in a known pattern that is then attenuated. The controller detects whether the reflected intensity changes with the varying output optical power. When the reflected intensity follows the known pattern of the varying output optical power, the controller terminates the attenuation.

[0010] The teleoperated surgical system includes an illuminator, a camera, and a controller. The illuminator provides an output optical power. The camera is configured to capture a scene including tissue. The controller is coupled to the illuminator and to the camera. The controller is configured to detect contact between the endoscope and the tissue. The controller is configured to attenuate the output optical power after detecting the contact.

[0011] The controller includes a statistics module coupled to the camera. The statistics module receives a scene captured by the camera. The statistics module creates a brightness histogram of the captured scene and determines an overall brightness of the captured scene. A contact detection module of the controller is coupled to the statistics module. The contact detection module receives the overall brightness of the captured scene and receives the camera exposure time. The contact detection module is also coupled to the illumination device. The contact detection module detects contact between the tip of the endoscope and tissue.

[0012] In one aspect, the illumination device includes a dither module coupled to the contact detection module, the dither module configured to vary an output optical power in a known pattern after being enabled by the contact detection module, and an auto-exposure module coupled to the statistics module, the auto-exposure module configured to detect luminance changes in a captured scene.

[0013] The teleoperated surgical system includes a control system. The control system includes a camera control unit and a lighting controller. The lighting controller is configured to control an optical power output from an illumination device of the teleoperated surgical system. The camera control unit is configured to receive a video stream from a camera of the teleoperated surgical system. The camera control unit is coupled to the lighting controller. The camera control unit is also configured to instruct the lighting controller to change the optical power output from a first optical power output to a second optical power output, and to instruct the camera to change a camera exposure time from the first exposure time to a second exposure time, whereby a frame is subsequently captured with light reflected from the second optical power output at the second exposure time.

[0014] In one aspect, a light source is coupled to the illumination control device, an endoscope is coupled to the light source, and a camera is coupled to the endoscope and to the camera control unit.

[0015] In one aspect, the camera control unit includes a statistics module coupled to the camera for receiving the video stream. The statistics module is configured to create a luminance histogram for frames of the video stream. The camera control unit also includes an auto-exposure module coupled to the statistics module. The auto-exposure module is configured to maintain a target luminance of a displayed scene, the displayed scene being a scene from the captured frames. The auto-exposure module is also configured to limit saturated pixels of a scene in the captured frames to less than a respective predetermined number of pixels of the scene. The auto-exposure module is also configured to maintain a minimum optical power output from the illumination device.

[0016] In another aspect, the camera control unit includes first and second control loops, the first control loop configured to automatically adjust one or both of a video pipeline gain and a camera exposure time such that a displayed scene has a target brightness.

[0017] The second control loop is configured to automatically adjust the output optical power and adjust the camera exposure time of subsequently captured frames. The second control loop is configured to adjust the output optical power and the camera exposure time based on the value of the camera exposure time.

[0018] The second control loop increases the output optical power and decreases the camera exposure time when the camera exposure time is greater than a first exposure threshold. The second control loop decreases the output optical power and increases the camera exposure time when the camera exposure time is less than a second exposure threshold. The second control loop leaves the output optical power and the camera exposure time unchanged when the camera exposure time is between the first exposure threshold and the second exposure threshold.

[0019] In one aspect, the first control loop and the second control loop are included in an auto-exposure module. The statistics module is coupled to the camera for receiving the video stream and is coupled to the auto-exposure module. The statistics module is configured to create a luminance histogram for frames of the video stream.

[0020] In one aspect, the camera control unit includes a control loop configured to automatically adjust a target luminance of a displayed scene to reduce the number of saturated pixels in subsequently captured frames, the control loop being configured to apply a reduction limit to the target luminance.

[0021] In yet another aspect, the camera control unit includes an auto-exposure module coupled to the statistics module to receive a luminance histogram of frames of the video stream. The auto-exposure module is also configured to determine an average luminance using information of the histogram. The auto-exposure module is also configured to adjust one or both of a video pipeline gain and a camera exposure time based on a relationship of the average luminance to the target luminance. The auto-exposure module reduces the target luminance when a number of saturated pixels in the captured scene is greater than a saturated pixel threshold. The auto-exposure module is also configured to instruct the lighting control device to change the first output of the lighting device to the second output of the lighting device and to instruct the camera to adjust the camera exposure to compensate for variations in the output of the lighting device.

[0022] A method of operating a teleoperated surgical system includes adjusting one or both of a video pipeline gain and a camera exposure time using an average luminance of a captured scene and a target luminance of a displayed scene. The method also uses the camera exposure time to set an output optical power of an illuminator and an exposure time of the camera. The method reduces the number of saturated pixels in a second captured frame by reducing the target luminance. [Brief description of the drawings]

[0023] [Figure 1A] FIG. 1 is a block diagram of a teleoperated surgical system including controllers for controlling some or all of the saturated pixel count of a scene, for controlling the gain of a video pipeline, for controlling output optical power, for controlling camera exposure time, and for detecting tissue contact. [Figure 1B] FIG. 1 is a block diagram of a teleoperated surgical system including controllers for controlling some or all of the saturated pixel count of a scene, for controlling the gain of a video pipeline, for controlling output optical power, for controlling camera exposure time, and for detecting tissue contact. [Diagram 2] FIG. 13 is another block diagram of a teleoperated surgical system including controllers for controlling the number of saturated pixels in a scene, for controlling the gain of the video pipeline, for controlling the output optical power, for controlling the camera exposure time, and for detecting tissue contact. [Diagram 3] 13 is a luminance histogram of the captured scene produced by the statistics module. [Figure 4] FIG. 13 is a process flow diagram of one embodiment of an auto exposure module. [Figure 5A] FIG. 5 is a process flow diagram of one embodiment of the saturated pixel limiting process of FIG. 4. [Figure 5B] FIG. 5 is a process flow diagram of one embodiment of the gain and exposure time adjustment process of FIG. [Figure 5C] FIG. 5 is a process flow diagram of one embodiment of the power and exposure time adjustment process of FIG. 4. [Figure 6] FIG. 13 illustrates the control of output optical power using camera exposure time. [Figure 7] 5 is a pipeline diagram of the camera control unit of FIG. 2 and a process flow diagram of FIG. 4. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] In the drawings, the first digit of a reference number identifies the figure in which the element containing that reference number first appears.

[0025] In one aspect, scenes captured by endoscope 101 of teleoperated surgical systems 100A, 100B (FIGS. 1A and 1B) and displayed on stereoscopic display unit 151 maintain consistent brightness even when the working distance between tissue 103 and the distal tip of endoscope 101 changes. Teleoperated surgical systems 100A, 100B automatically detect when endoscope 101 contacts tissue and adjust the output optical power of illumination device 110 so that tissue damage does not occur.

[0026] Illumination having optical power output from illumination device 110, e.g., visible light, exits endoscope 101 at the distal tip of endoscope 101, sometimes referred to as the tip. The illumination is reflected from tissue 103 and captured as a frame 122L containing a left color scene captured by a left image sensor 121L of left camera 120L, and as a frame 122R containing a right color scene captured by a right image sensor 121R of right camera 120R. Each of the left and right image sensors 121L, 121R can be said to capture a frame containing a scene.

[0027] The captured frames are passed as a frame-by-frame video stream to camera control units 130L, 130R, which process each frame and then pass it to video pipeline 140, which processes each frame in the same manner as a video pipeline in a conventional telesurgery system, except as described below, and then passes the frame to a stereoscopic display 151 on the surgeon console 150.

[0028] Thus, the tissue 103 is viewed at the surgeon console 150 by a user of the teleoperated surgical system 100A, 100B. The scene is displayed to the user as a succession of images of the tissue 103 while a video sequence of the scene is displayed on the stereoscopic display 151, allowing the user to, for example, see the movement of the tissue 103, see bleeding, see breathing motion, etc. However, the user may be distracted by changes in brightness of the scene displayed on the stereoscopic display 151. The teleoperated surgical system 100 maintains a consistent brightness of the displayed scene even if the working distance between the tissue 103 and the distal tip of the endoscope 101 changes, either increasing or decreasing. Although this example uses a stereoscopic endoscope, the systems 100A, 100B function in the same way as an endoscope with a single optical channel by processing a single captured scene.

[0029] The scenes displayed on the stereoscopic display 151 have a predetermined brightness, referred to as the target brightness. In one embodiment, the target brightness is set by the teleoperated surgical systems 100A, 100B. However, in some embodiments, the surgeon is allowed to adjust the target brightness so that the brightness of the displayed scene is acceptable to the surgeon.

[0030] In one aspect, the endoscope control system includes a controller, which in turn includes a camera control unit 130L, 130R and a lighting control unit 115. In one aspect, the camera control unit 130L, 130R includes two control loops 131, 132 (see FIG. 1A ). The combination of the control loops 131, 132 is configured to maintain a target brightness of the displayed scene (the displayed scene being the scene in the captured frame) and to maintain a minimum output of the lighting device.

[0031] If the gain and exposure control loop 131 detects a decrease in the overall brightness of the scene captured by the left image sensor 121L and the scene captured by the right image sensor 121R as the endoscope 101 moves away from the tissue 103, the gain and exposure control loop 131 automatically commands either the camera 120L, 120R to increase the camera exposure time so that subsequently captured images have a target brightness, or commands the video pipeline 140 to increase a gain that controls the brightness of the scene displayed on the display unit 151 so that the displayed scene has a target brightness. In some circumstances, the gain and exposure control loop 131 automatically commands both an increase in the camera exposure time and an increase in the video pipeline gain. The power and exposure control loop 132 determines whether to increase the optical power output from the illuminator 110. If the output optical power increases, the power and exposure control loop 132 increases the output optical power of the illuminator 110 and decreases the exposure time of the cameras 120L, 120R in synchronous linear steps of fixed size per frame.

[0032] The increase in output optical power and the decrease in exposure time are the same size change, for example, a 1% increase in output optical power results in a 1% decrease in exposure time. If the system were perfect, there would be no change in the overall brightness of the scene captured with the new output optical power and the new exposure time compared to the scene captured with the original output optical power and the original exposure time. However, the system is not perfect, so there may be changes in the overall brightness of the scene captured with the new output optical power and the new exposure time. Thus, the fixed size step is selected so that the surgeon does not notice the flicker in brightness of the displayed image caused by the synchronous changes in output optical power and exposure time.

[0033] Similarly, if the gain and exposure control loop 131 detects an increase in the brightness of the scene captured by the left image sensor 121L of the left camera 120L and the right image sensor 121R of the right camera 120R as the endoscope 101 moves towards the tissue 103, the gain and exposure control loop 131 automatically commands either the camera 120L, 120R to decrease the camera exposure time so that subsequently captured images have a target brightness, or commands the video pipeline 140 to decrease the gain that controls the brightness of the scene displayed on the display unit 151 so that the displayed scene has a target brightness. In some circumstances, the gain and exposure control loop 131 automatically commands both a decrease in the camera exposure time and a decrease in the gain of the video pipeline. The power and exposure control loop 132 determines whether to decrease the optical power output from the illuminator 110. When the output optical power needs to be reduced, the power and exposure control loop 132 reduces the output optical power of the illuminator 110 and increases the exposure time of the cameras 120L, 120R in fixed size synchronous linear steps per frame. However, when the reduction in output optical power reaches the minimum output optical power, the output optical power of the illuminator 110 is maintained at the minimum output optical power.

[0034] In another embodiment, the camera control units 130L, 130R include three control loops 131, 132, 133 (see FIG. 1B). The combination of the control loops 131, 132, 133 is configured to maintain a target brightness of the displayed scene (the displayed scene is the scene in the captured frame), to limit saturated pixels of the scene in the captured frame below a respective predetermined number of pixels of the scene, and to maintain a minimum output of the lighting device. In this embodiment, the control loops 131, 132 function in a similar manner as described above.

[0035] In addition to variations in brightness that can be distracting to the surgeon, areas of the displayed scene that fail to convey details of that area can be distracting to the surgeon. Image sensors 121L, 121R capture light into wells called pixels. Each well has a limited capacity, and if too much light is captured in a well, the well will overflow, thereby losing useful information. When a well overflows, the pixel is called a saturated pixel.

[0036] If an area of ​​tissue 103 reflects so much light that the pixels capturing the light reflected by that area of ​​tissue become saturated, the surgeon will see that area on the stereoscopic display 151 as a bright spot with no detail. Any information about the nature of the tissue, blood vessels, etc. in that area will be lost. Thus, in one aspect, the saturated pixels control loop 133 determines the number of saturated pixels in an entire frame, and if the number of saturated pixels is large enough to be distracting to the surgeon, the saturated pixels control loop 133 reduces the target brightness in fixed size linear steps for each frame until the number of saturated pixels for the frame is less than a number that would not distract the surgeon. However, the reduction in the target brightness is range limited. This means that the target brightness will be reduced to below a certain percentage of the original target brightness, such as set by the system or selected by the surgeon, e.g., the maximum change in target brightness is limited to less than 35% of the original target brightness. Thus, the saturated pixels control loop 133 is configured to apply a reduction limit to the target brightness.

[0037] Because the saturated pixel control loop 133 analyzes the total number of saturated pixels in the entire frame, the operation of the saturated pixel control loop 133 is independent of the location of the saturated pixels within the frame. This approach maintains consistent brightness of the displayed scene when the location of a feature causes changes in saturated pixels in the displayed scene, for example as a surgical instrument moves about in the scene.

[0038] As the endoscope 101 (FIGS. 1A and 1B) continues to move towards the tissue 103, eventually the endoscope 101 will contact the tissue 103. Contact is monitored by detecting a change in output optical power level that results in little or no change in the brightness reflected from the tissue 103. When this condition is detected by the camera control units 130L, 130R, the output optical power of the illuminator 110 is reduced to a level that is safe for contact with the tissue.

[0039] Situations other than tissue contact exist where a change in output optical power results in little or no change in the brightness reflected from tissue 103, for example when the distal tip of endoscope 101 is retracted into a cannula. To prevent false positives or the like from disrupting the operation of endoscope 101, the output optical power of illumination device 101 is brought to a level safe for tissue contact, but the output optical power is changed in a known manner. When this known change in output optical power is detected by camera control units 130L, 130R, endoscope 101 is not in contact with tissue and normal control of the output optical power is resumed by camera control units 130L, 130R.

[0040] Thus, in one aspect, the controller detects contact between the tip of the endoscope and the tissue by monitoring the brightness reflected from the tissue 103 and by monitoring the output optical power. Upon detecting contact, the controller attenuates the output optical power from the tip of the endoscope. Since the reflected brightness is not measured directly, the controller determines the reflected brightness as a ratio of the overall brightness of the captured scene to the camera exposure time, where the overall brightness represents the average brightness of the frame.

[0041] Thus, to detect contact, the controller monitors the reflected brightness, i.e., the ratio of the brightness of the entire captured scene to the camera exposure time, and monitors the optical power output from the endoscope 101. The controller determines that the tip of the endoscope has contacted tissue when a change from the first optical power output to the second optical power output does not result in a change in the ratio, i.e., no change in reflected brightness, compared to the ratio of the scene captured by light reflected from the first optical power output.

[0042] One example of a teleoperated surgical system 100 (FIGS. 1A and 1B) is the da Vinci® minimally invasive teleoperated surgical system commercially available from Intuitive Surgical, Inc. of Sunnyvale, Calif. Teleoperated surgical systems 100A, 100B are illustrative and not limiting. In this example, a surgeon at a surgeon console 250 remotely operates an endoscope 201 that is attached to a robotic manipulator arm (not shown). There are other components, cables, etc. associated with the da Vinci® surgical system, but these are not shown in FIGS. 1A and 1B to avoid distracting from this disclosure. Further information regarding remotely operated minimally invasive surgical systems can be found, for example, in U.S. patent application Ser. No. 11 / 762,165 (filed Jun. 13, 2007, disclosing “Minimally Invasive Surgical System”) and U.S. Patent No. 6,331,181 (filed Dec. 18, 2001, disclosing “Surgical Robotic Tools, Data Architecture, and Use”), both of which are incorporated herein by reference.

[0043] An illumination system, such as an illumination device 110, is coupled to the endoscope 101. In one embodiment, the illumination device 110 includes a light source 111 and an illumination control device 115. The illumination control device 115 is coupled to the light source 111 and to the camera control units 130L, 130R.

[0044] In the embodiment of FIG. 1, the light source 111 includes a multi-color component illumination source 112. In one embodiment, the multi-color component illumination source 112 includes a multi-light emitting diode (LEDs). The use of LEDs is merely exemplary and not limiting. The multi-color component illumination source 112 can also be implemented with, for example, a multi-laser source or a multi-laser diode instead of LEDs. Alternatively, the light source 111 can use a xenon lamp including an elliptical back reflector and bandpass filter coatings to generate broadband white illumination light for the visible image. The use of a xenon lamp is also merely exemplary and not limiting. For example, a high pressure mercury lamp, other arc lamps, or other broadband light sources may be used.

[0045] In this embodiment, the illumination device 110 is used in conjunction with at least one illumination pathway within the stereoscopic endoscope 101 to illuminate the tissue 103. Output light from the illumination device 110 is directed to a connector 116, which provides light to an illumination pathway within the stereoscopic endoscope 101, which then directs the light to the surgical site 103. The connector 116 and each of the illumination pathways within the stereoscopic endoscope 101 may be implemented with, for example, a fiber optic bundle, a single rigid or flexible rod, or optical fibers. In one embodiment, the endoscope 101 also includes two optical channels, or stereoscopic optical paths, that pass light reflected from the surgical site 103 to the cameras 120L, 120R.

[0046] Camera 120L is coupled to a stereoscopic display 151 of the surgeon console 150 by a left camera control unit 130L and a video pipeline 140. Camera 120R is coupled to a stereoscopic display 151 of the surgeon console 150 by a right camera control unit 130R and a video pipeline 140. The camera control units 130L and 130R receive signals from a system process 162. The system process 162 and central controller 160 represent some of the various controllers within the system 100.

[0047] The display mode selection switch 152 provides a signal to the user interface 161, which in turn passes the selected display mode, e.g., high power mode, to a system process 162. Various controllers within the system process 162 configure the lighting controller 115, configure the left and right camera control units 130L, 130R to acquire the desired images, and configure other elements of the video pipeline 140 necessary to process the acquired images to present the images required by the surgeon on the display 150. The video pipeline 140 is similar to known video pipelines, except for the details provided herein.

[0048] Although described as a central controller 160, it should be understood that the central controller 160, as well as each of the other controllers described herein, may actually be realized by any number of modules, and each module may include any combination of components. Each module and each component may include hardware, software running on a processor, firmware, or any combination of the three. Also, the functions and actions of each of the central controller 160 and other controllers as described herein may be performed by a single module, or may be divided among different modules or even among different components of a module. When divided among different modules or components, the modules or components may be centralized in one location or distributed across the systems 100A, 100B for distributed processing. Thus, each of the central controller 160 and other controllers described herein should not be construed as requiring a single physical entity, and in some aspects the controllers may be distributed across the systems 100A, 100B.

[0049] 1A and 1B, the cameras 120L, 120R and the light source 112 are shown as being external to the endoscope 101. However, in one aspect, the cameras 120L, 120R and the light source 112 are included at the distal tip of the endoscope 101. For example, in FIG. 2, the camera 220 and the light source 211 are included at the endoscope 201 of the teleoperated surgical system 200.

[0050] Again, one example of a teleoperated surgical system 200 is the da Vinci® minimally invasive teleoperated surgical system described above. Teleoperated surgical system 200 is illustrative only and not limiting. As with teleoperated surgical systems 100A and 100B, there are other components, cables, etc. associated with the da Vinci® surgical system that are not shown in FIG. 2 to avoid distracting from this disclosure.

[0051] In this example, the controls include a camera control unit 230 and a lighting control unit 215. In one aspect, the camera control unit 230 represents camera control units 130L and 130R, and the camera 220 represents cameras 120L and 120R. In another aspect, the camera control unit 230 is coupled to a single optical channel of the camera 220.

[0052] The camera control unit 230 includes an auto exposure module 232 that controls the brightness of the scene displayed on the display unit 251. The light power output control by the illumination control device 215 is also tied to the auto exposure module 232 to provide automatic dimming of the illuminator when the surgical scenario requires more illumination than the minimum light power output, in cases where the surgical scenario does not require maximum light power output or automatic brightening of the illuminator. In clinical applications, it is desirable to use as much light as is necessary to provide good quality video. This can avoid any negative tissue interactions that may result from high power illumination.

[0053] Light from the light source 211 reflected by a target 203, e.g., a surgical site, is captured as a color scene by the camera 220. The light source 211 has an optical power output 212 that is controlled by an illumination control device 215 in response to commands from an autoexposure module 232 and in response to commands from a contact detection module 233. The target 203 has a brightness 205 that is a measure of the brightness of the target 203. Typically, as the optical power output 212 increases or decreases, the brightness 205 of the target 203 also increases or decreases. To avoid confusion with the brightness of the captured image, the brightness of the target 203 is referred to as the "reflected brightness" and the brightness of the captured image is referred to as the "captured image brightness" or "image brightness."

[0054] In one aspect, camera 220 is a rolling shutter camera. As used herein, rolling shutter means that instead of reading out an entire frame from the camera's image sensor at once, information is read out from each row of the frame sequentially, from top to bottom.

[0055] The camera 220 captures a continuous sequence of frames that comprise the video stream. Each captured frame contains a color scene that is a snapshot of the target 203 at a moment in time. The video 221 is streamed from the camera 220 to a statistics module 231 in the camera control unit 230. The statistics module 231 collects real-time statistics about each frame of the video stream. These statistics include a histogram of pixel brightness for the frame that is provided to an auto-exposure module 232. As described more fully below, the auto-exposure module 232 controls the target brightness, the exposure time of the camera 220, the gain of the video pipeline 240, and the optical power output from the endoscope 201. In one aspect, the auto-exposure module 232: Maintaining a target brightness of the video image displayed on the display unit 251; maintaining the total number of saturated pixels in each scene captured by the camera 220 below a saturated pixel threshold; It is set to maintain the minimum optical power output (minimum illumination brightness) required to achieve the target brightness of the scene captured by the camera 220.

[0056] In another embodiment, the auto exposure module 232 includes: Maintaining a target brightness of the video image displayed on the display unit 251; It is set to maintain the minimum optical power output (minimum illumination brightness) required to achieve the target brightness of the scene captured by the camera 220.

[0057] As used herein, target brightness is the brightness of a scene displayed on the display unit 251. The initial target brightness, sometimes referred to as the original target brightness, is set in the teleoperated surgical system 200 at a target brightness of, for example, 1500. However, in one aspect, a slide switch is presented in a user interface shown on the display unit 251 to allow a user to select a target brightness of the displayed scene that is acceptable to the user. The initial target brightness is determined empirically based on feedback from users of teleoperated surgical systems, such as the da Vinci® teleoperated surgical system. As described more fully below, the target brightness is a control parameter of a control loop within the auto-exposure module 232.

[0058] As will be explained more fully below, the auto-exposure module 232 ensures a constant brightness of the displayed scene within a frame by ensuring that the average brightness of the scene is equal to a target brightness, where the average brightness of the scene is referred to as the overall brightness of the scene or the overall brightness of the frame. The auto-exposure module 232 also limits the effect of saturated pixels in the displayed scene by applying a falloff limit to the target brightness. The auto-exposure module 232 controls the optical power output of the illuminator 210 using a brightness threshold with hysteresis.

[0059] As described more fully below, the control system implemented by auto exposure module 232 controls video pipeline gain 236 and camera exposure time 238. In the absence of saturated pixels, the change in overall luminance of a captured scene is a linear function of video pipeline gain 236 and camera exposure time 238.

[0060] However, saturated pixels in a scene are inherently non-linear. Unfortunately, very bright specular highlights combined with the limited bit depth or limited dynamic range of the camera 220 and / or video pipeline 240 can result in saturation of pixels in the scene displayed on the display unit 251. Also, saturated pixels are undesirable in an image because detail is lost in the areas of the saturated pixels.

[0061] Specular highlights are formed from anatomical structures that are highly reflective, and from metallic (highly reflective) instruments moving across the field of view of the endoscope 201. The auto-exposure module 232 is configured to minimize scene brightness changes as the instrument moves across the scene, and to minimize user adjustments of auto-exposure parameters.

[0062] One way to minimize the effect of a reflective instrument moving throughout the scene is to use spatial information available to the teleoperated surgical system 200 about the most likely location of the instrument and focus the autoexposure on areas of the scene where the instrument is unlikely to be located. One approach is to define a rectangular region in the center of the scene. In this case, the statistics module 231 will generate image statistics only for pixel locations within the rectangular region. This allows the instrument to move around the scene without affecting the brightness of the displayed scene. This approach can be useful for surgical procedures and teleoperated surgical systems when the instrument does not have a large range of motion and therefore does not move around the rectangular region frequently.

[0063] However, if the instrument has a large range of motion and thereby moves through the rectangular region, using statistics only for the rectangular region would create an invisible barrier (the outline of the region) that would cause a dramatic change in the brightness of the scene if a reflective instrument crossed the barrier, as the auto exposure module 232 would adjust for the increased brightness of the region due to reflections from the highly reflective instrument. Such a change in brightness associated with an instrument crossing an invisible barrier is undesirable, as it typically distracts the surgeon. For this reason, statistics of the entire visible scene are considered in one aspect. In this aspect, the entire frame is used to generate the displayed scene, so the statistics are generated for the entire captured frame. In another aspect, pixels that were not used to generate the displayed scene do not include the statistics, e.g., video processing crops the captured frames to obtain the displayed scene, so the cropped pixels are not used to generate the statistics.

[0064] Thus, in one embodiment, the statistics module 231 creates a luminance histogram for every pixel location of a frame captured by the image sensor of the camera 220. At each pixel location of the frame, there are red, green and blue pixels. The statistics module 231 converts the red, green and blue pixel values ​​into values ​​proportional to the brightness of that pixel location, e.g., image luminance.

[0065] The statistics module 231 counts the number of pixel locations with each possible luminance value in the frame and creates a histogram. If luminance is represented as a byte number, there are 256 possible luminance values ​​ranging from 0 to 255. Figure 3 shows a typical histogram 300 of a frame containing saturated pixels. The possible luminance values ​​are plotted on the x-axis. The height of the bar for each luminance value represents the number of pixel locations with that luminance value in the frame. Also along the x-axis is a grayscale representation of the luminance.

[0066] The statistics module 231 generates statistics for captured frames without requiring a frame buffer, thus eliminating the need for additional storage to process the frames as well as the time required to save to and read from the frame buffer.

[0067] 4 is a process flow diagram of one embodiment of auto-exposure module 232. In this embodiment, auto-exposure module 232 includes a saturated pixel limit process 441, sometimes referred to as process 441, a gain and exposure time adjust process 442, sometimes referred to as process 442, and a power and exposure time adjust process 443, sometimes referred to as process 443.

[0068] The auto-exposure module 232 implements three control loops in FIG. 1A and FIG. 1B. The first control loop (saturated pixel control loop 133) includes a saturated pixel limit process 441, the second control loop (gain and exposure control loop 131) includes a gain and exposure time adjustment process 442, and the third control loop (power and exposure control loop 132) includes a power and exposure time adjustment process 443. Time constants are selected for these three control loops so that the control provided by the auto-exposure module 232 is stable. The first control loop, including the saturated pixel limit process 441, has a longer time constant than the time constant of the second control loop, including the gain and exposure time adjustment process 442. The third control loop, including the power and exposure time adjustment process 443, has a longer time constant than the time constant of the second control loop. Although FIG. 4 shows three processes 441, 442, and 443, in some embodiments, only processes 442 and 443 are implemented (see FIG. 1A).

[0069] As described more fully below, the Limit Saturated Pixels process 441 can minimize the total screen area of ​​the display 251 consumed by saturated pixels over time. The Limit Saturated Pixels process 441 controls the Adjusted Target Brightness 444 to perform this minimization. Figure 5A is a process flow diagram of one embodiment of the Limit Saturated Pixels process 441. Upon completion, the Limit Saturated Pixels process 441 transitions to the Adjust Gain and Exposure Time process 442.

[0070] The Gain and Exposure Time Adjustment process 442 controls the video pipeline gain and the camera exposure time. The Gain and Exposure Time Adjustment process 442 adjusts either the Video Pipeline Gain 445 or the Camera Exposure Time 446 based on the relationship of the average luminance of the frame to the Adjusted Target Luminance 444. The Video Pipeline Gain 445 provides the Video Pipeline Gain 236. The Camera Exposure Time 446 provides the Camera Exposure Time 238.

[0071] 5B is a process flow diagram of one embodiment of the gain and exposure time adjustment process 442. Upon completion, the gain and exposure time adjustment process 442 becomes the power and exposure time adjustment process. Transition to 443.

[0072] The power and exposure time adjustment process 443 controls the optical power output of the illuminator 210 and the camera exposure time. If the camera exposure time is less than the second exposure threshold (see FIG. 6), the optical power output is decreased and a positive adjustment is applied to the camera exposure time to compensate for the illumination change. If the camera exposure time is greater than the first exposure threshold (see FIG. 6), the optical power output of the illuminator 210 is increased and a negative adjustment is applied to the camera exposure time to compensate for the illumination change. Thus, the power and exposure time adjustment process 443 determines whether a change in the optical power output 448 and the camera exposure time adjustment 447 is necessary and then returns the process to the saturated pixel limit process 441 for processing the next frame. The optical power output 448 provides the optical power output 237. FIG. 5C is a process flow diagram of one embodiment of the power and exposure time adjustment process 443.

[0073] As noted above, Figure 5A illustrates one embodiment of the saturated pixel limit process 441. The saturated pixel count process 501 counts the number of pixels in the highest bin(s) of the histogram 300. If the number of pixels in the highest bin(s), e.g., bin 255, is greater than the saturated pixel threshold, then the excessive saturated pixel check process 502 transitions to a target luminance reduction process 503, otherwise transitions to an adjusted target luminance reduction check process 504. Although the saturated pixel count process 501 is shown in this embodiment as being included in the saturated pixel limit process 441, the saturated pixel count process 501 can be included in the statistics module 231 instead of the saturated pixel limit process 441.

[0074] In one embodiment, a count saturated pixels process 501, sometimes referred to as process 501, counts the number of saturated pixels 301 in the histogram 300. In the example of FIG.

[0075] If the endoscope 201 is a stereo endoscope, the statistics module 231 creates a histogram for each of the left and right optical channels. If the endoscope 201 is not a stereo endoscope, the statistics module 231 creates a histogram for a single optical channel. In one embodiment, two bytes are used to represent luminance, so the histogram has 512 bins along the x-axis. In one embodiment, the saturated pixel count process 501 determines the number of saturated pixels as the sum of the number of pixels in each bin 511 of the left and right histograms.

[0076] Some operating modes of the teleoperated surgical system 200 can mark one of the left and right scenes as invalid. In these operating modes, a count of saturated pixels is performed on the histogram of the valid scene.

[0077] In one aspect, a saturated pixel threshold for the number of saturated pixels permitted in a scene was empirically determined, and the teleoperated surgical system 200 was configured to vary the size of the saturated regions of the scene displayed on the display unit 251. Clinical trials with such a system were used to determine the saturated pixel threshold and the bins used to count the saturated pixels, thereby allowing a user of the system to perform a set of surgical tasks without labeling the region saturation as a hindrance or impediment to accomplishing the task.

[0078] In one embodiment, using 512 bins (labeled bin 0 to bin 511) of the histogram and a stereoscopic endoscope, the saturated pixel threshold is selected as 12000 at bin 511. Thus, for this embodiment, the over-saturated pixel check process 502 compared the saturated pixel threshold of 120000 to the sum of the total number of saturated pixels in the left histogram bin 511 and the total number of saturated pixels in the right histogram bin 511 received from the statistics module 231. By adjusting the maximum number of saturated pixels and the number of bins used to count the saturated pixels, the auto exposure module 232 can adjust the size of the saturated area on the screen and compensate for the gain later in the video pipeline. If the number of saturated pixels in the frame is greater than the saturated pixel threshold, the check process 502 transitions to the target brightness reduction process 503.

[0079] The total number of saturated pixels in the same bin in the left and right frames is used to provide a measure of the total area of ​​saturated pixels in the displayed scene, and its use is independent of the location of the saturated pixels on the screen (i.e., the saturated pixel area can move throughout the camera's field of view with little change to the pixel histogram). Nevertheless, if the number of saturated pixels exceeds the saturated pixel threshold, the target brightness is reduced by the target brightness reduction process 503. As a result, most of the saturated pixels will darken the scene displayed by the auto exposure module 232, causing non-reflective areas of the scene to become very dark. For this reason, the target brightness reduction is limited in range.

[0080] In one embodiment, the time constant of the saturated pixel limit process 441 is 4-5 seconds, and the maximum reduction in target brightness, the range limit of the target brightness is 35% of the original target brightness 266. The range limit of the target brightness is determined empirically. This range is determined to establish the minimum average brightness of the displayed scene that the user identifies as providing information that can be used in a surgical procedure. The ratio of the minimum average brightness to the original target brightness is used to define the range limit of the target brightness, for example, as follows: Target luminance range limit = (1-(minimum acceptable average luminance / original target luminance))*100

[0081] To determine the allowable change in target brightness per frame, a 7% change per second is allowed for a time constant of 5 seconds and a brightness range limit of 35%. For a camera capturing 60 frames per second, the change in target brightness per frame is a fixed linear step of 0.11% until either the number of saturated pixels in the captured scene is less than the saturated pixel threshold or the change from the initial target brightness 266 is at the range limit, e.g., the target brightness is reduced by the range limit of the target brightness. In this example, for an initial target brightness of 1500 and a range limit of 35%, the limit of the target brightness is (1500*(1-0.35)), a limit of 975.

[0082] Thus, if the adjusted target brightness is not at the limit of the target brightness range, then the target brightness reduction process 503 changes the adjusted target brightness by one fixed size linear step and saves the result as adjusted target brightness 444. After adjusting the target brightness, the saturated pixels limit process 441 transitions to the gain and exposure time adjustment process 442. If the adjusted target brightness is at the limit of the target brightness range, i.e., the target brightness is range limited, then the target brightness reduction process 503 does no action and transitions to process 442.

[0083] If the number of pixels in the highest bin(s) is less than the saturated pixel threshold, then the check for excessively saturated pixels process 502 transitions to a check for adjusted target brightness reduction process 504, sometimes referred to as process 504. Process 504 compares the adjusted target brightness 444 to the original target brightness 266. If the adjusted target brightness 444 is less than the original target brightness 266, then process 504 transitions to an increase target brightness process 505, otherwise it transitions to the adjust gain and exposure time process 442.

[0084] The Increase Target Brightness process 505 increases the adjusted exposure time by one fixed linear step of the size noted above and stores the result as Adjusted Target Brightness 444. Process 505 transfers to the Adjust Gain and Exposure Time process 442.

[0085] As indicated above, FIG. 5B illustrates one embodiment of the gain and exposure time adjustment process 442. The frame average brightness calculation process 511, sometimes referred to as process 511, determines the average value of brightness from the brightness histogram, sometimes referred to as the average brightness. As described above, if the endoscope 201 is a stereo endoscope, the statistics module 231 creates a histogram for each of the left and right optical channels. If the endoscope 201 is not a stereo endoscope, the statistics module 231 creates a single histogram. Thus, for a stereo endoscope, the process 511 determines a left average brightness of the left scene captured from the left optical channel, and a right average brightness of the right scene captured from the right optical channel. The left average brightness and the right average brightness are averaged by the process 501 to obtain a frame average brightness, i.e., the brightness of the entire frame. Although the frame average brightness calculation process 511 is shown in this embodiment as being included in the gain and exposure time adjustment process 442, the frame average brightness calculation process 511 can be included in the statistics module 231 instead of the gain and exposure time adjustment process 442.

[0086] Some operating modes of the teleoperated surgical system 200 may mark one of the left and right scenes as invalid. In these operating modes, the average of the valid scenes is taken as the frame average brightness by process 511. After determining the frame average brightness, process 511 transfers processing to frame brightness check against target brightness process 512.

[0087] A frame brightness check process 512 against target brightness, sometimes referred to as check process 512, compares the frame average brightness to the adjusted target brightness 444. If the frame average brightness is greater than the adjusted target brightness 444, then the check process 512 transfers to a decrease gain and exposure time process 514. If the frame average brightness is less than the adjusted target brightness 444, then the check process 512 transfers to an increase gain or exposure time process 513.

[0088] When processing transitions to Increase Gain or Exposure Time process 513, sometimes referred to as process 513, the brightness of the captured image is too low. If the camera exposure time is not at the maximum camera exposure time Emax, process 513 increases the camera exposure time (see FIG. 6) and then increases the frame average brightness of subsequently captured frames. However, if the camera exposure time is at the maximum camera exposure time Emax, the camera exposure time cannot be increased further. In this situation, process 513 increases the gain of the video pipeline.

[0089] Thus, process 513 first determines the ratio of adjusted target brightness 444 to the frame average brightness. Since the frame average brightness is less than adjusted target brightness 444, the ratio is greater than 1. As an example, assume a ratio of 1.2, which requires, for example, a 20% increase in brightness. Brightness is a linear function of camera exposure time. Thus, when camera exposure time 446 is less than the maximum camera exposure time Emax, process 513 multiplies camera exposure time 446 by the ratio of adjusted target brightness 444 to frame average brightness, e.g., camera exposure time is multiplied by 1.2, and the result is stored as camera exposure time 446. When camera exposure time 446 is at the maximum camera exposure time Emax, process 513 multiplies video pipeline gain 445 by the ratio of adjusted target brightness 444 to frame average brightness, e.g., video pipeline gain is multiplied by 1.2, and the result is stored as video pipeline gain 445. If multiplying camera exposure time 446 by the ratio of adjusted target brightness 444 to frame average brightness gives a camera exposure time greater than the maximum camera exposure time Emax, then the increase is divided between the camera exposure time and the video pipeline gain 445, so that the camera exposure time 446 is at the camera exposure time Emax, and the remainder of the increase is applied to the video pipeline gain 445. If camera exposure time 446 is 10% lower than the maximum camera exposure time Emax, for example with a 20% increase, then the camera exposure time 446 is increased such that the camera exposure time 446 is at the maximum camera exposure time Emax, and the video pipeline gain 445 is increased by the remainder of the increase. For example, New Video Pipeline Gain = 1.2*(1.0-0.1) = 1.08 Once complete, process 513 transfers to adjust power and exposure time process 443.

[0090] Processing transitions to a Decrease Gain or Exposure Time process 514, sometimes referred to as process 514, when the brightness of the captured image is too high. If the video pipeline gain is greater than one (see FIG. 6), process 514 decreases the video pipeline gain, decreasing the brightness of the displayed scene. However, if the video pipeline gain is equal to one, process 514 decreases the camera exposure time, decreasing the frame average brightness of the subsequently captured image.

[0091] Thus, process 514 calculates the ratio of the adjusted target luminance 444 to the frame average luminance. Since the frame average luminance is greater than the adjusted target luminance 444, The ratio is less than 1. As an example, a ratio of 0.0 requires the brightness to be reduced by 20%. Assume 8.

[0092] If the video pipeline gain 445 is greater than one, process 514 multiplies the video pipeline gain 445 by the ratio of adjusted target brightness 444 to frame average brightness, e.g., the video pipeline gain is multiplied by 0.8, and stores the result as the video pipeline gain 445. If the video pipeline gain 445 is one, process 514 multiplies the camera exposure time 446 by the ratio of adjusted target brightness 444 to frame average brightness, e.g., the camera exposure time is multiplied by 0.8, and stores the result as the camera exposure time 446. If multiplying the video pipeline gain 445 by the ratio of adjusted target brightness 444 to frame average brightness gives a video pipeline gain less than one, the reduction is divided between the camera exposure time and the video pipeline gain, so that the video pipeline gain 445 is one, and the remainder of the reduction is applied to the camera exposure time 446. For an example with a 20% falloff, if the video pipeline gain 445 is 1.1, the video pipeline gain 445 is lowered to 1 and the camera exposure time 446 is decreased by the remaining fraction of the falloff. For example, New Camera Exposure Time = Camera Exposure Time * (1-0.2) / (1 / 1.1)) Once complete, process 514 transfers to adjust power and exposure time process 443.

[0093] The power and exposure time adjustment process 443 controls the output optical power of the illuminator 210 to the desired output optical power of the light source 211 by sending commands to the illumination controller 215 and sends corresponding exposure time adjustments to the camera exposure time 446. The changes in output optical power and camera exposure time are synchronized as described below in connection with FIG.

[0094] Figure 5C is a process flow diagram of one embodiment of adjust power and exposure time process 443. However, before considering Figure 5C, the parameters used in the process of Figure 5C will be discussed.

[0095] In one aspect, process 443 increases or decreases the brightness of the lighting output from light source 211 in fixed size synchronous steps. The camera exposure time is changed the same percentage as the lighting change, e.g., if the step change multiplies the lighting output by 1.006, then the camera exposure time is divided by 1.006.

[0096] To ensure a fixed size step, a rise time tramp in seconds is specified as the period used to linearly ramp the illumination output from the light source 211 from the minimum optical power output Pmin to the maximum optical power output Pmax, and vice versa.

[0097] The fixed size optical power output step Pstep is defined as follows: Pstep=(Pmax-Pmin) / (tramp*(frames / camera seconds)).

[0098] As an example, consider a camera with a maximum optical power output of 800 milliwatts (mW), a minimum optical power output of 400 mW, a rise time tramp of 3 seconds, and capturing 60 frames per second. When the optical power output is changed, the fixed size step is + / - 2.22 mW / frame. If the current optical power output is 400 mW, the illumination output is multiplied by (1 + (2.22 / 400)), or 1.0056. Thus, the camera exposure time is divided by 1.0056.

[0099] As indicated above, illumination changes should not produce noticeable flicker in the displayed scene. Thus, the rise time is selected such that illumination changes do not produce noticeable luminance flicker in the scene displayed on the display unit 251 and such that the three control loops are stable, e.g., the rise time is the time constant of the third control loop. Thus, in these examples, the time constant of the first control loop is 5 seconds and the time constant of the third control loop is 3 seconds.

[0100] The new output optical power Pnew at the distal tip of the endoscope 201 is the current output optical power Pcurrent plus or minus the output optical power step Pstep. That is, Pnew=Pcurrent+ / -Pstep Here, in this example, the output optical power step Pstep is 0, +2.22, or -2.22. As explained below, one of three possible values ​​of the output optical power step Pstep is used depending on the value of the camera exposure time. Also, as explained more fully below in connection with FIG. 7, the changes in output optical power and exposure time are synchronized to the frame time of the pipeline.

[0101] Although the step change in output optical power per frame is known, the illumination controller 215 controls the current to each light source of the plurality of light sources 211 when the light sources are LEDs. However, the optical power output of an LED is not linear with respect to the change in current. Thus, a lookup table is used to convert the commanded change in output optical power to a current for each LED, which in turn provides the output optical power to the distal tip of the endoscope 201.

[0102] The values ​​in the lookup table are determined by calibration of the light source 211 against a known standard. Since the optical power output is at the distal tip of the endoscope 201, the current of each LED is determined so that the actual output of the light source 211 is high enough to account for any attenuation of light between the output of the light source 211 and the distal tip of the endoscope 201.

[0103] In one embodiment, the maximum optical power output Pmax is selected such that at the minimum working distance between the distal tip of the endoscope 201 and the target 203, the maximum optical power output Pmax does not cause damage to tissue.

[0104] The minimum optical power output Pmin is selected such that at the minimum working distance between the distal tip of the endoscope 201 and the target 203, the noise of the displayed scene is below a threshold and the optical power output exceeds the minimum achievable optical power output of the illumination device 210.

[0105] The value of the camera exposure time is used to check whether the change in the illumination output increases, decreases, or stays the same. For very small camera exposure times in the third region 603 (see FIG. 6) when the average brightness of the captured scene is high, the optical power output is dimmed. Specifically, for each successive frame with the camera exposure time in the third region 603, the optical power output is reduced by the optical power output Pstep, and the exposure time of the camera 220 is increased at the same rate as the reduction rate of the optical power output. If the optical power output step Pstep reduces the illumination output below the minimum optical power output Pmin, the illumination output does not change, the illumination output is maintained at the minimum optical power output Pmin, and the exposure time of the camera 220 is not changed.

[0106] Assume that the camera exposure time is increased towards the second region 602, with an increase in exposure time and a corresponding decrease in illumination resulting in a captured image with approximately the same average brightness as the previously captured image.

[0107] At very high camera exposure times in the first region 601 when the average brightness of the captured scene, overall scene brightness is low, the output optical power is increased. Specifically, for each successive frame with the exposure time of the first region 601, the output optical power is increased by a step Pstep of the output optical power, and the exposure time of the camera 220 is decreased at the same rate as the increase in the output optical power. If the change in the output optical power would increase the illumination output above the maximum output optical power Pmax, the illumination output is not changed, and the illumination output is maintained at the maximum output optical power Pmax, and the exposure time of the camera 220 is not changed.

[0108] A decrease in exposure time and a corresponding increase in illumination will result in a captured image with approximately the same average brightness as the previously captured image, assuming the camera exposure time is decreased towards the second region 602 .

[0109] Thus, the control loop including the power and exposure time adjustment process 443 varies the illumination such that the illumination output moves toward an illumination output range as represented by the second region 602. The second region 602 is a hysteresis region.

[0110] It is desirable to minimize lighting changes whenever possible. While brightness changes associated with lighting changes may not be noticeable to the surgeon over portions of the scene located at the average working distance between the distal tip of the endoscope 201 and the target 203, at locations in the scene greater than the average working distance, the surgeon is likely to notice distracting brightness changes. Thus, in this embodiment, the second region 602, where lighting and exposure changes are not implemented with changes in camera exposure time, is assigned 50% of the range of possible camera exposure times. In light of the present disclosure, the range of the second region 602 may be selected as a percentage other than 50% of the range of possible camera exposure times.

[0111] Thus, as shown in Fig. 6, the power and exposure time adjustment process 443 sets the output optical power and the camera exposure time based on the value of the camera exposure time. The third control loop, the power and exposure control loop 132, increases the illumination output and decreases the camera exposure time if the output optical power is less than the maximum output optical power Pmax when the camera exposure time is greater than the first exposure threshold. The third control loop decreases the output optical power and increases the camera exposure time if the output optical power is greater than the minimum output optical power Pmin when the camera exposure time is less than the second exposure threshold. The third control loop leaves the output optical power and the camera exposure time unchanged if the exposure time is between the first and second exposure thresholds.

[0112] In one embodiment, the maximum optical power output Pmax corresponds to an optical power output of 800 mW, the minimum optical power output Pmin corresponds to an optical power output of 400 mW, the maximum camera exposure time Emax is 4096, and the minimum camera exposure time Emin is zero.

[0113] 5C, check exposure time below second threshold process 521, sometimes referred to as process 521, determines whether camera exposure time 446 is below a second exposure threshold. If camera exposure time 446 is below the second exposure threshold, process 521 transfers processing to decrease illumination and increase exposure time process 523, otherwise it transfers processing to check exposure time below first threshold process 522.

[0114] When processing transfers to the Reduce Illumination and Increase Exposure Time process 523, sometimes referred to as process 523, the camera exposure time 446 is in the third region 603. Thus, if the illumination output is greater than the minimum optical power output Pmin, the process 523 decreases the optical power output 448, e.g., the optical power output Pcurrent, by the optical power output step Pstep. The process 523 loads the step increase to the camera exposure time adjustment 447, which then increases the camera exposure time 446. If the illumination output is equal to the minimum optical power output Pmin, the process 523 takes no action. The Reduce Illumination and Increase Exposure Time process 523 transfers processing to the Saturated Pixels Limit process 441.

[0115] If check process 521 transfers processing to check exposure time less than first threshold process 522, check process 522 determines whether camera exposure time 446 is less than the first exposure threshold. If camera exposure time 446 is less than the first exposure threshold, check process 522 transfers processing to increase illumination and decrease exposure time process 524, otherwise it transfers processing to limit saturated pixels process 441.

[0116] When processing transfers to the Increase Illumination and Decrease Exposure Time process 524, sometimes referred to as process 524, the camera exposure time 446 is in the first region 601. Thus, as long as the illumination output is less than the maximum optical power output Pmax, the process 524 increases the optical power output 448, e.g., the optical power output Pcurrent, by the optical power output step Pstep. The process 524 loads the camera exposure time adjustment 447 to stepwise decrease, and then decreases the camera exposure time 446. If the optical power output is equal to the maximum optical power output Pmax, the process 524 performs no action. The Increase Illumination and Decrease Exposure Time process 524 transfers processing to the Saturated Pixels Limit process 441.

[0117] When the power and exposure time adjustment process 443 is completed, the process returns to the saturated pixel limit process 441. Thus, the processes 441 to 443 are repeated for each captured frame.

[0118] To make the lighting changes invisible to the user of the teleoperated surgical system 200, the lighting changes are synchronized with compensation for changes in camera exposure time so that the overall brightness of the video displayed to the surgeon remains approximately constant with changes in lighting. For example, a decrease in output optical power and an increase in camera exposure time should occur in the same video frame. In this context, approximately constant means constant within the tolerance of the teleoperated surgical system 200.

[0119] The processing of the video stream in the teleoperated surgical system 200 is delayed. Lighting control, video pipeline gain control, camera exposure control, video statistics collection and analysis all result in different pipeline delays. For example, at frame time t0 (FIG. 7) for the first frame 701 in the video stream, the frame is captured, statistics are collected for frame 701 by the statistics module 231, and the gains from the previously captured frames are written to the video pipeline.

[0120] At frame time t1, the illumination output and camera exposure time are calculated by auto exposure module 232, and the camera exposure time is written to camera 220. At the start of the frame time t2, the output optical power changes to a new output optical power Pnew. Frame 702 is captured by the camera at frame time t3 with the exposure time and output optical power generated using the first frame 701. The new exposure time and output optical power are thus synchronized for the capture of frame 702.

[0121] These steps are pipelined such that exposure time and illuminator brightness are applied to each frame of video. In one aspect, pipeline synchronization is achieved by adding metadata to the video frame sync signal that travels throughout the system 200 with the video stream, i.e., attaching metadata to each frame of the video stream. The metadata includes camera exposure time, illuminator brightness, and video pipeline gain.

[0122] In one aspect, the teleoperated surgical system includes a method for increasing illumination of the surgical field upon command of the surgeon and a method for detecting contact between tissue and the endoscope 201. When tissue contact is detected, illumination from the endoscope 201 is automatically reduced to a level that is safe for contact with tissue.

[0123] The signal-to-noise ratio of the scene displayed on the display unit 251 decreases as less light is reflected back from the illumination target 203 to the image sensor of the camera 220. This decrease in signal-to-noise ratio can limit the usable working distance 204 of the endoscope 201 (the distance between the tip of the endoscope and the imaging target 203).

[0124] However, there is a trade-off between optical power output level and risk of tissue damage. As mentioned, increased optical power output can result in an increase in the available working distance 204. However, increased optical power output can also increase the risk of tissue damage if the optical power output is high enough and the working distance is made small enough so that the incident light does not exceed the damage level. The increased risk of tissue damage can be caused either by increased optical power density of the incident light or by the heated endoscope tip contacting tissue. In either case, the risk is largely eliminated by lowering the optical power output level.

[0125] To avoid risk of tissue damage, the optical power output level from the endoscope 201 is limited such that the optical power output level is safe for direct contact between the tip of the endoscope and tissue. However, this limits the maximum working distance of the endoscope 201. For illustrative purposes, optical power output levels of less than 1 W are typically considered safe. However, in one aspect, the teleoperated surgical system 200 includes a high beam mode of operation in which the optical power output is greater than a safe optical power output, for example, an optical power output greater than 1 Watt, such as 1.5 W.

[0126] In one aspect, the high beam mode of operation, sometimes referred to as the high power mode of operation, is initiated by the surgeon by activating a physical switch 265 on the surgeon console. In another aspect, the high beam mode of operation is initiated by the surgeon clicking a switch presented in a user interface shown on the display unit 251.

[0127] When the surgeon has control in the high beam mode, the high beam mode of operation should only be activated when there is a low risk of tissue damage, since in this mode of operation the optical power output is greater than is considered safe. Inspecting the surgical field in abdominal surgery is a common application of the high beam mode of operation. However, there is still a risk that the surgeon may inadvertently move the endoscope 201 too close to tissue or make unintended contact with tissue.

[0128] If the endoscope 201 is inadvertently moved very close to tissue, as explained above, the average brightness of the captured image will increase, which will cause the auto-exposure module 232 to decrease the camera exposure time. As the camera exposure time continues to decrease, the exposure time will reach a third region 603. The auto-exposure module 232 will then begin to decrease the output optical power. Thus, the increase in illumination due to the inadvertent movement of the endoscope 201 very close to tissue is automatically handled by the auto-exposure module 232 to decrease the output optical power.

[0129] If the endoscope 201 contacts tissue, the contact is detected by the contact detection module 233 and illumination is reduced to an appropriate level in either normal or high beam operating modes. The contact detection module 233 determines tissue contact by detecting that a change (increase or decrease) in the optical power output does not result in a corresponding change in the average brightness of the light reflected from the target 203, where the average reflected brightness is taken as the average brightness of the captured scene divided by the camera exposure time. The average brightness of the scene is determined by a frame average brightness calculation process 511 (FIG. 5C), where the camera exposure time is available in the camera exposure time 446. Alternatively, the frame average brightness calculation process 511 (FIG. 5C) can be incorporated into the contact detection module 233 or the statistics module 231.

[0130] For example, when the tip of the endoscope is covered, i.e., in contact with tissue, and thus light does not enter the camera lens, the automatic exposure module 232 increases the light power output. However, tissue contact still prevents reflected light from reaching the camera. As a result, when the light power output increases, there is no change in the average reflected luminance. Thus, the contact detection module 233 detects tissue contact and attenuates the light power output to a safe level for tissue contact. In one aspect, the light power output safe for contact with tissue is greater than the minimum light power output. However, when tissue damage is dominated by conductive heat due to contact between the tip of the endoscope and the tissue, the light power output safe for contact with tissue may be below the minimum light power output.

[0131] In one aspect, the contact detection module 233 disables the light power output control of the automatic exposure module 232 and commands the illumination control device 215 to reduce the light power output to a safe level. Alternatively, the contact detection module 233 can also command the illumination control device 215 to activate the attenuator of the light source 211 and reduce the light power output from the light source 211 to a safe level.

[0132] In one aspect, tissue contact detection is performed as follows. dL / dI = 0 Here, dL is the change in the average reflected luminance of two captured frames, and dI is the change in the light power output of two captured frames.

[0133] In another aspect, contact is detected when shown below. -Threshold < dL / dI < Threshold Here, the threshold includes the measurement uncertainty and noise in the measurement of dL and dI.

[0134] Thus, the contact detection method utilizes varying output optical power. In a system with dynamic illumination control, the output optical power changes as the tip of the endoscope approaches tissue. Automatic exposure control reduces the output optical power as tissue approaches the tip of the endoscope 201. Once tissue contacts the tip of the endoscope 201, little or no change in reflected brightness is detected with varying optical power. Upon detecting this condition, the brightness of the illuminator is attenuated, or reduced, to a predetermined level that is safe for tissue contact, e.g., illumination Pmin.

[0135] In another aspect, instead of monitoring changes in reflected brightness with changes in output optical power, a characteristic of the entire reflected brightness profile as tissue contacts the tip of the endoscope is used for contact detection, or improved contact detection. The characteristic of the entire reflected brightness profile can be in either the time domain or the frequency domain. Contact is detected when the average reflected brightness determined from the captured scene matches the characteristic of the entire brightness profile within a threshold range. In one aspect, the characteristic of the entire reflected brightness profile is determined empirically by measuring the overall reflected brightness of the scene as the tip of the endoscope approaches and contacts tissue.

[0136] A situation may exist where the system 200 is stuck in a low brightness state when removal of the contacting tissue does not result in a change in reflected brightness. To avoid a "stuck" state where the output optical power of the illuminator 210 does not change, a slow optical power output dithering technique is employed. When tissue contact is detected, the contact detection module 233 enables the dithering module 217. The dithering module 217 varies the output optical power in a known manner relative to a safe level to increase the reliability of detecting a no-contact state.

[0137] For example, dithering module 217 can vary the optical power output from light source 211 in a known time-dependent manner, e.g., a sinusoidal manner, about a safe level. When a time-dependent low-level reflected power output reaches the camera, auto-exposure module 232 detects a change in the optical power output, e.g., a change in the average brightness of the captured scene, signifying that the tip of endoscope 201 is no longer in contact with tissue. Thus, contact detection module 233 is reset such that auto-exposure module 232 assumes control of the optical power output.

[0138] The various modules described herein may be implemented by software running on a processor, hardware, firmware, or any combination of the three. When a module is implemented as software running on a processor, the software is stored in memory as computer readable instructions, and the computer readable instructions are executed on the processor. All or a portion of the memory may be in a different physical location than the processor, so long as the processor is coupled to the memory. Memory may refer to volatile memory, non-volatile memory, or any combination of the two.

[0139] Also, as described herein, the functionality of the various modules may be performed by one unit or may be divided among different components or different modules, each function may then be implemented by any combination of hardware, software running on a processor, and firmware. When divided among different components, the components may be centralized in one location or distributed throughout system 200 for distributed processing. Execution of the various modules results in a method that performs the processing described above for the various modules.

[0140] The processor is coupled to a memory that contains instructions that are executed by the processor. This can be accomplished within a computer system or through a connection to another computer via a modem and analog lines or a digital interface and digital carrier lines.

[0141] Here, a computer program product includes a computer readable medium configured to store or on which computer readable code for some or all of the processes described herein is stored. Some examples of computer program products are CD-ROM disks, DVD disks, flash memory, ROM cards, floppy disks, magnetic tapes, computer hard drives, servers on a network, and signals transmitted over a network representing computer readable program code. A non-transitory tangible computer program product includes a tangible computer readable medium configured to store or on which computer readable instructions for some or all of the processes are stored. A non-transitory tangible computer program product is a CD-ROM disk, DVD disk, flash memory, ROM cards, floppy disks, magnetic tapes, computer hard drives, and other physical storage media.

[0142] In view of this disclosure, the instructions used in some or all of the processes described herein may be implemented in a wide variety of computer system configurations using operating systems and computer programming languages ​​of interest to the user.

[0143] The above detailed description and the accompanying drawings illustrating aspects and embodiments of the present invention should not be construed as limiting, and the claims define the protected invention. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of the present specification and claims. In some instances, well-known circuits, structures, and techniques have not been shown or described in detail to avoid obscuring the present invention.

[0144] Additionally, the terminology in this detailed description is not intended to be limiting of the invention. For example, spatial terms such as "beneath," "below," "lower," "above," "upper," "proximal," "distal," and the like are used to describe the relationship of one element or feature to another element or feature shown in the figures. These spatial terms are intended to encompass different positions (i.e., locations) and orientations (i.e., rotational positions) of the device during use or operation in addition to the positions and orientations shown in the figures.

[0145] For example, if a device in the figures is turned over, elements described as "below" or "beneath" another element or feature would then be "above" or "over" the other element or feature. Thus, the exemplary term "below" can encompass both "above" and "below" positions and orientations. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatial descriptions used herein interpreted accordingly.

[0146] Similarly, descriptions of movement along and about various axes include various particular positions and orientations of the device. The singular forms "a," "an," and "the" are intended to include the plural forms unless the context dictates otherwise. The terms "comprises," "comprising," "including," and the like specify the presence of described features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups.

[0147] Components described as "coupled" may be directly coupled electrically or mechanically, or indirectly coupled through one or more intermediate components. In view of this disclosure, the instructions used in any or any combination of the operations described with respect to the improved display system may be implemented in a wide variety of computer system configurations using operating systems and computer programming languages ​​of interest to the user.

[0148] All examples and references to the description are non-limiting and should not be used to limit the scope of the claims to the specific implementations or embodiments described herein and their equivalents. Headings are for format only and should not be used to limit the subject matter in any manner, as the text under one heading may cross-reference, i.e., apply to the text under more than one heading. Finally, in light of this disclosure, a particular feature described in connection with one aspect or embodiment may be applicable to other disclosed aspects or embodiments of the invention, even though not specifically shown in the drawings or described in the text.

[0149] The contents of the claims as originally filed are described below as examples. [Example 1] A teleoperated surgical system, comprising: an illumination device having an optical power output; Camera and a control device coupled to the lighting device and the camera; The control device further includes a camera control unit and a lighting control device, the camera control unit being coupled to the lighting control device and the camera; the illumination controller is configured to control the optical power output from the illumination device in response to commands from the camera control unit; the camera control unit is configured to receive a video stream from the camera; the camera control unit is configured to instruct the lighting controller to change the optical power output from a first optical power output to a second optical power output, and to instruct the camera to change a camera exposure time from a first exposure time to a second exposure time, whereby a frame is subsequently captured by the camera at a second exposure time and with reflected light, the reflected light being light from the lighting device having the second optical power output before being reflected. Remotely operated surgery system. [Example 2] The remote surgical system of Example 1, wherein the camera control unit further includes a statistics module coupled to the camera to receive the video stream, the statistics module configured to create a brightness histogram for frames in the video stream. [Example 3] the camera control unit further includes an auto-exposure module coupled to the statistics module; the auto-exposure module is configured to maintain a target brightness of a displayed scene, the displayed scene being a scene within a frame captured by the camera; the auto-exposure module is configured to limit saturated pixels in scenes of captured frames to less than a predetermined number of pixels in each of the scenes; 3. The teleoperated surgical system of example 2, wherein the auto-exposure module is configured to maintain a minimum optical power output from the illumination device. [Example 4] The camera control unit includes: a first control loop configured to automatically adjust one or both of a video pipeline gain and a camera exposure time; and a second control loop configured to automatically set the output optical power and adjust the camera exposure time for subsequently captured frames. [Example 5] The first control loop is configured to adjust the gain of the video pipeline such that the displayed scene has a target brightness, the teleoperated surgery system described in Example 4. [Example 6] 5. The teleoperated surgical system of example 4, wherein a second control loop is configured to adjust the output optical power and the camera exposure time based on the value of the camera exposure time. [Example 7] 7. The teleoperated surgical system of embodiment 6, wherein a second control loop increases the output optical power and decreases the camera exposure time when the camera exposure time is greater than a first exposure threshold. [Example 8] 7. The teleoperated surgical system of example 6, wherein a second control loop decreases the output optical power and increases the camera exposure time when the camera exposure time is less than a second exposure threshold. [Example 9] The teleoperated surgical system of example 6, wherein a second control loop leaves the output optical power and the camera exposure time unchanged when the camera exposure time is between a first exposure threshold and a second exposure threshold. [Example 10] The remote surgical system of Example 4, wherein the camera control unit further includes a third control loop configured to automatically adjust a target brightness of the displayed scene to reduce the number of saturated pixels in subsequently captured frames. [Example 11] The remote surgical system of Example 10, wherein the third control loop is configured to apply a decrease width limit to the target brightness. [Example 12] The remote surgical system of Example 1, wherein the camera control unit further includes a control loop configured to automatically adjust a target brightness of the displayed scene to reduce the number of saturated pixels in subsequently captured frames. [Example 13] 13. The teleoperated surgical system of claim 12, wherein the control loop is configured to apply a decay limit to the target brightness. [Example 14] The teleoperated surgical system of Example 4, wherein the camera control unit further includes an auto-exposure module including a first control loop and a second control loop. [Example 15] The remote surgical system of Example 2, wherein the camera control unit further includes an auto-exposure module coupled to the statistics module to receive the brightness histogram of frames in the video stream, the auto-exposure module configured to determine an average brightness of the histogram, and the auto-exposure module configured to adjust one or both of a video pipeline gain and the camera exposure time based on a relationship of the average brightness to a target brightness. [Example 16] The remote surgical system of Example 15, wherein the automatic exposure module is further configured to instruct the lighting control device to change the first optical power output to the second optical power output, and to instruct the camera to adjust the exposure of the camera to compensate for the change in the output of the lighting device. [Example 17] 2. The teleoperated surgical system of claim 1, wherein the camera control unit is configured to attach metadata to each frame of the video stream, the metadata providing information for pipeline synchronization. [Example 18] the camera control unit is configured to detect contact between an endoscope and tissue; 2. The teleoperated surgical system of claim 1, wherein the camera control unit is configured to instruct the lighting control device to attenuate the output optical power after detecting a contact. [Example 19] The remote surgical system of Example 18, wherein the camera control unit further includes a statistics module coupled to the camera to receive a scene captured by the camera, the statistics module configured to determine an overall brightness of the captured scene. [Example 20] The remote surgical system of Example 19, wherein the camera control unit is coupled to the statistics module to receive the overall brightness of the captured scene, coupled to receive a camera exposure time, and further includes a contact detection module coupled to the lighting control device, the contact detection module configured to detect contact between the tip of the endoscope and tissue. [Example 21] 21. The remote surgical system of example 20, wherein the contact detection module is further configured to monitor reflected luminance and monitor the optical power output from the illumination device. [Example 22] The remote surgical system of Example 20, wherein the contact detection module is further configured to determine that the tip of the endoscope is in contact with the tissue when a change from the first optical power output to the second optical power output does not result in a change in the reflected brightness of the second optical power output compared to the reflected brightness of the first optical power output. [Example 23] The remote surgical system of Example 20, wherein the lighting control device further includes a dithering module coupled to the contact detection module, the dithering module being configured to vary the optical power output in a known pattern after being enabled by the contact detection module. [Example 24] The teleoperated surgical system of example 3 further comprising a high power mode switch coupled to the automatic exposure module. [Example 25] A teleoperated surgical system, comprising: A control device including a camera control unit and a lighting control device, the illumination controller configured to control optical power output from an illumination device of the teleoperated surgical system in response to commands from the camera control unit; the camera control unit is configured to receive a video stream from a camera of the teleoperated surgical system, the camera control unit being coupled to the lighting control device; the camera control unit is configured to instruct the lighting controller to change an optical power output from a first optical power output to a second optical power output, and to instruct the camera to change a camera exposure time from a first exposure time to a second exposure time, whereby a frame is subsequently captured by the camera at a second exposure time and with reflected light, the reflected light being light from the lighting device having the second optical power output before being reflected. Remotely operated surgery system. [Example 26] A teleoperated surgical system, comprising: an illumination device having an optical power output; a camera configured to capture a scene including the tissue; a control device coupled to the lighting device and the camera, the controller is configured to detect contact between the endoscope and the tissue; the controller is configured to attenuate the output optical power after detecting a contact. Remotely operated surgery system. [Example 27] 1. A method, comprising: detecting, by a controller, contact between the tip of the endoscope and tissue; and attenuating, by the controller, optical power output from the tip of the endoscope when the controller detects contact. method. [Example 28] 1. A method of operating a teleoperated surgical system, the method comprising: adjusting one or both of a video pipeline gain and a camera exposure time using the average luminance of the captured scene and a target luminance of the displayed scene; and using the camera exposure time to set the output optical power of an illumination device and an exposure time of a camera. How it works. [Example 29] 29. The method of claim 28, further comprising the step of reducing the number of saturated pixels in the second captured frame by decreasing the target luminance.

Claims

1. 1. A system including a controller coupled to a lighting device and a camera, The control device includes: adjusting one or both of a gain of a video pipeline of the camera and a camera exposure time based on an average luminance of frames of the scene captured by the camera and a target luminance of the scene to be displayed; and and controlling the output optical power of the illumination device and the camera exposure time based on the camera exposure time. system.

2. Controlling the output optical power and the camera exposure time based on the camera exposure time includes: increasing the output optical power and decreasing the camera exposure time if the camera exposure time is greater than a first exposure threshold; decreasing the output optical power and increasing the camera exposure time if the camera exposure time is less than a second exposure threshold; and 2. The system of claim 1, further comprising: leaving the output optical power and the camera exposure time unchanged if the camera exposure time is between the first exposure threshold and the second exposure threshold.

3. The control device includes: generating a luminance histogram for the frame of the scene; and The system of claim 1 , further configured to determine the average luminance of the frames of the scene based on the luminance histograms of the frames of the scene.

4. 4. The system of claim 3, wherein adjusting one or both of the video pipeline gain and the camera exposure time is based on a relationship of the average luminance of the frames of the displayed scene to the target luminance of the scene.

5. The system of claim 1 , wherein the controller is further configured to adjust the target luminance of the displayed scene to reduce a number of saturated pixels in subsequently captured frames.

6. 6. The system of claim 5, wherein adjusting the target luminance of the displayed scene to reduce the number of saturated pixels in subsequently captured frames includes applying a decrease limit to the target luminance.

7. 7. The system of claim 1, wherein the controller is further configured to maintain a minimum optical power output from the illumination device.

8. The system of claim 1 , wherein the controller is configured to adjust one or both of a gain of the video pipeline and the camera exposure time such that the displayed scene has the target brightness.

9. 1. A method, comprising: a control device coupled to the lighting device and the camera adjusting one or both of a gain of a video pipeline of the camera and a camera exposure time based on an average luminance of frames of the scene captured by the camera and a target luminance of the scene to be displayed; and the controller controlling the output optical power of the illumination device and the camera exposure time based on the camera exposure time. method.

10. controlling the output optical power and the camera exposure time based on the camera exposure time, increasing the output optical power and decreasing the camera exposure time if the camera exposure time is greater than a first exposure threshold; if the camera exposure time is less than a second exposure threshold, decreasing the output optical power and increasing the camera exposure time; 10. The method of claim 9, further comprising: if the camera exposure time is between the first exposure threshold and the second exposure threshold, then leaving the output optical power and the camera exposure time unchanged.

11. the controller generating a luminance histogram for the frame of the scene; The method of claim 9 , further comprising the controller determining the average luminance of the frames of the scene based on the luminance histograms of the frames of the scene.

12. 12. The method of claim 11, wherein adjusting one or both of the video pipeline gain and the camera exposure time is based on a relationship of the average luminance of the frames of the displayed scene to the target luminance of the scene.

13. 10. The method of claim 9, further comprising the step of the controller adjusting the target luminance of the displayed scene to reduce a number of saturated pixels in subsequently captured frames.

14. 14. The method of claim 13, wherein adjusting the target luminance of the displayed scene to reduce the number of saturated pixels in subsequently captured frames comprises applying a decrease limit to the target luminance.

15. 15. The method of claim 9, further comprising the step of the controller maintaining a minimum optical power output from the illumination device.

Images