Dynamic aperture positioning for stereoscopic endoscope camera

Abstract

The invention relates to dynamic aperture positioning for stereoscopic endoscope cameras. The techniques described herein may be embodied in stereoscopic endoscopes where the orientation of the two eyes may be dynamically determined as the endoscope is moved and rotated. This can achieve an improved user experience when an angled stereoscopic endoscope is used even when it is rotated (e.g., to view a side of a cavity).

Description

[0001] This application is a divisional application of Chinese patent application 2019800367596 (PCT / US2019 / 024608) filed on March 28, 2019 and entitled “Dynamic Aperture Positioning for Stereoscopic Endoscopic Cameras”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Application No. 15 / 956,655, filed on April 18, 2018, entitled “Dynamic Aperture Positioning for Stereo Endoscopic Cameras,” which claims the benefit of U.S. Provisional Application No. 62 / 652,826, filed on April 4, 2018, entitled “Dynamic Aperture Positioning for Stereo Endoscopic Cameras.” The disclosures of both of the above applications are incorporated herein by reference in their entirety. Technical Field

[0004] The present disclosure relates to a three-dimensional (3D) endoscope that may be used in a surgical system. Background Art

[0005] Minimally invasive surgical systems are being developed to reduce the trauma suffered by patients undergoing surgical interventions. These systems require only very small incisions, and the surgeon uses a stick-like object and instruments like a camera to perform the operation. In addition to reducing trauma, this type of remote operation system increases the surgeon's flexibility and allows the surgeon to operate on the patient from a remote location. Telesurgery is a general term for surgical systems in which the surgeon uses some form of remote control (e.g., servomechanism, etc.) to manipulate the movement of surgical instruments instead of directly holding and moving the instruments by hand. In such a remote surgical system, an image of the surgical site is provided to the surgeon through a display device. Based on the visual feedback received through the display device, the surgeon performs surgery on the patient by manipulating a master input device, which in turn controls the movement of the remote robotic instrument. Summary of the invention

[0006] The technology described herein describes a stereoscopic vision camera with a controllable pupil position. The controllable pupil position allows stereoscopic vision views adapted to axial rotation of an endoscope with angled views. Methods are described on how to control the position of the pupil to create natural stereoscopic views during rotation of the scope or movement of the observer's head. The device is an electronic device and can be controlled by software.

[0007] In one aspect, the document features a method for generating a view of a scene, the method comprising determining, by one or more processing devices, an orientation angle defined by a line connecting a first aperture position and a second aperture position of a stereoscopic vision endoscope relative to a reference orientation. The method also includes adjusting at least one of the first aperture position and the second aperture position while maintaining a spacing between the first aperture position and the second aperture position to maintain the orientation angle across multiple endoscope orientations. The method also includes creating an aperture at each of the first aperture position and the second aperture position, and generating a representation of the view using a signal based on light captured through the aperture created at the first aperture position and the second aperture position for presentation on a display device associated with the stereoscopic vision endoscope.

[0008] On the other hand, the present document describes a system comprising a stereoscopic endoscope, a display device, and one or more processing devices. The stereoscopic endoscope includes at least one image sensor for sensing a first image and a second image of a pair of stereoscopic images. The first image and the second image are sensed based on light passing through apertures at first aperture positions and second aperture positions respectively electronically defined on a liquid crystal layer within the endoscope. One or more processing devices are configured to determine an orientation angle defined by a line connecting the first aperture position and the second aperture position of the stereoscopic endoscope relative to a reference orientation, adjust at least one of the first aperture position and the second aperture position, while maintaining the interval between the first aperture position and the second aperture position to maintain the orientation angle across multiple endoscope orientations, and create an aperture at each position of the first aperture position and the second aperture position. One or more processing devices are also configured to generate a representation of a view of a surgical scene using a pair of stereoscopic images captured by the apertures created at the first aperture position and the second aperture position. The display device is configured to present a representation of the view.

[0009] In another aspect, the present document describes one or more machine-readable storage devices having encoded thereon computer-readable instructions for causing one or more processing devices to perform various operations. The operations include determining an orientation angle defined by a line connecting a first aperture position and a second aperture position of a stereoscopic endoscope relative to a reference orientation; and adjusting at least one of the first aperture position and the second aperture position while maintaining a spacing between the first aperture position and the second aperture position to maintain the orientation angle across multiple endoscope orientations. The operations also include creating an aperture at each of the first aperture position and the second aperture position, and generating a representation of a view using a signal based on light captured through the aperture created at the first aperture position and the second aperture position for presentation on a display device associated with the stereoscopic endoscope.

[0010] Embodiments of the above aspects may include one or more of the following features. The reference orientation may be perpendicular to the direction of gravity of the earth. Information indicating the orientation of the head of a user operating the stereoscopic vision endoscope may be received, and the reference orientation may be determined based on the information indicating the orientation of the head of the user. Adjusting at least one of the first aperture position and the second aperture position may include selecting the position of a pair of liquid crystal display (LCD) segments from a group of LCD segments arranged in a substantially annular configuration in the optical path of the stereoscopic vision endoscope. Creating an aperture at each of the first aperture position and the second aperture position may include controlling a first LCD segment in a pair of LCD segments so that the first LCD segment changes to a state in which the first LCD segment allows more light to pass through compared to a different, relatively dark state, and controlling a second LCD segment in a pair of LCD segments so that the second LCD segment changes to a state in which the second LCD segment allows more light to pass through compared to a different, relatively dark state. The first LCD segment and the second LCD segment may be controlled to acquire a first image and a second image, respectively, substantially simultaneously. Light passing through the first LCD segment may pass through a first polarizer, and light passing through the second LCD segment may pass through a second polarizer, the second polarizer polarizing light differently from the first polarizer. The first polarizer may be orthogonal to the second polarizer. Light passing through an aperture created at a first aperture position and a second aperture position can be sensed using a first sensor and a second sensor, respectively, the first sensor and the second sensor being disposed on two opposing sides of a polarizing beam splitter. Apertures can be created at the first aperture position and the second aperture position in a sequential pattern. Light passing through the apertures created at the first aperture position and the second aperture position can be sensed using a single sensor. A representation of the view can be presented on a display device. In response to presenting a representation of the view on the display device, user input related to operating a surgical device at a surgical scene can be received.

[0011] In another aspect, the document features a stereoscopic vision endoscope that includes at least one image sensor for sensing a first image and a second image of a pair of stereoscopic images. The first image is sensed based on light passing through a first aperture within the endoscope, and the second image is sensed based on light passing through a second aperture within the endoscope. A liquid crystal layer disposed between two layers of glass includes a first electrode arrangement such that each of the first aperture and the second aperture is created in the liquid crystal layer using a portion of the first electrode arrangement.

[0012] Embodiments may include one or more of the following features. Light passing through the first aperture may be polarized differently than light passing through the second aperture. The stereoscopic vision endoscope may include a first image sensor, a second image sensor, and an optical element that directs incident light to the first image sensor or the second image sensor based on the polarization state of the incident light. In response to a corresponding control signal provided by a first electrode arrangement, each of the first aperture and the second aperture may be controlled to be located at a respective position on the liquid crystal layer. The positions of the first aperture and the second aperture may be controlled according to the orientation of the endoscope relative to a reference orientation. The stereoscopic vision endoscope may include a first image sensor and a second image sensor, wherein the first image and the second image are sensed by the first image sensor and the second image sensor, respectively, substantially simultaneously. The first image and the second image may be sensed sequentially by a single image sensor. The stereoscopic vision endoscope may include a first portion that accommodates a front end lens assembly and a second portion that includes an elongated shaft that accommodates a liquid crystal layer and at least one image sensor. The second portion may be arranged at an angle relative to the first portion. The angle may be one of: 0°, 30°, and 45°. The position of at least one of the first aperture and the second aperture may be electronically adjusted using a first electrode arrangement to maintain an angle between (i) a line connecting the first aperture and the second aperture and (ii) a reference orientation. The angle between (i) a line connecting the first aperture and the second aperture and (ii) a reference orientation may be maintained while also maintaining a spacing between the first aperture and the second aperture. The angle between (i) a line connecting the first aperture and the second aperture and (ii) a reference orientation may be maintained using one or more control signals calculated based on one or more previously captured images.

[0013] Some or all of the embodiments described herein may provide one or more of the following advantages. By providing a split pupil endoscopic camera, in which a liquid crystal display element is used to control the pupil position, an electronically controllable imaging device with few (if any) mechanical moving parts can be implemented in a space-constrained and / or resource-constrained endoscopic environment. The pupil position can be controlled based on information associated with the orientation of the surgeon's head. Other control inputs may also be used, and other control inputs may facilitate certain image processing requirements. In this way, even if the camera is rotated during operation of the endoscope, the two pupils can be kept separated by a predetermined distance along a reference direction. For example, the separation direction of the pupils relative to gravity can be controlled at will. Maintaining such a predetermined distance between the stereoscopic pupils can allow accurate 3D representations to be displayed for different orientations of the endoscope. In some cases, such a representation may be consistent with the natural way in which the surgeon views the corresponding surgical scene, and therefore may help improve the surgeon's user experience. If desired, the pupil separation distance (or inter-pupillary distance) may also be controlled. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a perspective view of an exemplary patient side cart for a computer-assisted, teleoperated surgical system.

[0015] Figure 2 is a front view of an exemplary surgeon's console for a computer-assisted, teleoperated surgical system.

[0016] Figure 3 is a side view of an exemplary robotic manipulator arm assembly of a computer-assisted, teleoperated surgical system.

[0017] Figure 4 is a schematic diagram illustrating the optical path through the example of a 30° stereoscopic vision endoscope.

[0018] Figure 5A and Figure 5B is a schematic diagram of an example of a polarizer that may be used with the techniques described herein.

[0019] Fig. 6A and Figure 6B is an example of an optical mask that can be used with the techniques described herein.

[0020] Fig. 7A and Figure 7B is an example arrangement of electrodes that may be used in a first portion of a multi-pupil device in accordance with one implementation of the technology described herein.

[0021] Fig. 8A is available in Fig. 7A An exemplary arrangement of electrodes used in the second part of the multi-pupil device.

[0022] Figure 8B is to use Fig. 7A , Figure 7B and Fig. 8A Examples of rotator segments created by combinations of electrode arrangements.

[0023] Figure 8C is with Fig. 8A The electrodes in the Figure 8B Examples of electrodes with polarization control patterns are shown.

[0024] Fig. 9A is available in Fig. 7A Another exemplary arrangement of electrodes used in the second part of the multi-pupil device.

[0025] Fig. 9B is to use Fig. 7A , Figure 7B and Fig. 9A Examples of rotator segments created by combinations of electrode arrangements.

[0026] Fig. 9C is used in conjunction with those electrodes in 9A to create Fig. 9B Examples of electrodes with polarization control patterns are shown.

[0027] Fig. 10A and Fig. 10B Examples of pupil positions for two different orientations for stereoscopic endoscopes.

[0028] Fig. 10C and Fig. 10D Examples of different configurations and positions of the pupil are illustrated.

[0029] Fig.11 is a schematic diagram illustrating the optical path through the example of a 0° stereoscopic vision endoscope with two sensors.

[0030] Fig. 12A is a schematic diagram illustrating an example optical path through a 0° stereoscopic vision endoscope that uses one sensor for sensing two stereoscopic vision images substantially simultaneously.

[0031] Fig. 12B is a schematic diagram illustrating the optical path through the example of a 30° stereoscopic vision endoscope, which uses one sensor for sensing two stereoscopic vision images.

[0032] Fig.13 is a schematic diagram illustrating a process of sensing two stereoscopic vision images using a single sensor with a rolling shutter.

[0033] Fig.14is a flow chart of an exemplary process for generating a 3D representation using the techniques described herein. DETAILED DESCRIPTION

[0034] This document describes a technique that helps to automatically correct the position of the camera aperture in a stereoscopic vision endoscope so that the aperture position remains substantially fixed relative to a reference frame even when the endoscope is oriented at different angles during surgery. In some embodiments, this can allow a more natural perception of a 3D representation of a surgical scene presented by a stereoscopic image on the surgeon's console. For example, even when the endoscope is oriented at an arbitrary angle, the surgeon can more naturally visualize the surgical scene (e.g., without having to tilt or reorient her head). In some cases, this can improve the surgeon's overall user experience during surgery. By allowing the aperture position to be electronically controlled in a manner that may have few or no moving mechanical parts, the techniques described herein facilitate implementations suitable for space-constrained and / or resource-constrained environments for stereoscopic vision endoscopes. The endoscope herein can be a rigid device that combines optics and one or more image sensors to create a camera system; a flexible device with a wrist; a flexible device with a camera at the distal end; an optical endoscope with a camera (sometimes called a Hopkins endoscope) or a similar device.

[0035] The company primarily uses the da Various aspects of the technology are described in terms of embodiments of surgical systems. An example of such a surgical system is a Xi TM Surgical System (Model IS4000). It should be understood that the various aspects disclosed herein can be embodied and implemented in a variety of ways, including computer-assisted, non-computer-assisted, and hybrid combinations of manual and computer-assisted embodiments and implementations. Implementations on surgical systems (e.g., model IS4000 described for illustrative purposes) are considered to limit the scope of the inventive aspects disclosed herein. When applicable, various inventive aspects can be embodied and implemented in relatively small handheld, manual devices and relatively large systems with additional mechanical support, as well as in other embodiments of computer-assisted remotely operated medical devices. Although the technology is described primarily with reference to examples of peer displays, the technology can also be used for other types of wearable or non-wearable display devices, such as head-mounted display devices for virtual or augmented reality (VR / AR) systems. The captured images can also be displayed on large displays such as 3D TV devices, projected onto a screen or viewed by a user wearing stereo glasses. Alternatively, an autostereoscopic type display can be used. Examples of autostereoscopic devices include columnar liquid crystal displays (LCDs), which can also include head and / or eye tracking of the observer (user).

[0036] Reference Figure 1 and Figure 2 , a system for minimally invasive computer-assisted telesurgery (also referred to as MIS) may include a patient-side cart 100 and a surgeon's console 50. Telesurgery is a general term for surgical systems in which the surgeon uses some form of remote control, such as a servomechanism, etc., to manipulate the movement of surgical instruments rather than directly grasping and moving the instruments by hand. Robotically manipulable surgical instruments may be inserted into small, minimally invasive surgical apertures to treat tissue at a surgical site within a patient's body, thereby avoiding the trauma associated with larger incisions required for open surgery. These robotic systems can move the working ends of surgical instruments with sufficient flexibility to perform fairly complex surgical tasks, typically by pivoting the shaft of the instrument at a minimally invasive aperture, sliding the shaft axially through the aperture, rotating the shaft within the aperture, and the like.

[0037] In the depicted embodiment, the patient side cart 100 includes a base 110, a first robotic manipulator arm assembly 120, a second robotic manipulator arm assembly 130, a third robotic manipulator arm assembly 140, and a fourth robotic manipulator arm assembly 150. Each robotic manipulator arm assembly 120, 130, 140, and 150 is pivotally coupled to the base 110. In some embodiments, less than four or more than four robotic manipulator arm assemblies may be included as part of the patient side cart 100. Although in the depicted embodiment, the base 110 includes casters to allow for easy movement, in some embodiments, the patient side cart 100 is fixedly mounted to a floor, ceiling, operating table, structural frame, etc.

[0038] In a typical application, two of the robotic manipulator arm assemblies 120, 130, 140, or 150 hold surgical instruments, while the third holds a stereo endoscope. The remaining robotic manipulator arm assembly is available so that a third instrument can be introduced at the work site. Alternatively, the remaining robotic manipulator arm assembly can be used to introduce a second endoscope or another image capture device (such as an ultrasound transducer) to the work site.

[0039] Each robotic manipulator arm assembly 120, 130, 140 and 150 is generally formed of links coupled together and manipulated by actuatable joints. Each robotic manipulator arm assembly 120, 130, 140 and 150 includes a mounting arm and a device manipulator. The mounting arm positions the device it holds so that a pivot point occurs at its entry aperture into the patient. The device manipulator can then manipulate the device it holds so that the held device can pivot about the pivot point, be inserted into and retracted from the entry aperture, and rotate about the axis of its shaft.

[0040] In the depicted embodiment, the surgeon console 50 includes a stereoscopic peer display 45 so that the user can view the surgical work site in stereoscopic vision from images captured by a stereoscopic vision camera used in conjunction with the patient side cart 100. Left eyepiece 46 and right eyepiece 47 are provided in the stereoscopic peer display 45 so that the user can view the left and right display screens within the display 45 with the user's left and right eyes, respectively. While viewing the image of the surgical site, typically on a suitable viewer or display, the surgeon performs surgery on the patient by manipulating a master control input device, which in turn controls the movement of the robotic instrument.

[0041] The surgeon console 50 also includes a left input device 41 and a right input device 42, which the user can grasp with his / her left and right hands, respectively, to manipulate devices (e.g., surgical instruments) held by the robotic manipulator arm assemblies 120, 130, 140, and 150 of the patient side cart 100, preferably with six or more degrees of freedom (“DOF”). A foot pedal 44 with toe and heel controls is provided on the surgeon console 50 so that the user can control the movement and / or actuation of the device associated with the foot pedal.

[0042] For control and other purposes, a processing device 43 is provided in the surgeon console 50. The processing device 43 performs various functions in the medical robotic system. One function performed by the processing device 43 is to translate and transmit the mechanical motion of the input devices 41, 42 to actuate their respective joints in their associated robotic manipulator arm assemblies 120, 130, 140 and 150 so that the surgeon can effectively manipulate the device, such as a surgical instrument. Another function of the processing device 43 is to implement the methods, cross-coupled control logic and controllers described herein.

[0043] The processing device 43 may include one or more processors, digital signal processors (DSPs), field programmable gate arrays (FPGAs), and / or microcontrollers, and may be implemented as a combination of hardware, software, and / or firmware. Moreover, its functions described herein may be performed by one unit or divided among multiple subunits, each of which may in turn be implemented by any combination of hardware, software, and firmware. In addition, although shown as part of or physically adjacent to the surgeon's console 50, the processing device 43 may also be distributed throughout the telesurgical system as subunits. One or more of the subunits may be physically remote from the telesurgical system (e.g., located on a remote server).

[0044] Also refer to Figure 3 , the robotic manipulator arm assemblies 120, 130, 140, and 150 can manipulate devices such as endoscopic stereo cameras and surgical instruments to perform minimally invasive surgery. For example, in the depicted arrangement, the robotic manipulator arm assembly 120 is pivotally coupled to the instrument holder 122. The cannula 180 and the surgical instrument 200 are in turn releasably coupled to the instrument holder 122. The cannula 180 is a hollow tubular member that is located at the patient interface during surgery. The cannula 180 defines a lumen in which an elongated shaft 220 of an endoscopic camera (or endoscope) or a surgical instrument 200 is slidably disposed. As further described below, in some embodiments, the cannula 180 includes a distal portion having a body wall retractor member. The instrument holder 122 is pivotally coupled to the distal end of the robotic manipulator arm assembly 120. In some embodiments, the pivotable coupling between the instrument holder 122 and the distal end of the robotic manipulator arm assembly 120 is a motorized joint actuatable from the surgeon console 50 using the processing device 43 .

[0045] The instrument holder 122 includes an instrument holder frame 124, a sleeve clamp 126, and an instrument holder bracket 128. In the depicted embodiment, the sleeve clamp 126 is fixed to the distal end of the instrument holder frame 124. The sleeve clamp 126 can be actuated to couple with the sleeve 180 or uncouple from the sleeve 180. The instrument holder bracket 128 is movably coupled to the instrument holder frame 124. More specifically, the instrument holder bracket 128 can be linearly translated along the instrument holder frame 124. In some embodiments, the movement of the instrument holder bracket 128 along the instrument holder frame 124 is a motorized translational movement that can be actuated / controlled by the processing device 43. The surgical instrument 200 includes a transmission assembly 210, an elongated shaft 220, and an end effector 230. The transmission assembly 210 can be releasably coupled to the instrument holder bracket 128. The shaft 220 extends distally from the transmission assembly 210. An end effector ( 230 ) is disposed at the distal end of the shaft ( 220 ).

[0046] The shaft 220 defines a longitudinal axis 222 that coincides with the longitudinal axis of the cannula 180. As the instrument holder cradle 128 translates along the instrument holder frame 124, the elongated shaft 220 of the surgical instrument 200 moves along the longitudinal axis 222. In this manner, the end effector 230 may be inserted and / or retracted from a surgical workspace within a patient.

[0047] Laparoscopic surgery may require the surgeon to visualize the surgical site with an endoscope and perform fine motor manipulation with laparoscopic instruments to perform exploration, dissection, suturing, and other surgical tasks. These tasks often require fine two-handed interaction with tissue. In some cases, such two-handed motor tasks can often be performed more easily when the surgeon is presented with a 3D view of the surgical scene. The surgical workspace (surgical scene) within the patient's body can be presented as a 3D visualization to the surgeon via a stereoscopic display 45. Although the technology described herein primarily uses the example of a peer-to-peer stereoscopic display, other types of stereoscopic displays and non-stereoscopic displays are also within the scope of the technology. A peer-to-peer stereoscopic display refers to a display that allows a user to view the display without having to wear it or share it with another user at the same time. A stereo microscope can be an example of a peer-to-peer stereoscopic device. As Figure 2 As shown, stereoscopic display 45 is another example of a peer-to-peer stereoscopic display.

[0048] In some embodiments, the peer stereoscopic display 45 may include two display screens (one for each eye), each display screen displaying one of two images corresponding to a stereoscopic pair of images captured using a stereoscopic endoscope. When these stereoscopic pair images are displayed, these stereoscopic pair images allow the user to perceive depth in the displayed images. The stereoscopic pair images can be captured in various ways, including, for example, using two spatially separated cameras in a "dual camera" approach, collecting light fields in a plenoptic camera approach, or as described herein, using a dual pupil type device, where two spatially separated apertures (also called pupils) enable two slightly different images to be captured from two slightly different angles.

[0049] In a dual-camera approach, two separate cameras and associated optics may be oriented so that they capture essentially the same scene from two locations separated by a "pupillary distance." This approach mimics human vision, in which two spatially separated eyes are used to capture a 3D view of the world. The orientation of the camera pair at the time of acquisition determines the orientation at which the image can be reproduced. For example, the cameras may be separated along a horizontal axis so that views corresponding to a straight (or "normal") head or an inverted head may be captured. However, for arbitrary orientation angles of the two cameras, the horizontal distance between the cameras changes, making 3D rendering challenging. A camera pair with cameras facing the horizon, separated in a plane perpendicular to gravity, has fixed pupils and captures one image from above the other when rotated 90 degrees; this is not always the desired behavior.

[0050] In another approach, a 3D scene can be captured by sampling a light field, for example, using a plenoptic camera that captures both the intensity and direction of some of the many rays that pass through an optical system. A plenoptic camera can include, for example, a camera array, a movable camera, or a camera containing an array of lenses for sampling the light field at some spatial and angular resolution. In this approach, a synthetic camera view after acquisition can be reconstructed, where the depth of field can be manipulated to control the focus of different parts of the scene. Multi-view images can also be created in which the camera appears to have moved a small amount relative to the scene, thereby showing the depth in the acquired scene. These plenoptic systems can simulate two views of a stereo pair by selecting appropriate ray pens. However, typically the resolution is greatly reduced in such systems, which is undesirable because the information from most pixels is not used, thereby wasting these pixels.

[0051] In some embodiments of a dual pupil device, a single optical system can be used for both eyes. In some cases, a mechanical aperture can be moved from one position to another (e.g., from right to left) to allow light to pass through two positions sequentially. If this operation is done at a sufficiently high rate (e.g., 120 Hz), a smooth stereoscopic image sequence can be acquired using a suitable high-speed camera. However, in a stereoscopic endoscope, limitations in space and power resources can limit mechanical motion, so it may be challenging to implement such a mechanical aperture. The technology described herein provides an electronically controllable aperture that substantially avoids the disadvantages of a mechanical aperture to achieve a camera system suitable for incorporation into an endoscope. In addition, by adjusting the position of the aperture based on the orientation of the endoscope, the technology facilitates 3D visualization that can be perceived as natural, thereby greatly improving the user experience of the surgeon during the surgical procedure. Using the technology described herein, an image sequence is acquired using an electronic shutter that can be synchronized with a display (e.g., based on activating / deactivating a liquid crystal display (LCD) segment to pass / block light in the light path). This may result in the left eye observing a view through one pupil and the right eye observing a view through another pupil. Such a split-pupil camera can therefore represent a simple light field camera where only two views are sampled and each view is at full resolution with no wasted pixels.

[0052] In some embodiments, a stereo endoscope can include optics configured to look directly into the end of a tube or shaft that houses a portion of the endoscope. Such an endoscope may be referred to as a 0° endoscope. However, in some cases, a surgeon may also need an endoscope that looks "down" at his or her working area. For example, in laparoscopic surgery, the view is typically oriented at 30° relative to the axis of the endoscope. Endoscopes with other orientations (e.g., endoscopes used in arthroscopy) are also possible; viewing angles (e.g., 45 and 70 degrees) are also common.

[0053] Figure 4 and Fig.11 Examples of 30° endoscope and 0° endoscope are described respectively. Specifically, Figure 4 is a schematic diagram illustrating the optical path of the present invention through a schematically simplified 30° stereoscopic vision endoscope 400, Fig.11 is a schematic diagram illustrating the optical path through an example of a 0° stereoscopic vision endoscope 1100. Figure 4, the endoscope 400 includes an optical arrangement 401 of a plurality of lenses and other optical elements that direct light from a target imaging region 403 onto two image sensors 402 and 404 that are configured to sense right and left views, respectively. In some embodiments, the optical arrangement 401 includes a beam splitter 406 that allows a set of rays 408 to pass through a surface 410 and reflects a set of rays 412 toward the image sensor 404. The surface 410 can be configured to reflect rays of one polarization state and allow rays of a different polarization state to pass through. For example, the surface 410 can include a wire grid layer (e.g., a wire grid layer developed by Moxtek Inc. of Utah, USA, or a wire grid layer developed by Meadowlark Optics of Colorado, USA) that separates different linear polarization states by reflection and transmission. In some embodiments, this facilitates substantially simultaneous acquisition of the left and right images of a stereoscopic endoscope in a compact form factor that avoids the use of larger diameter optics that may be required for separate physical optical channels for the left and right images. One of the advantages of the technology described herein is that the left eye and the right eye are matched in performance because they use the same optical path. For example, if the optics 450 are moved longitudinally to focus the image on sensors 402 and 404, the left eye and the right eye will inherently match in terms of their respective focus curves. In addition, aberrations due to tolerances in the optical system will also match between the images on the two sensors. This is beneficial for maintaining stereoscopic image quality.

[0054] In some embodiments, the front end of the optical arrangement 401 may include a sapphire window 414, a front negative lens 416, a first portion 418 of a 30° prism, and a second portion 420 of a 30° prism. The 30° prism may include an air gap at an interface between the first portion 418 and the second portion 420, such that light reflected from a surface of the second portion 420 is totally internally reflected at the interface between the first portion 418 and the second portion 420, as shown in FIG. Figure 4 Some embodiments may not include the sapphire window 414 that is provided to protect other parts of the optical arrangement 401.

[0055] In some embodiments, the optical arrangement 401 includes two separate pupils or apertures defined to allow light corresponding to the right view and the left view to pass through, respectively. The two apertures can be defined by a combination of multiple optical elements (e.g., liquid crystal layers, electrodes, and polarizers) so that the apertures can be electronically generated and blocked, thereby allowing dynamic control of the position of the apertures. In some embodiments, the apertures can be defined using a first polarizer 422, a first liquid crystal layer 424, a second polarizer 426 orthogonal to the first polarizer 422, and a second liquid crystal layer 428. The first liquid crystal layer 424 can be excited using a transparent electrode disposed in a glass layer 430 adjacent to the liquid crystal layer 424 and a complementary electrode disposed in a glass layer 431 on the opposite side of the liquid crystal layer 424 compared to the glass layer 430. Similarly, the second liquid crystal layer 428 can be excited using a transparent electrode disposed in a glass layer 432 adjacent to the liquid crystal layer 428 and a complementary electrode disposed in a glass layer 433 on the opposite side of the liquid crystal layer 428 compared to the glass layer 432. Segments formed in the liquid crystal layer 424 (as described in more detail below) can be excited to form a right pupil or aperture 434 that allows light corresponding to a right view to pass through. Segments formed in the liquid crystal layer 428 can be excited to define another region 436 that polarizes light from the pupil 434 in a manner that transmits light through as light 408 to the surface 410 of the polarizing beam splitter 406. Segments formed in the liquid crystal layer 424 can also be excited to form a left pupil or aperture 438 that allows light corresponding to a left view to pass through. Accordingly, segments formed in the liquid crystal layer 428 can be excited to define another region 440 that polarizes light from the pupil 438 in a manner that reflects light from the surface 410 of the polarizing beam splitter 406 toward the image sensor 404 as light 412. Unwanted light (e.g., light that does not pass through the desired aperture) can be absorbed by the mask or second polarizer 426.

[0056] In some embodiments, liquid crystal element 428 rotates light passing through pupils 434 and 438 so that light passing through rotator pupil 436 is orthogonally polarized relative to light passing through rotator pupil 440. In some embodiments, liquid crystal layer 424 can be segmented radially around an annular region, and one or more segments can be used to define a pupil region. Light emitted from the two pupils defined on liquid crystal layer 424 has the same polarization and will pass through polarizer 426. Light not in the pupil region is not rotated and is therefore affected by the extinction rate of the two crossed polarizers 422 and 426. An exemplary schematic (or symbol) of such a polarizer is shown in Figure 5A and Figure 5B In some cases, they can be reversed (possibly at the expense of changing contrast) and polarizers 422 and 426 can be made parallel to each other and the unwanted light rotated.

[0057] Reference Figure 5A In some embodiments, the orientation of polarizer 422 can be configured based on the light source illuminating the surgical scene. For example, if the light source illuminating the surgical scene is polarized, the polarization direction and orientation of polarizer 422 can be configured to be orthogonal to the polarization direction and orientation of the light source, for example, to reduce the significance of specular reflections from the tissue surface. In some embodiments, polarizer 422 can be combined with a quarter wave plate along with a corresponding quarter wave plate on the illumination source to provide orthogonality of circular polarization between the illumination and imaging paths. In some cases, polarizers 422 and / or 426 can include or be enhanced with compensation films in a manner similar to the compensation films used on LCD displays to improve angular performance, thereby improving the contrast of the system by improving the transmittance and extinction of the liquid crystal pupil forming component.

[0058] Reference Figure 5B , polarizer 426 can be configured to transmit light of appropriate polarization and corresponding to the pupil area defined using the liquid crystal layer. For example, polarizer 426 can be configured to transmit light that passes through an area that is appropriately excited using electrodes on glass layers 430 and 432. Light that passes through non-excited areas can be absorbed, for example, by a combination of polarizers 422 and 426 and one or more mask layers that are configured to absorb light. In some embodiments, the polarization orientation of polarizer 422 can be orthogonal to the polarization orientation of polarizer 426. However, other relative polarization orientations of the two polarizers are also possible - for example, the "transmit" and "blocking" regions can be reversed and / or the polarizers can be parallel rather than crossed, but these configurations are generally not optimal. In some embodiments, the orientation can depend on the degree of retardation imparted by the liquid crystal layer 424.

[0059] In some embodiments and preferably, a black mask may be provided within the optical arrangement 401 to restrict light from passing through areas that do not define the pupil by the use of the device. An example of such a mask 600 is shown in FIG. Fig. 6A In some embodiments, mask 600 can be formed by metal deposition, a thin metal mask, formed by a semiconductor-like process, pad printing, or otherwise formed on a glass surface or polarizer 422. In some embodiments, mask 600 can be implemented as a separate film (e.g., metal or polyester film) or a glass plate. Fig. 6AIn the example shown, region 605 is a transparent region, and region 603 is opaque or minimally transmissive. In this example, region 607 is also opaque or minimally transmissive, depending on the nature of the pupil. In some embodiments, the diameter of region 607 may vary based on the pupil design, and may be zero in some cases. In some embodiments, region 603 may be implemented on one layer (e.g., on polarizer 422, or as a separate layer), and region 607 may be implemented on a subsequent layer (e.g., glass layer 430). These mask regions may be implemented in an opaque thin foil or by depositing chrome or similar material or by pad printing ink or other or substantially black material in a thin layer.

[0060] In some embodiments, a mask may also be located adjacent to the rotator pupils 436 and 440. In some embodiments, a second mask adjacent to the rotator pupils may be used in conjunction with mask 600. An example of such a mask 620 is shown in FIG. Figure 6B Because there may be a glass layer 432 between the pupils 434, 438 defining the liquid crystal shutter and the subsequent rotator pupils 436, 440, the area of ​​the rotator pupils 436, 440 can be made larger (compared to the pupils 434, 438) to account for the angle of light passing through the pupils. Therefore, the mask 620 ( Figure 6B ) may have a transparent area greater than area 605 ( Fig. 6A ) to illustrate the light diffused between the two liquid crystal layers 424 and 428. The transparent region 625 is defined by dark regions 623 and 627. In some embodiments, region 627 may not exist or have a vanishing diameter. In some embodiments, a compensation film may be used to increase the contrast in the resulting image.

[0061] In some embodiments, instead of having two liquid crystal layers, the two apertures or pupils in a single layer can be polarized differently. For example, a polarizing element can be placed before each aperture in the optical path, at each aperture, or after each aperture so that the two polarizing elements for the two pupils are orthogonal to each other. In some embodiments, the two polarizing elements can be linear polarizers that are orthogonal to each other. In some embodiments, the polarizing element can include a circular polarizer or a color notch filter.

[0062] Fig. 7A and Figure 7B is an exemplary arrangement of transparent electrodes that can be used to control pupil position in liquid crystal layer 424. Specifically, Fig. 7A An exemplary arrangement of transparent conductive electrodes disposed on glass layer 430 is illustrated. The electrodes may be disposed on a surface of glass layer 430 that interfaces with liquid crystal layer 424 and may be configured to control LCD segments that form shutter pupils 434 and 438. Fig. 7A The example of shows a radial pattern of electrodes; adjacent electrodes 711 and 713 are two of a plurality of electrodes that control corresponding LCD segments defined in the liquid crystal layer 424. The adjacent electrodes are independent and electrically isolated from each other. The electrodes can be coupled to electronic circuitry (not shown) using, for example, electrical contacts disposed near the edge of the glass layer 430. The electronic circuitry can be configured to drive the electrodes to activate / deactivate or excite / deenergize corresponding LCD segments.

[0063] Figure 7B An exemplary arrangement of a transparent conductive electrode disposed in the glass layer 431 is illustrated. The electrode may be disposed on a surface of the glass layer 431 bonded to the liquid crystal layer 424 and may be bonded to the liquid crystal layer 424. Fig. 7A The electrodes shown are used in combination and can be configured to control the LCD segments that form shutter pupils 434 and 438. Figure 7B In the example shown, electrode 717 is connected to a central region 719 that engages the LCD segments in the liquid crystal layer 424. Electrode 717 can be connected to electronic circuitry configured to control activation / deactivation of the LCD segments. In some embodiments, at least a portion of the electronic circuitry can be disposed on the glass layer 431. The order of the electrodes can be reversed, for example, depending on design constraints and the impact of edge effects at the segmented electrodes. Also, in the particular example shown, the octagonal shapes of layers 430 and 431 are elongated in orthogonal directions to facilitate attachment of drive electronics to the transparent electrodes. Figure 4 and Fig.11 , the octagonal shape and differential sizes of layers 430 and 431 (symmetric circular portions are shown) are not shown because electronic requirements do not affect the optically relevant features emphasized in these figures.

[0064] Fig. 8A is an exemplary arrangement of electrodes that may be used to control rotator regions 436, 440. Transparent electrodes may be disposed on glass layer 432 to impart appropriate polarization to light passing through two pupils 434 and 438 formed in liquid crystal layer 424. Light passing through glass layer 432 is polarized by second polarizer 426. In some embodiments, glass layer 432 includes a radial pattern of electrodes such as electrodes 805 and 810. Adjacent electrodes are independent, electrically isolated from one another and coupled to electronic circuitry configured to control the operation of the electronic pupils. The electrodes are driven by electronics conductively attached to the periphery of the glass. In some embodiments, electrode segments (such as Fig. 7A4 (shown in FIG. 4 ), the electrode segments disposed on the glass layer 432 have a larger outer radius and a smaller inner radius to account for light passing through the glass layer 430 at an angle. In some embodiments, the configuration of the electrodes disposed in the glass layer 432 can be substantially the same as the configuration of the electrodes disposed in the glass layer 430, at the potential expense of introducing vignetting on the inner and outer edges of the annular aperture segments. For example, this can be achieved by using a mask 600 ( FIG. 4 ) that is radially undersized relative to the electrode array. Fig. 6A ) and using a radially enlarged mask 620 ( Figure 6B ) to alleviate.

[0065] Figure 8B is used in the liquid crystal layer 428 Fig. 7A , Figure 7B and Fig. 8A , Figure 8C An example of a candidate rotator segment created by a combination of electrode arrangements. The incorporation of liquid crystal cells or segments 853 and 855 and the intervening liquid crystal layer forms a device that can rotate the polarization of incident light into two orthogonal polarization states, respectively. In some embodiments, segment 853 can polarize light from one pupil so that the light is orthogonally polarized relative to light from the other pupil passing through segment 855. For example, Figure 8B The two groups of segments shown in (as depicted by the two different shadings) may represent segments that rotate the polarization of light by 90° and segments that pass the light effectively unrotated, respectively.

[0066] Figure 8C It is possible to Fig. 8A The segments shown in FIG. 4 are examples of transparent electrodes used in combination. Liquid crystal material 428 (e.g., a typical twisted nematic (TN) type) may be arranged Fig. 8A and Figure 8C A portion of the transparent electrode 864 may be configured to face Fig. 8A , so that the portion faces the electrode regions 805, 810 to form a liquid crystal cell or segment (such as those found in the LCD display of a digital watch). Figure 8C The example of (and those in other figures) shows the electrodes as dark shaded portions, but in reality, the electrode layer can be substantially transparent and composed of a transparent conductive material such as indium tin oxide (ITO). The active electrode region 864 can be driven by, for example, electronics attached to the electrodes at locations 862 and / or 863. Various techniques can be used to join the electronics at locations 862, 863, such as using conductive adhesives, metal clips, or zebra-type connectors. By electronically controlling the voltage on the electrode region 864 and the Fig. 8A The voltage on the electrode, and in the electrode region 864 and Fig. 8AThe generation of a controlled voltage difference between selected segments of a liquid crystal cell can impart a pattern of polarization rotation on light traveling through the liquid crystal cell. Examples of segments that support patterns are shown in Figure 8B Shown in.

[0067] Reference again Fig. 8A , the electrode segments disposed on the glass layer 432 have the same characteristics as those disposed on the layer 430 (e.g. Fig. 7A 432). However, other configurations are possible. In some embodiments, the electrode segments disposed in the glass layer 432 can have a coarser resolution. For example, only four electrode segments in the glass layer 432 can be used to achieve a pupil that is a priori known to be disposed substantially 180° relative to each other. This utilizes Fig. 9A , Fig. 9B and Fig. 9C Specifically, Fig. 9A is another exemplary arrangement of electrodes forming four segments 902 , 904 , 906 , and 908 . Fig. 9C An example of a transparent electrode that completes a liquid crystal cell is shown. The liquid crystal material 428 is between Fig. 9A and Fig. 9C The transparent electrode region 964 is disposed on the glass layer 433. The electrode is driven by one or both connections through locations 962, 963, which are connected to the drive electronics. Thus, when segments 902, 904, 906, and 908 are electronically controlled together with the electrode region 964, a Fig. 9B In some embodiments, light passing through rotator segment 910 is orthogonally polarized to light passing through rotator segment 912. Thus, polarization control is achieved through electrical manipulation of first LCD layer 424 and second LCD layer 428. Fig. 9C The electrodes shown in Fig. 9A The electrodes in the Fig. 9B The rotator segment described in . Fig. 9B In the example of , rotator segment 910 applies one type of polarization, while rotator segment 912 applies a different type of polarization (e.g., a polarization that is orthogonal to the polarization applied by rotator segment 910). Fig. 9B The four rotator segments illustrated in may be sufficient to maintain a 180° separation between pupils 434 and 438. Naturally, other numbers of segments are possible, and the examples shown are illustrative.

[0068] Reference again Figure 4, the surgeon may wish to be able to rotate the axis of the endoscope 400 by 30° to view the side of the cavity. In this case, a pair of statically defined pupils will end up being positioned one above the other, or in some other arbitrary position where the horizontal separation required for the corresponding stereoscopic images is lost. The techniques described herein allow the pupils to be dynamically defined using LCD segments so that the pupil position adapts to the rotation of the endoscopic camera 400. During rotational or rolling motions, the endoscope sees a different view relative to gravity, and the pupils can be gradually adjusted as described to keep the left and right eyes of the camera properly separated relative to the horizontal line. Accordingly, the image from the image sensor can be electronically rotated to keep the orientation of the image displayed from the sensor in the desired orientation on the display screen. In some embodiments, a circular image can be presented to the surgeon.

[0069] Fig. 10A and Fig. 10B 4 and 4, respectively, are examples of pupil positions of two different orientations of a stereoscopic endoscope. In the examples of these figures, by using corresponding electrodes ( Fig. 7A ) in combination with electrodes 717 disposed on glass layer 431 to energize six LCD segments, thereby forming two pupils 434 and 438 in liquid crystal layer 424. Specifically, segments 1010a, 1010b, and 1010c (collectively 1010) are energized to define pupil 438, and segments 1015a, 1015b, and 1015c (collectively 1015) are energized to define pupil 434. Note that while the energized segments are shown as dark shades while the other segments are white, this is for illustrative purposes only; as is known to those skilled in the art, liquid crystal appears transparent in both states, and only becomes "dark" between properly oriented polarizers. In practice, the shaded segments will be energized so that they become substantially transparent and allow light to pass through, while the remaining segments remain in the other state to block light from passing through polarizer 426. Additionally, while Fig. 10A The example shows three segments per pupil, but a greater or lesser number of segments per pupil may be used. Using more segments per pupil increases the effective diameter of the corresponding aperture (potentially increasing light throughput), but reduces the effective inter-pupillary distance; as measured by the distance between the centroids of the two regions 1010 and 1015.

[0070] exist Fig. 10AIn the example of , a stereoscopic vision endoscope is oriented at a particular angle relative to a reference orientation, such as defined by an axis 1020 connecting LCD segments 1010b and 1015b. In some embodiments, axis 1020 can represent a "horizontal line"; that is, an orientation perpendicular to the direction of the earth's gravity. However, other reference orientations are also possible. For example, the orientation of the surgeon's eyes relative to a large display mounted on a wall or boom can be measured, and the position of the endoscope's pupil can be adjusted substantially similarly so that the surgeon's stereoscopic perception remains accurate even if the surgeon's head is tilted relative to the display. In some cases, adapting the pupil position to the surgeon's head orientation may particularly improve the surgeon's user experience; for example, in laparoscopic surgical situations, where the position of the surgeon's hands may dictate a non-ideal body posture relative to the display.

[0071] If the endoscope is rotated to another angle, e.g. Fig. 10B In the example shown in , the axial orientation of the endoscope is changed relative to the reference orientation represented by axis 1020. In this case, in order to maintain the same orientation angle of pupils 434, 438 relative to gravity (such as Fig. 10A In the case of Fig. 10B In the example of , LCD segments 1025a, 1025b, and 1025c (collectively 1025) are activated to define pupil 438, and segments 1030a, 1030b, and 1030c (collectively 1030) are activated to define pupil 434. Thus, in order to maintain the orientation angle across multiple endoscope orientations, the aperture position corresponding to the pupil is determined to maintain a predetermined interval between the first aperture position and the second aperture position, and then the aperture is created at the new position by activating the corresponding LCD segments. Although in 10A and Fig. 10B The examples in show a liquid crystal layer with radial LCD segments, but other arrangements of such segments are possible. In addition, the predetermined distance requirement is generally somewhat arbitrary and may be used in the case of Fig. 7A , Figure 7B and Fig. 8A , Figure 8C The pattern shown is changed in the system. Fig. 9A and Fig. 9C The pattern shown in illustrates a separation of 180 degrees.

[0072] In some embodiments, it may be desirable to be able to vary the distance between the pupils and change their relative orientation. Fig. 10Cillustrates two pupils formed by controlling segments in two LCD layers. In this case, the pupils are not separated by 180 degrees. In this case, segments 1040a, 1040b, 1040c form one pupil, while segments 1050a, 1050b, 1050c form a second pupil. Figure 4 In the configuration of , through the described operation of the LCD layer, these pupils are orthogonally polarized and will form images on sensors 402 and 404. The spacing and size of the pupils can be controlled, Fig. 10D Movement and changes in size of the pupil are shown. Fig. 10D Pupils 1070 and 1080 are shown to be orthogonally polarized and are at Fig. 10C The pupils may be moved in orientation and / or relative position based on a control signal calculated from an image acquired by the endoscope (e.g., physically or in software) or some other source of control signal. For example, the interpupillary distance may be dynamically controlled based on, for example, the distance from the endoscope to the surgical site.

[0073] Although the above description mainly uses Figure 4 The example of a 30° endoscope 400 is shown in FIG. 4 , but the technology can also be used for other endoscopes, such as a 0° endoscope. Fig.11 is a schematic diagram of an example of a 0° endoscope camera 1100, which includes two sensors 1102 and 1104. The endoscope 1100 can be substantially similar to the reference Figure 4 Endoscope 400 is described, except that the 0° endoscope includes a lens arrangement 1105 instead of the 30° prism of endoscope 400. The techniques described above with reference to the 30° endoscope 400 can be extended for endoscope 1100 to control the position of pupils 1112 and 1114 so that the corresponding captured light can be separated using a polarizing beam splitter 1106, with an appropriate coating or wire grid polarization layer on surface 1110 that separates the two orthogonal polarizations, thereby capturing stereo images using sensors 1104 and 1102, respectively. Selecting which pupil to use allows the 0 degree endoscope to be electronically rotated without moving parts - the pupil can be rotated as described, and the image can also be rotated electronically (or in software). From the user's perspective, this may have the effect of appearing to rotate the stereo camera system. Machine vision algorithms can also use this capability to calculate properties of the observed surface, such as calculating distances and surface normals.

[0074] In some embodiments, the electronic pupil control described herein can also be used to sequentially capture stereo images using a single image sensor. For example, two pupils corresponding to a left image and a right image can be sequentially created, and the corresponding images can be captured using a single sensor in a time division multiplexing arrangement. Fig. 12A1 is a schematic diagram illustrating an example of a 0° stereoscopic endoscope camera 1200 that uses one sensor to sense two stereoscopic images. In this method, light from a surgical scene 1202 enters the endoscope 1200 through a lens arrangement 1105. In some embodiments, the lens arrangement 1105 may include a first polarizer to polarize the incident light. The optical path of the light passing through the endoscope 1200 is represented by an exemplary sample ray 1206 passing through the right pupil 1207 and an exemplary ray 1208 passing through the left pupil 1209. In some embodiments, the same as in the endoscope 400 ( Figure 4 ) in order to support thinner cell geometries for ferroelectric liquid crystal materials, which can be used for rapid switching of LCD segments defining the pupil. In some cases, thick glass layer 1210 provides greater stability and stiffer mechanical support, making it potentially easier to create and maintain thin cells for such materials. In some embodiments, ferroelectric liquid crystal materials can be selected to facilitate rapid switching (e.g., once every 5 ms, 2 ms; 100 μs or less).

[0075] The ferroelectric crystalline material can be disposed as a thin layer (e.g., a few μm in width) between active electrodes disposed on a layer of glass (or other transparent substrate). In some embodiments, the ferroelectric liquid crystal layer can be configured to allow linearly polarized light to pass through effectively unchanged, or to rotate 90° in response to electronic control. In some embodiments, light passing through the first pupil can be polarized differently than light passing through the second pupil, for example, as described above with reference to Figure 4 In some embodiments, light from both the left pupil and the right pupil is polarized in the same manner and transmitted by the second polarizer. The light that passes through the second polarizer then passes through the optical device 1212, which is connected to the endoscope 1100 ( Fig.11 ), beam splitter 1106 is not included. Instead, light from the right pupil (represented by ray 1214) and light from the left pupil (represented by ray 1216) pass through optics 1212 to image sensor 1218. In some embodiments, a pad-printed black mask or some other light-blocking element can be incorporated into the optical path, for example, to help improve the contrast of an image sensed by the image sensor.

[0076] In embodiments where light from both right and left pixels is sensed by the same sensor, the two corresponding images may be sensed in different manners. Fig. 12A An embodiment is shown in which the light from the two pupils is polarized differently, and corresponding polarizers can be used in the image sensor 1218 to distinguish between the corresponding images. Fig. 12AAn expanded version of portion 1220 of exemplary image sensor 1218 is shown. Portion 1220 includes a plurality of pixels with a polarizer overlaid on each pixel. The polarizers used for pixels in the first group are different from the polarizers used for pixels in the second group, so that the pixels in the first group and the second group can selectively sense light having corresponding polarization states. Fig. 12A In the example shown, the polarizer for pixel 1224 is orthogonal relative to the polarizer for pixel 1226. Therefore, if the light from the pupil is properly polarized and matched with the pixelated polarizer at image sensor 1218, the image sensor can be configured to sense light from both pupils simultaneously.

[0077] exist Fig. 12A In the example of , a subset of the pixels of sensor 1218 senses light from the right pupil, and a different subset of the pixels senses light from the left pupil. Thus, the spatial resolution of each eye image is less than (half) the resolution provided by sensor 1218. In some embodiments, the full spatial resolution of image sensor 1218 can be used to sense two images by using the sensor sequentially for each pupil on a time division multiplexing basis. This Fig. 12B 1260 is illustrated using the example of a 30° endoscope with one image sensor 1260. In this example, the electronic circuits that control the state of pupils 1252 and 1254 are synchronized with, for example, a timing signal that controls the frame rate of sensor 1260. In the case of a sensor with a global shutter, which pupil is activated alternately at each frame boundary. Thus, when the left image is acquired, the left pupil 1252 is open (and the right pupil 1254 is dark). Similarly, when the right image is acquired, the right pupil 1254 is open (and the left pupil 1252 is dark). This sequence can be repeated to acquire stereoscopic images with the spatial resolution provided by sensor 1260, but at the expense of reducing the frame rate of the stereoscopic images. For example, if the frame rate of the imager is 60 Hz, a stereo pair is acquired at a rate of 30 Hz despite the "new" image being acquired at a rate of 60 Hz. In another example, if the imager is running at a frame rate of 120 Hz, a 60 Hz stereoscopic image stream can be generated.

[0078] In some embodiments, if the sensor 1260 uses a rolling shutter, the exposure times and readouts of the two pupils may overlap and need to be taken into account. An example of this is shown in Fig.13 In the example shown, each pupil is in the "open" state for two consecutive time units synchronized with the frame boundary. During the second of those two time units (each time unit may also be referred to as a frame time), data corresponding to the pupil with an open shutter is acquired. Fig.13A readout marked as "imperfect" in is an image where the illumination on the sensor is incorrect, so the data corresponding to that readout is discarded. Therefore, the acquisition rate of the stereo pair in this arrangement is one quarter of the base frame rate. For example, if the frame rate is 240Hz, the stereo pair is acquired at 60Hz. In some embodiments, because the data acquisition of one pupil is completed before the data acquisition of the other pupil, a sequential display arrangement can be used to reduce the end-to-end latency of the system. In some embodiments, if the surgeon's head is moving, the pupil position can be updated at 120Hz instead of 60Hz for the 240Hz frame rate to potentially improve the system response under rapid head movement. For example, images of the left and right pupils can be acquired at subsequent moments in time so that the pupils can be placed in the ideal position for each eye. In some embodiments, because the frames are processed sequentially rather than simultaneously, Fig.13 The pipelined nature of the arrangement described in provides reduced latency and resource-related advantages.

[0079] Fig.14 is a flow chart of an exemplary process 1400 for generating a 3D representation using the techniques described herein. In some embodiments, the operations of process 1400 may be performed at least in part by the above reference Figure 2 The processing device 43 described above is performed. In some embodiments, the operations of process 1400 can be performed by one or more processing devices of an endoscope, such as endoscopes 400, 1100, 1200, or 1250 described above. The operations of process 1400 include determining an orientation angle (1410) defined by a line connecting a first aperture position and a second aperture position of the stereoscopic endoscope relative to a reference orientation. In some embodiments, the reference orientation can be defined as a "horizontal line" perpendicular to the direction of gravity of the earth. In some embodiments, the line connecting the first aperture position and the second aperture position can be substantially similar to Fig. 10A Axis 1020 is shown, and axis 1020 passes through positions corresponding to LCD segments 1010b and 1015b. In some embodiments, determining the reference orientation may include receiving information indicating the orientation of a head of a user operating the stereoscopic vision endoscope, and determining the reference orientation based on the information indicating the orientation of the user's head; and / or determining the reference orientation relative to a display on a wall or a hanging pole.

[0080] The operations of process 1400 also include adjusting at least one of the first aperture position and the second aperture position while maintaining a spacing between the first aperture position and the second aperture position to maintain the orientation angle across the plurality of endoscope orientations (1420). This may include, for example, selecting a position of a pair of liquid crystal display (LCD) segments from a set of LCD segments disposed in a substantially annular configuration in an optical path of the stereoscopic endoscope. In some embodiments, the annular configuration of the LCD segments may be substantially as Figure 8B as shown in the example.

[0081] The operation of process 1400 further includes creating an aperture at each of the first aperture position and the second aperture position (1430). In some embodiments, this can be done, for example, using the above reference Fig. 7A , Figure 7B and Fig. 8A For example, creating an aperture can include controlling a first LCD segment in a pair of LCD segments so that the first LCD segment changes to a state in which the first LCD segment allows more light to pass through compared to a different, relatively minimal transmission or dark state, and controlling a second LCD segment in a pair of LCD segments so that the second LCD segment changes to a state in which the second LCD segment allows more light to pass through compared to a different, relatively minimal transmission or dark state. In some embodiments, the aperture can be created in a continuous pattern, such as Fig.13 As described in .

[0082] The operations of process 1400 also include generating a 3D representation using a signal based on light captured through the aperture created at the first aperture position and the second aperture position for presentation on a display device associated with the stereoscopic vision endoscope (1440). In some embodiments, the display device can be substantially similar to the display device associated with the surgeon console 50, as described above with reference to Figure 2 In some cases, process 1400 may also include receiving user input in response to presenting a visual representation of the surgical scene on a display device. For example, the user input may involve operating a surgical device at the surgical scene (e.g., referring to Figure 3 1. The robot manipulator arm assembly 120 described herein. In some embodiments, the signal may correspond to a first image and a second image acquired substantially simultaneously. In such a simultaneous acquisition, light passing through the first LCD segment may be configured to pass through a first polarizer, and light passing through the second LCD segment may be configured to pass through a second polarizer, the second polarizer polarizing light differently than the first polarizer. In some embodiments, the first polarizer may be substantially orthogonal to the second polarizer.

[0083] In some embodiments, the first sensor and the second sensor are used to sense light passing through the apertures created at the first aperture position and the second aperture position, respectively. The first sensor and the second sensor can be disposed on opposite sides of the polarizing beam splitter, for example, as described above with reference to Figure 4 In some embodiments, a single sensor is used to sense light passing through the aperture created at the first aperture position and the second aperture position, such as in the above reference Fig. 12A and Fig. 12B In the described arrangement.

[0084] The functionality (hereinafter “functionality”) of the teleoperated surgical system described herein, or portions thereof, and various modifications thereof, may be implemented at least in part by a computer program product (e.g., a computer program tangibly embodied in an information carrier), such as one or more non-transitory machine-readable media or storage devices whose operations are executed or controlled by one or more data processing devices (e.g., a programmable processor, DSP, microcontroller, computer, multiple computers, and / or programmable logic components).

[0085] A computer program may be written in any form of programming language (including compiled or interpreted languages) and may be deployed in any form (including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment). A computer program may be deployed to be executed on one or more processing devices at one site, or distributed across multiple sites and interconnected by a network.

[0086] The actions associated with implementing all or part of the functions can be performed by one or more programmable processors or processing devices executing one or more computer programs to perform the functions of the processes described herein. All or part of the functions can be implemented as a dedicated logic circuit, for example, an FPGA and / or an ASIC (Application Specific Integrated Circuit).

[0087] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory, or both. The components of a computer include a processor for executing instructions and one or more memory devices for storing instructions and data.

[0088] Although this specification contains many specific implementation details, these should not be interpreted as limitations on what can be claimed, but rather as descriptions of possible specific features of a particular embodiment. Other embodiments may also be within the scope of the technology described herein. For example, although the technology has been described with reference to dual mirror and single mirror reflections, the technology can be extended to any odd number of mirror reflections without departing from the scope of the present disclosure. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments or in any suitable sub-combination, respectively. In addition, although features may be described herein as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be cut out of the combination in some cases, and the claimed combination may involve a sub-combination or a variant of a sub-combination.

[0089] The elements of different embodiments described herein can be combined to form other embodiments not specifically described above. Elements can be excluded from the structure described herein without adversely affecting their operation. In addition, various separated elements can be combined into one or more independent elements to perform the functions described herein.

Claims

1. A method for generating a view of a scene, the method comprising: determining, by one or more processing devices, an orientation angle defined by a line connecting a first aperture position and a second aperture position of the stereoscopic vision endoscope relative to a reference orientation; adjusting at least one of the first aperture position and the second aperture position while maintaining a spacing between the first aperture position and the second aperture position to maintain the orientation angle across a plurality of endoscope orientations; creating an aperture at each of the first aperture location and the second aperture location; as well as A representation of the view is generated using a signal based on light captured through the aperture created at the first aperture position and the second aperture position for presentation on a display device associated with the stereoscopic vision endoscope.

2. The method of claim 1, wherein the reference orientation is perpendicular to the direction of Earth's gravity.

3. The method according to claim 1, further comprising: receiving information indicative of an orientation of a head of a user operating the stereoscopic vision endoscope; as well as The reference orientation is determined based on the information indicative of the orientation of the head of the user.

4. The method of claim 1 , wherein adjusting at least one of the first aperture position and the second aperture position comprises selecting a position of a pair of LCD segments from a group of liquid crystal display segments, i.e., a group of LCD segments, arranged in a substantially annular configuration in an optical path of the stereoscopic vision endoscope.

5. The method of claim 4, wherein creating the aperture at each of the first aperture location and the second aperture location comprises: controlling a first LCD segment of the pair of LCD segments so that the first LCD segment is changed to a state in which the first LCD segment allows more light to pass therethrough compared to a different, relatively less transmissive dark state; as well as A second LCD segment of the pair of LCD segments is controlled such that it changes to a state in which the second LCD segment allows more light to pass therethrough compared to a different, relatively less transmissive dark state. 6 . The method of claim 5 , wherein the first LCD segment and the second LCD segment are controlled to acquire a first image and a second image, respectively, substantially simultaneously.

7. The method of claim 6, wherein light passing through the first LCD segment passes through a first polarizer, and light passing through the second LCD segment passes through a second polarizer, the second polarizer polarizing light differently than the first polarizer. The method of claim 7 , wherein the first polarizer is orthogonal relative to the second polarizer.

9. The method of claim 7, wherein light passing through the aperture created at the first aperture position and the second aperture position is sensed using a first sensor and a second sensor, respectively, the first sensor and the second sensor being disposed on two opposite sides of a polarizing beam splitter.

10. The method of claim 1, wherein the apertures are created at the first aperture locations and the second aperture locations in a sequential pattern.

Images