Anatomical structure measurement

Abstract

A surgical measurement system for minimally invasive surgery includes: a first image capture device configured to be capable of capturing at least one first image of an interior of a body cavity of a patient; a second image capture device configured to be capable of capturing at least one second image of an interior of a body cavity of the patient; at least one display; a processor; and a non-transitory computer-readable medium storing storage instructions that cause the processor to perform a set of actions, the set of actions including: creating a depth map based on the at least one first image and the at least one second image; displaying the at least one first image on the at least one display; determining distances among a plurality of specified points based on the depth map and the at least one first image; and displaying the distance on the at least one display.

Description

Background Art

[0001] The surgical system can be combined with an imaging system that allows a clinician to view the surgical site and / or one or more parts thereof on one or more displays (such as a monitor). The display can be a local and / or remote display of a surgical operating room. The imaging system may include an observation mirror with a camera that views the surgical site and transmits the view to a display that the clinician can view. Observation mirrors include but are not limited to laparoscopes, robotic laparoscopes, arthroscopy, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-nephroscopes, sigmoidoscopes, thoracoscopes, ureteroscopes and exoscopes.

[0002] Surgical imaging systems may also involve stereoscopic vision, which may allow for 3D reconstruction of the patient's anatomy captured by the imaging system. Scene reconstruction or 3D reconstruction is the process of capturing the shape and appearance of real objects. Thus, allowing medical professionals to use scope-based imaging systems to capture, reconstruct, track, and potentially measure internal areas of the patient and any tools present in the image.

[0003] While various types of surgical instruments and image capture systems have been made and used, it is believed that the invention described herein has not been made or used before the inventors. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] While the specification is followed by claims which particularly point out and distinctly claim the invention, it is believed that the invention will be better understood from the following description of certain examples taken in conjunction with the accompanying drawings in which like reference numerals represent similar elements, and wherein:

[0005] Figure 1 An exemplary system for digital measurement is depicted;

[0006] Figure 2A and Figure 2B depicts an interface that can be used to allow a user to select points and display cross-sections and three-dimensional distances between those points;

[0007] Figure 3 depicts an interface showing a cross-sectional view overlaid on an image of an interior of a body cavity of a patient;

[0008] Figure 4 depicts an interface showing a particular object with a highlighted resection boundary;

[0009] Figure 5 depicts an interface in which a three-dimensional ("3D") representation is shown in a modal window on top of a surgical scene;

[0010] Figure 6 depicts an example interface including tools and an indication of a distance between the tools;

[0011] Figure 7 A schematic diagram of the potential relationships between multiple data sources that can be used to generate a panoramic reconstruction is provided;

[0012] Figure 8 Provides a schematic diagram of the potential relationship between multiple data sources when using keyframes to generate a panoramic reconstruction;

[0013] Fig. 9 A schematic diagram of potential relationships between multiple data sources when multiple keyframes are used in parallel to generate a panoramic reconstruction is provided; and

[0014] Fig.10 Described is a method that can be used to overlay distances between points in three-dimensional space onto a two-dimensional image of the interior of a patient's body cavity.

[0015] The drawings are not intended to be limiting in any way, and it is contemplated that various embodiments of the invention may be performed in a variety of other ways, including those not necessarily shown in the drawings. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several aspects of the invention and, together with the description, serve to explain the principles of the invention; however, it should be understood that the invention is not limited to the precise arrangements shown. DETAILED DESCRIPTION

[0016] The following description of some examples of the present invention should not be used to limit the scope of the present invention. According to the following description shown by way of example (one of the best modes envisioned for implementing the present invention), other examples, features, aspects, embodiments and advantages of the present invention will become apparent to those skilled in the art. As will be appreciated, the present invention can have other different and obvious aspects, all of which do not depart from the present invention. Therefore, the drawings and description should be considered as illustrative and non-restrictive in nature.

[0017] For clarity of disclosure, the terms "proximal" and "distal" are defined herein relative to a surgeon or other operator grasping a surgical device. The term "proximal" refers to a position of an element that is disposed closer to the surgeon, and the term "distal" refers to a position of an element that is further away from the surgeon. In addition, to the extent that spatial terms such as "top," "bottom," "upper," "lower," "vertical," "horizontal," and the like are used herein with reference to the accompanying drawings, it should be understood that such terms are used for exemplary descriptive purposes only and are not intended to be limiting or absolute. In this regard, it should be understood that surgical instruments such as those disclosed herein may be used in a variety of orientations and positions that are not limited to those shown and described herein.

[0018] Furthermore, the terms "about" and "approximately" as used herein in connection with any numerical values ​​or ranges are intended to encompass the exact value recited and a suitable tolerance such that the recited feature or combination of features works for the intended purpose described herein.

[0019] Similarly, the phrase “based on” should be understood to refer to a relationship in which one thing is at least partially determined by the thing it is designated as “based on.” This includes, but is not limited to, relationships in which one thing is exclusively determined by another thing, which relationships may be referred to using the phrase “exclusively based on.”

[0020] I. System Overview

[0021] Various systems and / or methods are disclosed herein that generally involve acquiring laparoscopic stereoscopic images and using image processing techniques to perform digital measurements. The digital measurements may be based on a variety of factors, which are discussed in greater detail herein.

[0022] The benefits of using minimally invasive surgery (MIS) are widespread and well known. Therefore, improving the ability of surgeons to perform MIS and / or expanding the range of procedures that can be performed using MIS can improve patient care.

[0023] Reference now Figure 1 , shows an exemplary system 100 for digital measurement. In some embodiments, and as shown, the system 100 may include a control system 101, a stereoscopic imaging device 102 (such as a laparoscope), one or more three-dimensional (3D) monitors 103, and one or more user input devices (such as a 2D touch screen display 104). In some embodiments, the stereoscopic imaging device 102 can capture at least two stereoscopic images of a desired surgical scene containing an object to be measured (e.g., anatomical structures, surgical tools, etc.). For example, a user (e.g., a surgeon, a first assistant, etc.) can navigate the stereoscopic imaging device 102 to allow for the user to view the object on the monitor (e.g., a 2D monitor or a 3D monitor such as a laparoscope). Figure 1 The surgeon monitor 103 and / or the 2D monitor 104, either of which may be capable of accepting touch inputs, can be implemented by viewing the desired scene on the 3D monitor 103 shown in the figure, and / or the 2D monitor 104, either of which may be capable of accepting touch inputs) to view the scene, which monitor is capable of displaying images captured by the stereoscopic imaging device in 3D or 2D. It should be understood that although the stereoscopic imaging device 102 allows the creation of depth maps and 3D images, when the user is generally viewing a surgical scene, the surgical scene may not need to be reproduced or visualized in 3D. For example, in some cases, either the surgeon monitor 103 and / or the touch display 104 may display a two-dimensional image captured by the stereoscopic imaging device (for example, in the presence of a left image sensor and a right image sensor, an image of the surgical scene captured by the left image sensor or the right image sensor may be displayed instead of a 3D reconstruction). For another example, in some cases, three-dimensional reconstruction may be used to generate a two-dimensional image from the perspective of a virtual camera and display it on the surgeon monitor 103 and / or the touch screen 104.

[0024] In some embodiments, once the stereo imaging device 102 captures at least two stereo images, those images may be sent to the control system 101. The control system 101 may then process the captured images. In one embodiment, the control system 101 may have one or more computer processing algorithms installed to allow processing of the pair of stereo images. Generally speaking, the captured images are considered to be red-green-blue (RGB) images, however, it should be understood that various other color spaces may be used, such as, for example, cylindrical coordinate color models, color models that separate brightness (or lightness) from chrominance signals, and / or grayscale. In another embodiment, the system may also utilize hyperspectral wavelengths for imaging.

[0025] Once two or more stereo images are captured by the stereo imaging device 102, the control system 101 may utilize one or more processing algorithms to derive depth information of the surgical scene. Figure 1The system shown in can use the known distance between the image capture devices to determine the three-dimensional position of a specific point (e.g., a position on a tool, as described below) by triangulation, or can create a disparity map, depth map, or point cloud of the surgical scene. In order to support the export of this type of three-dimensional information, the image sensors in the stereoscopic image device can undergo a calibration process to generate matrices and other data structures that will allow the images captured by those devices to be corrected, any distortions to be corrected, and to be related to each other. In general, the creation of depth information can utilize any known method or future method of 3D reconstruction, such as, for example, projective reconstruction, affine reconstruction, Euclidean reconstruction, etc. In general, the difference between how two image sensors perceive (e.g., see) the same scene can be used to create / calculate a depth map. The images and resulting depth maps can then be pre-processed and post-processed to compensate for potential errors or problems (e.g., reflections, image noise, etc.) that may be prevalent in current laparoscopic images.

[0026] Once the depth information (e.g., a depth map) has been derived, in some embodiments, the system can generate or construct a 3D representation of the surgical scene (including any objects present in the scene). For example, in some embodiments, there may be surgical tools in the scene. In other embodiments, there may be specific patient anatomical structures (e.g., cysts, tumors, lesions, etc.). As will be discussed in more detail herein, using the depth information and / or 3D reconstruction, the system can determine and / or measure distances between points in the surgical scene.

[0027] In some embodiments, and as Figure 1 As shown, some operating rooms ("ORs") may have one or more display devices (e.g., 3D monitor 103, 2D monitor / touch screen 104, wearable / head mounted display (not shown), etc.). Using one or more of display devices 103 / 104, the system may present 2D or 3D images to the surgeon or surgical team. Now referring to Figure 2A and Figure 2B , shows an illustrative example of a display. In some embodiments, and as shown, display device 103 / 104 can display a surgical scene 201. Two or more points 202 can be selected, such as by a user touching those points or dragging his or her finger across those points on a touch screen display showing a two-dimensional image of the surgical site. Using these points and depth information of the surgical site, the system can calculate and display a measured distance between the two or more points within the surgical scene.

[0028] As will be discussed herein, there are various methods for determining the locations of the plurality of designated points 202. In other words, the systems and methods as disclosed herein may be capable of receiving input from various input-output (I / O) devices, such as, for example, a touch screen, a keyboard, a mouse, gesture recognition, audio recording, etc. By way of non-limiting example, in some embodiments, the system may display one of the stereoscopic images (e.g., the left image or the right image) of the surgical scene 201 on the touch screen device 104, while in other embodiments, the system may display a 3D reconstructed image of the surgical scene 201 on the touch screen device 104. The user may then use their finger or a stylus to provide user input to the interface (e.g., by dragging a stylus across a two-dimensional image shown on a touch display) in order to select and / or designate two or more points 202 to be measured. These measurements may then be displayed on the image on which they are specified (e.g., a snapshot of the surgical scene), or may be displayed on a real-time image of the surgical scene during surgery (e.g., the positions of points may be tracked over time, and data such as the Euclidean distances between points may be overlaid on a display that is updated with new information about the surgical scene as it becomes available). In another embodiment, the user may also perform additional actions, such as navigating and / or drawing on the displayed image.

[0029] In another embodiment, the system may overlay various additional information, such as, for example, a 3D representation 203 of the area to be measured. Figure 3 , an additional embodiment is shown, in which a graphical representation 303 shows a cross-section of an area to be measured overlaid on a surgical scene 201, including various depth values ​​and distances. Specifically, the graph 303 represents a side view of a point-to-point measurement of a hernia defect in vivo. In some embodiments, and as shown, the graphical representation 303 may utilize a basic 3D reconstruction of the surgical scene from the acquired stereoscopic images, wherein the cutting plane defining the side view plane may be defined by a line between selected points and an average scene surface plane (i.e., a plane including the line between the points selected by the user, which is perpendicular to the plane that minimizes the square of the distance from the surface of the surgical scene) and / or a direction axis of the camera (i.e., a plane including the line between the points selected by the user, which is parallel to the direction axis of the camera).

[0030] Thus, in some embodiments, the 3D surgical view 203 and / or cross-sectional view 303 may be used to help the user determine the surface topography and associated measurements of the surgical scene to help the user ensure that the measurements reflect the correct points in the surgical scene. In another embodiment, the display 203 / 303 may be rotated and / or scaled to help the user view and / or understand how the selected points and measurements relate to the scene topography.

[0031] In addition, if Figure 2A and Figure 2B As shown, when the user selects a point 202, the point may be presented on both the 2D surgical scene 201 and the 3D surgical view 203 to help the user determine and confirm the exact location of the point. Figure 2A , Figure 2B and Figure 3 The additional views 203 / 303 are shown as an overlay, but these views may also be shown separately (eg, on a separate device). Figure 1 , a non-limiting example of such separate showing may be, for example, showing a 2D view on one or more 2D monitors 104 while showing a 3D view on one or more 3D monitors 103. It is also possible that, rather than the surgeon monitor 103 showing a 3D view of the information displayed in the 2D view on the touch screen 104, the display of the touch screen 104 may be replicated on the surgeon view 103, such as using a picture-in-picture, on a vertical split, in an overlay window, or using such other simultaneous display mechanisms as may be appropriate in a given situation. In this way, the surgeon may continue to see real-time information from the surgical scene (e.g., as captured by the laparoscope 102) while seeing the touch screen, thereby allowing him or her to follow the interaction on the touch screen 104 (e.g., by an assistant) without shifting the view on the surgeon monitor 103 or requiring the surgeon himself or herself to directly interact with the touch screen 104.

[0032] like Figure 2A , Figure 2B and Figure 3 As shown, although the measurement input can be received relative to the 2D image, the actual calculation of the measurement can be performed on the underlying 3D scene 203 and / or the determined depth value 303. Therefore, by considering the camera pose (e.g., by using the camera direction to define the cutting plane) and the scene morphology, the measured distance can reflect the actual distance within the surgical scene. In some embodiments, the system can calculate the distance using a direct or Euclidean method, in which the straight-line distance (e.g., between two points in three-dimensional space) is calculated. In other embodiments, when calculating the distance measurement, the system can follow the morphology of the surgical scene. Another embodiment can allow the user to select the calculation method (e.g., in real time or through user preference settings), or alternatively, the system can automatically select the method based on various factors (e.g., what is being measured, the time required to calculate the measurement, the number of points being measured, etc.). There may be other embodiments in which the system automatically calculates the distance in multiple ways at the same time. In some such embodiments, the system can provide each calculated distance and its method to the user.

[0033] II. Object Selection and Boundary Generation

[0034] Reference now Figure 4 , the example interface is provided with a specific object 401 highlighted (in Figure 4 In this case, it is the gallbladder, but the same approach can be applied to other objects, such as tumors or other parts of the patient's anatomy). In some cases, a system implemented based on the present disclosure may allow a user to define related objects, such as by drawing a boundary around the object of interest. Alternatively, in some cases, a system implemented based on the present disclosure may allow a user to select a point on an object (e.g., a gallbladder), and the boundaries of the object may then be automatically determined, such as by object recognition using computer vision or multi / hyperspectral imaging, such as described in U.S. Patent Publication No. 2020 / 0015925, entitled "Combination Emitter and Camera Assembly," published on January 16, 2020, the disclosure of which is incorporated herein by reference in its entirety. As yet another alternative, in some cases, a system implemented based on the present disclosure may initially identify an object (e.g., using object recognition or hyperspectral imaging), and may allow a user to modify and / or adjust the boundaries or other points or attributes of the object, such as using Figure 4 Drawing tools in the interface shown in .

[0035] like Figure 4 As shown, an interface such as may be shown by a system implemented based on the present disclosure, in addition to showing the boundaries of an object in a surgical scene 401, a boundary 402 surrounding the boundary of the object may also be shown. For example, as shown, a user may specify that a 20 mm boundary that tracks the edge of a gallbladder 401 and extends along the surface of the patient's body in three-dimensional space should be shown, which, as shown, may result in an irregularly shaped boundary 402 when shown in a two-dimensional interface. In addition to or as an alternative to user specification, in some embodiments, the boundary may be determined based on various factors such as, for example, the surgical plan, one or more surgical tools being used, object recognition of the patient's anatomy, etc. In addition, regardless of the implementation method of the boundary, in some embodiments, the system may update and / or modify or may allow the user to update and / or modify the boundary (e.g., size, color, etc.).

[0036] In some embodiments, if a user specifies a boundary that should be displayed around an object, a system implemented based on the present disclosure can use a depth map of the surgical scene to automatically generate that boundary and display it in a location such as Figure 4402. The boundary is displayed in the interface shown. This can be used, for example, to illustrate a resection boundary, which can be an area of ​​tissue surrounding a tumor (not shown) that appears to be non-tumor tissue, but can still be removed surgically for the safety of the patient. As noted, in some embodiments, the resection boundary 402 can be user-specified, while in other embodiments, the system can determine the resection size based on one or more known factors (e.g., surgical plan, preoperative images, user input, results of an anatomical structure identification algorithm, or other known factors about the patient's anatomy). Similar to Figure 2A , Figure 2B and Figure 3 The example shown in Figure 5 An embodiment is shown in which a 3D representation 503 is shown overlaid on a surgical scene 501 .

[0037] It should be understood that the above description of the interface showing specific objects and the method of identifying those objects and their resection boundaries should be understood as merely illustrative, and other variations are possible. For example, in some cases, in addition to or as an alternative to highlighting the boundaries of a specified object, a system implemented based on the present disclosure may automatically calculate (e.g., in a manner similar to that in Figure 2A and Figure 2B ) and display the longest measurement of a specified object in a manner discussed in the context of . As another example, in some cases, identification of a specified object may be facilitated by initially introducing fluorophores into the patient's body and using light from those fluorophores in identification of the object of interest. Similarly, in some cases, objects within the interior of a patient's body cavity may be initially identified in a preoperative image of the patient (e.g., a computed tomography or magnetic resonance imaging image) and may then be identified by registration of the preoperative image with a real-time image of the patient. Other variations will be apparent to those of ordinary skill in the art based on this disclosure and may be implemented without undue experimentation. Thus, in contrast to the foregoing description of Figure 4 and Figure 5 Similar to the discussion of , the above description of variations should not be taken as implying a limitation on the protection provided by this document or any other document claiming the benefit of this document.

[0038] III. Tool-Based Measurement

[0039] In addition to measuring points specified by the user and / or points automatically specified by the system, in some embodiments, the system can detect one or more tools within the surgical scene and use their relative positions to determine measurements. Thus, in some embodiments, the system can analyze at least two images (e.g., a first image from a first image capture device and a second image from a second image capture device) to identify a first surgical tool and a second surgical tool. It should be understood that although the figures and disclosure generally refer to two tools, the system can utilize more than two tools when performing tool-based measurements.

[0040] Reference now Figure 6 , an example surgical scene 600 is shown to include a first tool 601 and a second tool 602. In order for the system to detect the tools 601 / 602, computer vision is used to detect and determine the depth of the tool tip 603 / 604 (or other identifiable points on the tool, such as the pivot point of the tool, the handle, the tracking marker, the tip other than the distal tip, or any other identifiable point) of each tool. Specifically, the system may utilize a pair of stereo images including two tools 601 / 602 and a calculated real-time 3D reconstruction of the surgical scene to identify both tool tips 603 / 604. In an alternative embodiment, the system may perform direct triangulation of the tool (e.g., the tool tip) 603 / 604. As will be appreciated by those of ordinary skill in the art, in an embodiment in which the system uses direct triangulation, a calculated 3D reconstruction of the surgical scene may not be required. Therefore, in some embodiments, the system may be able to increase speed and reduce calculations by performing direct triangulation of the tools and their relative positions.

[0041] In order for the system to accurately identify the two tools 601 / 602 and determine the specified point to be measured (e.g., tool tip 603 / 604), the system may obtain or already have information associated with a specific tool. For example, the system may have received a surgical plan that includes a list of all possible tools and their specific characteristics that will be used during the operation. In an alternative example, the system may have or obtain a database of surgical tools (including their specific characteristics). In other words, before analyzing the image, the system may know what tools are expected to exist in the surgical scene 600 and what their specific characteristics should be. Specifically, the system may know or obtain tool size, shape, color, construction material, etc. In another embodiment, the tool may have an identification marker that the system can use to associate the tool with the characteristics of a known tool. In an alternative embodiment, the system can use a deep learning neural network that has been trained on various surgical tools to track and / or identify tools.

[0042] Once the tool tips 603 / 604 are identified, the system uses their locations as designated points and, as discussed herein, calculates the measured distance between them. In some embodiments, the system may display (e.g., on a display device 103 / 104) the measurement values ​​605. In other words, in some embodiments, the system may analyze the stereoscopic image to identify a first surgical tool and a second surgical tool, and then determine a set of designated points based on a first point associated with the first surgical tool and a second point associated with the second surgical tool. The measurements are then calculated based on one of the methods disclosed herein (e.g., direct triangulation, using a depth map, and / or using 3D reconstruction) and provided to the user (e.g., via a display or audio device).

[0043] IV. Temporally Extended Image Acquisition for Panoramic Reconstruction

[0044] While the examples set forth herein may be implemented using 3D reconstructions generated based on combining left and right images from overlapping fields of view of image sensors of stereo cameras, it should be appreciated that the disclosed techniques may also be used in embodiments that provide reconstructions based on images that extend beyond the individual fields of view of the image sensors of the stereo cameras, such as images captured over time as the stereo cameras pan across a surgical scene. Figures 7 to 9 Examples of approaches that may be employed to support this type of panoramic reconstruction are described in the context of FIG.

[0045] First turn Figure 7 , which provides a schematic diagram of the potential relationship between multiple data sources (shown as RGBD registration and IMU data) that can be used to generate a panoramic three-dimensional reconstruction. As will be appreciated by those skilled in the art, the surgical scene of minimally invasive surgery (MIS) may be characterized by mucosal tissue, reflections, dark areas, and the like, a fact that can complicate the application of vision-based methods for image stitching and temporal reconstruction. Therefore, in some cases, an imaging device (e.g., a laparoscopic camera) may be equipped with an inertial measurement unit (IMU), such as one that can provide three-axis measurements of linear acceleration and angular velocity when the imaging device is used to capture images. This information can be used to determine how the pose of the imaging device (i.e., its position and orientation) changes between frames. This pose information can then be used to determine how to stitch the frames together to perform a panoramic reconstruction of the surgical scene. However, as Figure 7As shown in , in some cases, rather than simply using the pose information determined using the IMU data, a combination of the IMU data and the visual data from the stereo camera (for red-green-blue depth, labeled RGBD) can be used to determine the pose information. In this case, the difference between the RGBD images between frames can be used to determine a transformation matrix representing the pose change between frames. The pose information determined from the RGBD and IMU data can then be combined (e.g., by averaging the RGBD and IMU pose changes between frames) to obtain an estimated pose at each frame, which can be more accurate than the IMU or RGBD poses alone.

[0046] Other approaches to determining the pose of the imaging device by combining multiple types of data are also possible and may be used in some embodiments to provide panoramic reconstruction functionality. Figure 8 An example of this alternative approach is provided in FIG, which shows the use of RGBD data to be based on the differences between key frames (rather than based on Figure 7 Specifically, when using a method such as Figure 8 In the implementation of the method shown in , to determine the pose of the camera when the frame was captured, the IMU data is used in conjunction with Figure 7 The RGBD data will be used in the same way as described in the context of , and the RGBD data will be used to determine the position and orientation of the camera at that frame relative to the camera at the most recent keyframe (in Figure 8 The change between the position and orientation at (indicated by the designation "k" before the frame number in the figure) can then be made as in Figure 7 RGBD and IMU based pose information are combined as described in the context of , thereby providing a reconstruction characterized by the increased accuracy of combining multiple types of data, while also reducing the risk that frame-to-frame drift in the RGBD data may become a source of error.

[0047] Also scalable Figure 8 A keyframe-based approach can be used to provide further stability of the RGBD pose information. Fig. 9 As shown in , in some cases, the system implemented based on the present disclosure can use parallel processing based on multiple key frames to simultaneously generate multiple poses based on RGBD data. Fig. 9 The system implemented by the method shown in can determine the relative position of frame 3 to the first key frame ( Fig. 9 The first RGBD pose of frame 3 is generated by moving the frame 3 relative to the second key frame ( Fig. 9) to generate a second RGBD pose for frame 3. These different RGBD poses can then be combined (e.g., by averaging) to provide further stability before they are combined with the pose determined based on the IMU data to generate the final pose to be used for panoramic reconstruction. The approach can also be extended to cover additional parallel keyframe calculations, thereby allowing systems implemented based on the present disclosure to provide additional stability in specific contexts and when appropriate. Even further variations on this type of keyframe-based approach (e.g., having a keyframe every 15 frames instead of every 5 frames, dynamically determining a keyframe when the RGBD pose deviates from the IMU pose by more than a threshold amount, etc.) are possible and will be apparent to those skilled in the art in view of the present disclosure. Therefore, the above examples of potential approaches for combining multiple frames for panoramic image reconstruction should be understood to be merely illustrative and should not be considered limiting.

[0048] Regardless of the method used to determine the 3D reconstruction, whether panoramic or otherwise, in some embodiments, such reconstruction may be performed as shown in FIG. Fig.10 Used in the method shown. Fig.10 As shown, in some embodiments, the system may utilize a first image capture device and a second image capture device (potentially housed together) to capture a first image and a second image 1001 of the interior of a patient's body cavity. The system may then create a depth map 1002 based on these images. During or after creating the depth map 1002, the system may display (e.g., on a display device such as 103 / 104) the first image or the second image 1003. The system may then identify the distance between multiple points (e.g., points selected by a user, algorithmically determined, or algorithmically determined based on user input, as described herein) 1004. Once the distance is calculated, the system may display the distance 1005. The system may also display a three-dimensional view of a portion of the interior of the patient's body cavity, a cross-sectional view of a portion of the interior of the patient's body cavity, or a graphical representation of the distance on the display device.

[0049] V. Exemplary Combinations

[0050] The following examples relate to various non-exhaustive ways in which the teachings herein may be combined or applied. It should be understood that the following examples are not intended to limit the coverage of any claims that may be provided at any time in this patent application or in the subsequent submissions of this patent application. It is not intended to make a disclaimer. The following examples are provided only for illustrative purposes. It is envisioned that the various teachings herein may be arranged and applied in a variety of other ways. It is also envisioned that some variations may omit certain features mentioned in the following examples. Therefore, any of the aspects or features mentioned below should not be considered decisive unless otherwise explicitly indicated by the inventor or the successor with an interest in the inventor at a later date. If any claim proposed in this patent application or in the subsequent submissions related to this patent application includes additional features other than those mentioned below, these additional features should not be assumed to be added for any reason related to patentability.

[0051] Example 1

[0052] A system comprises: a) a first image capture device, the first image capture device being configured to capture a first image of the interior of a patient's body cavity; b) a second image capture device, the second image capture device being configured to capture a second image of the interior of the patient's body cavity; c) a two-dimensional display; d) a processor; and e) a non-transitory computer-readable medium, the non-transitory computer-readable medium storing instructions, the instructions being operable to cause the processor to perform a set of actions when executed, the set of actions comprising: i) displaying the two-dimensional image of the interior of the patient's body cavity on the two-dimensional display; ii) determining three-dimensional distances between multiple points on the two-dimensional image; and iii) displaying the three-dimensional distances on the two-dimensional display.

[0053] Example 2

[0054] A system according to embodiment 1, wherein: a) the two-dimensional display is a touch display; b) the multiple points on the two-dimensional image include a first point and a second point; and c) the non-transient state also stores instructions, which are operable to cause the processor to: i) receive the multiple points on the two-dimensional image as user input provided by touching the touch display; ii) determine the three-dimensional position of each of the multiple points using triangulation based on the first image and the second image; iii) determine the three-dimensional distance as the length of a straight line connecting the first point and the second point in three-dimensional space and having endpoints at the first point and the second point; iv) identify a cutting plane, the cutting plane comprising the straight line connecting the first point and the second point and having endpoints at the first point and the second point; and v) displaying on the two-dimensional display simultaneously with the two-dimensional image of the interior of the body cavity of the patient a depiction of the straight line connecting the first point and the second point in three-dimensional space and having endpoints at the first point and the second point, the depiction being performed on an image selected from the group consisting of: A) a cross-sectional view of a portion of the interior of the body cavity of the patient taken on the cutting plane; and B) a three-dimensional reconstruction of the interior of the body cavity of the patient, the three-dimensional reconstruction highlighting the surface of the interior of the body cavity of the patient that intersects the cutting plane.

[0055] Example 3

[0056] A system according to embodiment 2, wherein the system includes a laparoscope that accommodates the first image capture device and the second image capture device, and wherein the cutting plane is selected from the group consisting of: a) a plane parallel to the viewing direction of the laparoscope; and b) a plane perpendicular to the plane defined by the average surface of the body cavity of the patient.

[0057] Example 4

[0058] A system according to Example 1, wherein the two-dimensional image of the interior of the body cavity of the patient is selected from the group consisting of: a) the first image of the interior of the body cavity of the patient; b) the second image of the interior of the body cavity of the patient; and c) a three-dimensional reconstruction of the interior of the body cavity of the patient.

[0059] Example 5

[0060] A system according to embodiment 1, wherein the instructions stored on the non-transitory computer readable medium are capable of being operated to cause the processor, when executed, to: a) create a depth map based on the first image and the second image; and b) use the depth map to determine the three-dimensional distance between the multiple points on the two-dimensional image as the distance between the first point and the second point along the surface of the interior of the body cavity of the patient on a plane including a straight line connecting the first point and the second point of the multiple points.

[0061] Example 6

[0062] A system according to Example 1, wherein: a) the multiple points on the two-dimensional image include: i) points on the boundary of the anatomical object in the interior of the body cavity of the patient; and ii) points on the outer edge of the resection limit surrounding the anatomical object in the interior of the body cavity of the patient; and b) the non-transitory computer readable medium also stores instructions, which are operable to cause the processor to: i) highlight the boundary of the anatomical object in the two-dimensional image of the interior of the body cavity of the patient; and ii) highlight the outer edge of the resection limit surrounding the anatomical object in the two-dimensional image of the interior of the body cavity of the patient.

[0063] Example 7

[0064] A system according to Example 1, wherein: a) the system also includes: i) a laparoscope that houses the first image capture device; and ii) an inertial measurement unit ("IMU") connected to the laparoscope; and b) the non-transitory computer-readable medium also stores instructions that are operable to cause the processor, when executed, to: i) generate multiple representations of the interior of the patient's body cavity, wherein each of the multiple representations corresponds to a time in a plurality of times; and ii) for each of the multiple times, determine a pose corresponding to the time based on: A) movement information captured from the IMU at the time; and B) a representation of the multiple representations that corresponds to the time; and iii) generate a panoramic view of the interior of the patient's body cavity based on combining the representations using poses corresponding to the times corresponding to the multiple representations of the interior of the patient's body cavity.

[0065] Example 8

[0066] A system according to Example 7, wherein, for each of the multiple times, determining the pose corresponding to the time includes: a) determining a set of potential poses by: for each potential pose in the set of potential poses, determining the potential pose based on: i) the representation corresponding to the time, and ii) a different representation among the multiple representations corresponding to a previous time; b) determining a representation pose corresponding to the time based on the set of potential poses; and c) determining the pose corresponding to the time based on: i) the representation pose corresponding to the time; and ii) an IMU pose based on the movement information captured from the IMU at the time.

[0067] Example 9

[0068] A system according to Example 7, wherein: a) each of the multiple representations is a three-dimensional representation of the interior of the body cavity of the patient; and b) the non-transitory computer-readable medium stores instructions that are operable to cause the processor, when executed, to: generate the representation based on a pair of images captured by the first image capture device and the second image capture device at corresponding times for each of the multiple representations.

[0069] Example 10

[0070] A system according to embodiment 1, wherein: a) the non-transitory computer-readable medium also stores instructions that are operable to cause the processor, when executed, to: i) analyze the first image and the second image to identify a first surgical tool and a second surgical tool; and ii) determine a first point associated with the first surgical tool and a second point associated with the second surgical tool based on the analysis; and b) the multiple points on the two-dimensional image include the first point associated with the first surgical tool and the second point associated with the second surgical tool.

[0071] Embodiment 11

[0072] A system according to embodiment 10, wherein the non-transitory computer-readable medium also stores instructions that are operable to cause the processor, when executed, to: display on the two-dimensional display simultaneously with the two-dimensional image of the interior of the body cavity of the patient a depiction of a straight line connecting the first point associated with the first surgical tool and the second point associated with the second surgical tool and having endpoints at the first point and the second point.

[0073] Example 12

[0074] A method comprising: a) capturing a first image of the interior of a body cavity of a patient and a second image of the interior of the body cavity of the patient; b) displaying the two-dimensional image of the interior of the body cavity of the patient on a two-dimensional display; c) determining three-dimensional distances between multiple points on the two-dimensional image; and d) displaying the three-dimensional distances on the two-dimensional display.

[0075] Example 13

[0076] The method of claim 12, wherein: a) the two-dimensional display is a touch display; b) the plurality of points on the two-dimensional image include a first point and a second point; and c) the method further comprises: i) receiving the plurality of points on the two-dimensional image as user input provided by touching the touch display; ii) determining the three-dimensional position of each of the plurality of points using triangulation based on the first image and the second image; iii) determining the three-dimensional distance as the length of a straight line connecting the first point and the second point in three-dimensional space and having endpoints at the first point and the second point; iv) identifying a cutting plane, the cutting plane including a line connecting the first point and the second point; and v) displaying on the two-dimensional display simultaneously with the two-dimensional image of the interior of the body cavity of the patient a depiction of the straight line connecting the first point and the second point in three-dimensional space and having endpoints at the first point and the second point, the depiction being performed on an image selected from the group consisting of: A) a cross-sectional view of a portion of the interior of the body cavity of the patient taken on the cutting plane; and B) a three-dimensional reconstruction of the interior of the body cavity of the patient, the three-dimensional reconstruction highlighting the surface of the interior of the body cavity of the patient that intersects the cutting plane.

[0077] Embodiment 14

[0078] A method according to Example 13, wherein: a) the first image and the second image of the interior of the body cavity of the patient are captured by a first image capture device and a second image capture device, respectively; b) the first image capture device and the second image capture device are housed in a laparoscope; c) the cutting plane is selected from the group consisting of: i) a plane parallel to the viewing direction of the laparoscope accommodating the first image and the second image; and ii) a plane perpendicular to the plane defined by the average surface of the body cavity of the patient.

[0079] Embodiment 15

[0080] A method according to Example 12, wherein the two-dimensional image of the interior of the body cavity of the patient is selected from the group consisting of: a) the first image of the interior of the body cavity of the patient; b) the second image of the interior of the body cavity of the patient; and c) a three-dimensional reconstruction of the interior of the body cavity of the patient.

[0081] Example 16

[0082] A method according to embodiment 12, wherein the method further includes: a) creating a depth map based on the first image and the second image; and b) using the depth map, determining the three-dimensional distance between the multiple points as the distance between the first point and the second point along the surface of the interior of the body cavity of the patient on a plane including a straight line connecting the first point and the second point of the multiple points.

[0083] Embodiment 17

[0084] A method according to Example 12, wherein: a) the multiple points on the two-dimensional image include: i) points on the boundary of the anatomical object in the interior of the body cavity of the patient; and ii) points on the outer edge of the resection limit surrounding the anatomical object in the interior of the body cavity of the patient; and b) the method also includes: i) highlighting the boundary of the anatomical object in the two-dimensional image of the interior of the body cavity of the patient; and ii) highlighting the outer edge of the resection limit surrounding the anatomical object in the two-dimensional image of the interior of the body cavity of the patient.

[0085] Embodiment 18

[0086] A method according to Example 12, wherein the method further includes: a) generating multiple representations of the interior of the body cavity of the patient, wherein each of the multiple representations corresponds to a time among multiple times; b) for each of the multiple times, determining a posture corresponding to the time based on: i) movement information captured at the time from an inertial measurement unit ("IMU") of a laparoscope connected to a first image capture device and a second image capture device that accommodate the first image and the second image of the interior of the body cavity of the patient; and ii) a representation of the multiple representations that corresponds to the time; and c) generating a panoramic view of the interior of the body cavity of the patient based on combining the representations using postures corresponding to the times corresponding to the multiple representations of the interior of the body cavity of the patient.

[0087] Embodiment 19

[0088] A method according to Example 18, wherein, for each of the multiple times, determining the posture corresponding to the time includes: a) determining a set of potential postures by: for each potential posture in the set of potential postures, determining the potential posture based on: i) the representation corresponding to the time, and ii) a different representation among the multiple representations corresponding to a previous time; b) determining a representation posture corresponding to the time based on the set of potential postures; and c) determining the posture corresponding to the time based on: i) the representation posture corresponding to the time; and ii) an IMU posture based on the movement information captured from the IMU at the time.

[0089] Embodiment 20

[0090] A method according to embodiment 18, wherein the method includes generating the representation based on a pair of images captured by the first image capture device and the second image capture device at a corresponding time of each representation in the plurality of representations.

[0091] Embodiment 21

[0092] A method according to embodiment 12, wherein: a) the method further includes: i) analyzing the first image and the second image to identify a first surgical tool and a second surgical tool; and ii) determining a first point associated with the first surgical tool and a second point associated with the second surgical tool based on the analysis; and b) the multiple points on the two-dimensional image include the first point associated with the first surgical tool and the second point associated with the second surgical tool.

[0093] Embodiment 22

[0094] A method according to Example 21, wherein the method also includes displaying on the two-dimensional display simultaneously with the two-dimensional image of the interior of the body cavity of the patient a depiction of a straight line connecting the first point associated with the first surgical tool and the second point associated with the second surgical tool and having endpoints at the first point and the second point.

[0095] It should be understood that any one or more of the teachings, expressions, embodiments, examples, etc. described herein may be combined with any one or more of the other teachings, expressions, embodiments, examples, etc. described herein. Therefore, the above teachings, expressions, embodiments, examples, etc. should not be considered isolated from each other. Various suitable ways in which the teachings herein can be combined will be apparent to those of ordinary skill in the art based on the teachings herein. Such modifications and variations are intended to be included within the scope of the claims.

[0096] It should be understood that any patent, patent publication, or other public material, whether in whole or in part, allegedly incorporated herein by reference is incorporated herein only to the extent that the incorporated material does not conflict with existing definitions, statements, or other public materials listed in the present disclosure. Therefore, and to the extent necessary, the disclosure explicitly listed herein supersedes any conflicting material incorporated herein by reference. Any material or portion thereof allegedly incorporated herein by reference but conflicting with existing definitions, statements, or other public materials listed herein will be incorporated only to the extent that there is no conflict between the incorporated material and the existing public materials.

[0097] Various embodiments of the present invention have been shown and described, and further improvements of the methods and systems described herein may be achieved by appropriate modifications by those of ordinary skill in the art without departing from the scope of the present invention. Several such possible modifications have been mentioned, and other modifications will be apparent to those of skill in the art. For example, the embodiments, embodiments, geometries, materials, dimensions, ratios, steps, etc. discussed above are illustrative and not required. Therefore, the scope of the present invention should be considered in light of the following claims, and should be understood not to be limited to the details of the structure and operation shown and described in the specification and drawings.

Claims

1. A system comprising: a) a first image capture device configured to capture a first image of an interior of a body cavity of a patient; b) a second image capture device configured to capture a second image of the interior of the body cavity of the patient; c) two-dimensional display; d) processor; and e) a non-transitory computer readable medium storing instructions operable, when executed, to cause the processor to perform a set of actions comprising: i) displaying a two-dimensional image of the interior of the body cavity of the patient on the two-dimensional display; ii) determining three-dimensional distances between a plurality of points on the two-dimensional image; and iii) displaying the three-dimensional distance on the two-dimensional display.

2. The system of claim 1, wherein: a) the two-dimensional display is a touch display; b) the plurality of points on the two-dimensional image include a first point and a second point; and c) the non-transitory computer readable medium further stores instructions operable, when executed, to cause the processor to: i) receiving the plurality of points on the two-dimensional image as user input provided by touching the touch display; ii) determining a three-dimensional position of each of the plurality of points using triangulation based on the first image and the second image; iii) determining the three-dimensional distance as the length of a straight line connecting the first point and the second point in the three-dimensional space and having endpoints at the first point and the second point; iv) identifying a cutting plane, the cutting plane comprising the straight line connecting the first point and the second point and having endpoints at the first point and the second point; as well as v) displaying, on the two-dimensional display, simultaneously with the two-dimensional image of the interior of the body cavity of the patient, a depiction of the straight line connecting the first point and the second point in three-dimensional space and having endpoints at the first point and the second point, the depiction being performed on an image selected from the group consisting of: A) a cross-sectional view of a portion of the interior of the body cavity of the patient taken on the cutting plane; and B) A three-dimensional reconstruction of the interior of the body cavity of the patient, the three-dimensional reconstruction highlighting surfaces of the interior of the body cavity of the patient that intersect the cutting plane.

3. The system according to claim 2, wherein: The system comprises a laparoscope housing the first image capture device and the second image capture device, and wherein the cutting plane is selected from the group consisting of: a) a plane parallel to the viewing direction of the laparoscope; and b) a plane perpendicular to a plane defined by a mean surface of the body cavity of the patient.

4. A system according to any one of the preceding claims, wherein: The instructions stored on the non-transitory computer readable medium are operable, when executed, to cause the processor to: a) creating a depth map based on the first image and the second image; as well as b) using the depth map, determining the three-dimensional distance between the plurality of points on the two-dimensional image as the distance between the first point and the second point along the surface of the interior of the body cavity of the patient on a plane including a straight line connecting the first point and the second point of the plurality of points.

5. A system according to any one of the preceding claims, wherein: a) The plurality of points on the two-dimensional image include: i) a point on a boundary of an anatomical object in the interior of the body cavity of the patient; and ii) a point on the outer edge of a resection limit surrounding the anatomical object in the interior of the body cavity of the patient; and b) the non-transitory computer readable medium further stores instructions operable, when executed, to cause the processor to: i) highlighting the boundary of the anatomical object in the two-dimensional image of the interior of the body cavity of the patient; and ii) highlighting the outer edge of the resection limit surrounding the anatomical object in the two-dimensional image of the interior of the body cavity of the patient.

6. A system according to any one of the preceding claims, wherein: a) The system further comprises: i) a laparoscope housing said first image capture device; and ii) an inertial measurement unit ("IMU") coupled to the laparoscope; and b) the non-transitory computer readable medium further stores instructions operable, when executed, to cause the processor to: i) generating a plurality of representations of the interior of the body cavity of the patient, wherein each representation of the plurality of representations corresponds to a time of a plurality of times; and ii) for each of the plurality of times, determining a pose corresponding to the time based on: A) movement information captured from the IMU at the time; and B) a representation corresponding to the time from among the plurality of representations; as well as iii) generating a panoramic view of the interior of the body cavity of the patient based on combining the representations using the poses corresponding to times corresponding to the plurality of representations of the interior of the body cavity of the patient.

7. The system according to claim 6, wherein: For each of the multiple times, determining the posture corresponding to the time includes: a) determining a set of potential poses by: for each potential pose in the set of potential poses, determining the potential pose based on: i) said representation corresponding to said time, and ii) a different representation of the plurality of representations corresponding to a previous time; b) determining a representation pose corresponding to the time based on the set of potential poses; and c) determining the pose corresponding to the time based on: i) said representation pose corresponding to said time; and ii) an IMU pose based on the movement information captured from the IMU at the time.

8. The system of claim 6, wherein: a) each representation of the plurality of representations is a three-dimensional representation of the interior of the body cavity of the patient; and b) The non-transitory computer-readable medium stores instructions operable, when executed, to cause the processor to: generate the representation based on a pair of images captured by the first image capture device and the second image capture device at a corresponding time for each of the plurality of representations.

9. A system according to any one of the preceding claims, wherein: a) the non-transitory computer readable medium further stores instructions operable, when executed, to cause the processor to: i) analyzing the first image and the second image to identify a first surgical tool and a second surgical tool; as well as ii) determining a first point associated with the first surgical tool and a second point associated with the second surgical tool based on the analyzing; and b) The plurality of points on the two-dimensional image include the first point associated with the first surgical tool and the second point associated with the second surgical tool.

10. The system according to claim 9, wherein: The non-transitory computer-readable medium also stores instructions that are operable to cause the processor, when executed, to: display on the two-dimensional display simultaneously with the two-dimensional image of the interior of the body cavity of the patient a depiction of a straight line connecting the first point associated with the first surgical tool and the second point associated with the second surgical tool and having endpoints at the first point and the second point.

11. A method comprising: a) capturing a first image of an interior of a body cavity of a patient and a second image of the interior of the body cavity of the patient; b) displaying a two-dimensional image of the interior of the body cavity of the patient on a two-dimensional display; c) determining three-dimensional distances between a plurality of points on the two-dimensional image; as well as d) displaying the three-dimensional distance on the two-dimensional display.

12. The method according to claim 11, wherein: a) the two-dimensional display is a touch display; b) the plurality of points on the two-dimensional image include a first point and a second point; and c) the method further comprises: i) receiving the plurality of points on the two-dimensional image as user input provided by touching the touch display; ii) determining a three-dimensional position of each of the plurality of points using triangulation based on the first image and the second image; iii) determining the three-dimensional distance as the length of a straight line connecting the first point and the second point in the three-dimensional space and having endpoints at the first point and the second point; iv) identifying a cutting plane, the cutting plane comprising the straight line connecting the first point and the second point and having endpoints at the first point and the second point; and v) displaying, on the two-dimensional display, simultaneously with the two-dimensional image of the interior of the body cavity of the patient, a depiction of the straight line connecting the first point and the second point in three-dimensional space and having endpoints at the first point and the second point, the depiction being performed on an image selected from the group consisting of: A) a cross-sectional view of a portion of the interior of the body cavity of the patient taken on the cutting plane; and B) A three-dimensional reconstruction of the interior of the body cavity of the patient, the three-dimensional reconstruction highlighting surfaces of the interior of the body cavity of the patient that intersect the cutting plane.

13. The method according to claim 12, wherein: a) the first image and the second image of the interior of the body cavity of the patient are captured by a first image capture device and a second image capture device, respectively; b) the first image capture device and the second image capture device are housed within a laparoscope; c) the cutting plane is selected from the group consisting of: i) a plane parallel to the viewing direction of the laparoscope accommodating the first image and the second image; and ii) a plane perpendicular to a plane defined by a mean surface of said body cavity of said patient.

14. The method according to any one of claims 11 to 13, wherein: The method further comprises: a) creating a depth map based on the first image and the second image; and b) using the depth map, determining the three-dimensional distance between the plurality of points as the distance between the first point and the second point along the surface of the interior of the body cavity of the patient on a plane including a straight line connecting the first and second points of the plurality of points.

15. The method according to any one of claims 11 to 14, wherein: a) The plurality of points on the two-dimensional image include: i) a point on a boundary of an anatomical object in the interior of the body cavity of the patient; and ii) a point on the outer edge of a resection limit surrounding the anatomical object in the interior of the body cavity of the patient; and b) the method further comprises: i) highlighting the boundary of the anatomical object in the two-dimensional image of the interior of the body cavity of the patient; and ii) highlighting the outer edge of the resection limit surrounding the anatomical object in the two-dimensional image of the interior of the body cavity of the patient.

16. The method according to any one of claims 11 to 15, wherein: The method further comprises: a) generating a plurality of representations of the interior of the body cavity of the patient, wherein each representation of the plurality of representations corresponds to a time of a plurality of times; and b) for each of the plurality of times, determining a pose corresponding to the time based on: i) movement information captured at said time from an inertial measurement unit ("IMU") of a laparoscope coupled to a first image capture device and a second image capture device housing the first image and the second image of the interior of the body cavity of the patient; and ii) a representation corresponding to the time in the plurality of representations; as well as c) generating a panoramic view of the interior of the body cavity of the patient based on combining the representations using poses corresponding to times corresponding to the plurality of representations of the interior of the body cavity of the patient.

17. The method according to claim 16, wherein: For each of the multiple times, determining the posture corresponding to the time includes: a) determining a set of potential poses by: for each potential pose in the set of potential poses, determining the potential pose based on: i) said representation corresponding to said time, and ii) a different representation of the plurality of representations corresponding to a previous time; b) determining a representation pose corresponding to the time based on the set of potential poses; and c) determining the pose corresponding to the time based on: i) said representation pose corresponding to said time; and ii) an IMU pose based on the movement information captured from the IMU at the time.

18. The method according to claim 16, wherein: The method includes generating the representation based on a pair of images captured by the first image capture device and the second image capture device at a corresponding time for each representation in the plurality of representations.

19. The method according to any one of claims 11 to 18, wherein: a) The method further comprises: i) analyzing the first image and the second image to identify a first surgical tool and a second surgical tool; and ii) determining a first point associated with the first surgical tool and a second point associated with the second surgical tool based on the analyzing; and b) The plurality of points on the two-dimensional image include the first point associated with the first surgical tool and the second point associated with the second surgical tool.

20. The method according to claim 19, wherein: The method also includes displaying on the two-dimensional display simultaneously with the two-dimensional image of the interior of the body cavity of the patient a depiction of a straight line connecting the first point associated with the first surgical tool and the second point associated with the second surgical tool and having endpoints at the first point and the second point.

21. A non-transitory computer readable medium storing instructions, The instructions are operable to, when executed by a processor, cause a surgical visualization system to: a) capturing a first image of an interior of a body cavity of a patient and a second image of the interior of the body cavity of the patient; b) displaying a two-dimensional image of the interior of the body cavity of the patient on a two-dimensional display; c) determining three-dimensional distances between a plurality of points on the two-dimensional image; and d) displaying the three-dimensional distance on the two-dimensional display.

22. A computer program comprising instructions operable, when executed by a processor, to cause a surgical visualization system to: a) capturing a first image of an interior of a body cavity of a patient and a second image of the interior of the body cavity of the patient; b) displaying a two-dimensional image of the interior of the body cavity of the patient on a two-dimensional display; c) determining three-dimensional distances between a plurality of points on the two-dimensional image; and d) displaying the three-dimensional distance on the two-dimensional display.

Images