Method and system for estimating position data in image

Abstract

A method performed by a surgical system. The method includes receiving a first image from an endoscope, the first image having a surgical instrument at a first location within a field of view of the endoscope. The method matches a three-dimensional (3D) model of the surgical instrument to the first image. The method receives a second image from the endoscope, the second image having the surgical instrument at a second location within the field of view of the endoscope, and matches the 3D model of the surgical instrument to the second image. The method estimates a distance between the first position and the second position based on both the match and one or more parameters of the endoscope.

Description

Technical Field

[0001] Aspects of the present disclosure generally relate to surgical systems, and more particularly to surgical systems for estimating position data (e.g., distances between positions) in an image. Other aspects are also described. Background Art

[0002] Minimally invasive surgery (MIS) such as laparoscopic surgery uses techniques designed to reduce tissue damage during a surgical procedure. A laparoscopic procedure typically requires making several small incisions in a patient's body (e.g., in the abdomen), and then inserting several surgical tools (such as an endoscope, a scalpel, a grasper, and a needle) through these incisions into the patient's body. Gas is injected into the abdomen, which inflates the abdomen, thereby providing more space around the ends of the tools and making it easier for the surgeon to see (via the endoscope) and manipulate the tissue at the surgical site. A surgical robotic system can be used to perform MIS faster and with less fatigue on the part of the surgeon, in which the surgical tools are operably attached to the distal ends of robotic arms, and a control system actuates the arms and their attached tools. When the surgeon manipulates a handheld user input device (UID), the end of the tool will mimic its position and orientation movements. A surgical robotic system can have multiple surgical arms, with one or more of the surgical arms having an attached endoscope, and the other surgical arms having attached surgical instruments for performing certain surgical actions.

[0003] Control inputs from a user (e.g., a surgeon or other operator) are captured via one or more user input devices and then translated into control of the robotic system. For example, when a surgical tool is positioned at a surgical site of a patient, a tool driver having one or more motors can actuate one or more degrees of freedom of the surgical tool in response to a user command. Summary of the Invention

[0004] Laparoscopic surgery can involve inserting several surgical tools such as an endoscope, a blade, and a grasper into a patient's body (e.g., the patient's abdomen). To perform the surgery, the surgeon can view the surgical site and the surgical tools inside the patient (e.g., in real time) via video (image) captured by the endoscope and displayed on a monitor, and can perform surgical tasks on (or at) the surgical site by manipulating the surgical tools. For example, to make an incision at the surgical site, the surgeon can manipulate the blade while viewing the incision on the monitor.

[0005] During (and / or after) a surgical procedure, a surgeon may need to determine position data related to a surgical site. Returning to the previous example, prior to making an incision, the surgeon may need to determine the distance (or length) along the surgical site to be cut with a blade. In the case where the endoscope is a stereoscopic camera that captures stereoscopic video, this distance can be estimated. For example, the surgical system can estimate the distance between two points captured by two separate cameras (based on the relative positions of the cameras with respect to each other). However, in the case where the endoscope is a monocular camera, since there is only one camera, it may not be possible to estimate the distance. Thus, the surgeon relies on using a physical ruler for the measurement, which can be cumbersome and inaccurate. Accordingly, there is a need for a surgical system that is configured to (e.g., intraoperatively) use one or more images captured by a (e.g., monocular) camera, such as an endoscope, to estimate position data, such as the distance along the surgical site.

[0006] The present disclosure provides a surgical system that estimates position data using images during (and / or after) a surgical procedure. Specifically, the system receives a first image from an endoscope, the first image having a surgical instrument at a first position within the field of view of the endoscope. For example, the first image can show the surgical instrument when the surgeon touches a location of an object (e.g., tissue). The system matches a three-dimensional (3D) model of the surgical instrument to the first image. For example, the system can adjust the model (e.g., in 3D space) to match the position and / or orientation of the surgical instrument. The system receives a second image from the endoscope, the second image having the surgical instrument at a second position within the field of view of the endoscope. In this case, the surgeon may have moved the surgical instrument and touched a different location of the object and wishes to measure the distance between the two points. The system matches the 3D model of the surgical instrument to the second image. Again, the system can manipulate the model to a new position and / or orientation to match the instrument. The system estimates the distance between the two positions based on both the match and one or more parameters of the endoscope (e.g., the focal length of the lens of the endoscope). Thus, the system can estimate the distance based on the relative transformation between two 3D models that match the surgical instrument at two positions, which allows the system to provide the distance to the surgeon regardless of whether the endoscope is a monocular camera (or a stereoscopic camera).

[0007] In one aspect, one or more parameters of the endoscope include at least one of the focal length of the lens of the endoscope, the principal point associated with the lens, and the distortion of the lens. In another aspect, the system estimates a first six-dimensional (6D) pose of the surgical instrument at a first position based on one or more parameters of the endoscope and a matching 3D model of the surgical instrument in a first image, and estimates a second 6D pose of the surgical instrument at a second position based on one or more parameters of the endoscope and a matching 3D model of the surgical instrument in a second image, wherein the distance is estimated using the first 6D pose and the second 6D pose. In some aspects, the second image is received after the first image, wherein the system further determines that the endoscope has moved from a first position where the endoscope captured the first image to a different second position where the endoscope captured the second image, and wherein the second 6D pose is determined based on the movement of the endoscope.

[0008] In one aspect, the first position and the second position are on a portion of an object within the first image and the second image, wherein the second image has a different object perspective than the first image, wherein the system further determines a 3D reconstruction of the portion of the object based on the first image and the second image, and wherein the estimated distance has a path along the 3D reconstruction of the portion of the object between the first position and the second position.

[0009] In one aspect, the surgical instrument is arranged to be manually manipulated by a user. In another aspect, the system displays 1) a first marker at the first position and a second marker at the second position, and 2) the estimated distance overlaid on top of the second image.

[0010] The above Summary of the Invention does not include an exhaustive list of all aspects of the present disclosure. It is contemplated that the present disclosure includes all systems and methods that can be implemented by all suitable combinations of the various aspects outlined above, as well as those disclosed in the Detailed Description below and particularly pointed out in the claims. Such combinations may have specific advantages not specifically recited in the above Summary of the Invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Aspects are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like reference numerals indicate like elements. It should be noted that a reference to "one" or "an" aspect of the present disclosure is not necessarily to the same aspect, and they mean at least one. Additionally, for the sake of brevity and to reduce the total number of figures, a given figure may be used to illustrate features of more than one aspect, and not all elements in the figures may be required for a given aspect.

[0012] Figure 1 A pictorial view of an example surgical system in an operating site is shown.

[0013] Figure 2Block diagram of a surgical system including estimated position data within an image, according to one aspect.

[0014] Figure 3a and Figure 3b Flowchart of a process for an aspect of estimating the distance between two positions.

[0015] Figure 4a and Figure 4b Shows several stages, showing a display with a surgical instrument, where the distance between two positions where the surgical instrument touches an object is estimated.

[0016] Figure 5 Flowchart of a process for another aspect of estimating the distance between two positions using an image received from an endoscope of a surgical system.

[0017] Figure 6 Flowchart of a process for an aspect of estimating the distance between two surgical instruments using an image received from an endoscope of a surgical system.

[0018] Figure 7 Shows several stages, showing a display with two surgical instruments, where the distance between the two instruments is estimated. DETAILED DESCRIPTION

[0019] Now, several aspects of the present disclosure will be explained with reference to the accompanying drawings. Whenever the shape, relative position, and other aspects of the parts described in a given aspect are not explicitly defined, the scope of the present disclosure is not limited herein only to the parts shown, and the parts shown are for illustrative purposes only. Additionally, although many details are set forth, it should be understood that some aspects may be practiced without these details. In other instances, well-known circuits, structures, and techniques are not shown in detail so as not to obscure the understanding of this specification. Furthermore, unless the meaning clearly dictates otherwise, all ranges recited herein are considered to include the endpoints of each range.

[0020] Figure 1 Shows a pictorial view of an exemplary (e.g., laparoscopic) surgical system (which may hereinafter be referred to as "the system") 1 in an operating site. The system 1 includes a user console 2, a control tower 3, and one or more surgical robotic arms 4 at a surgical robotic operating table (surgical table or surgical platform) 5. In one aspect, the arm 4 may be mounted to, such as Figure 1The operating table or bed on which the patient is located as shown in the example. In one aspect, at least some of the arms in arm 4 can be configured differently. For example, at least some of the arms can be mounted on the ceiling, side wall, or another suitable structural support (such as a cart separate from the operating table). System 1 can be combined with any number of devices, tools, or accessories for performing surgery on patient 6. For example, system 1 can include one or more surgical tools (instruments) 7 for performing a surgical procedure (surgical protocol). Surgical tool 7 can be an end effector attached to the distal end of surgical arm 4 for performing a surgical protocol.

[0021] Each surgical tool 7 can be manually manipulated, robotically manipulated, or both during a surgical procedure. For example, surgical tool 7 can be a tool for accessing, viewing, performing surgical tasks, and / or manipulating the internal anatomy of patient 6. In one aspect, surgical tool 7 is a gripper that can grasp the tissue of the patient. Surgical tool 7 can be manually controlled by a bedside operator 8; or it can be robotically controlled via the actuation movement of the surgical robot arm 4 to which it is attached. For example, when manual control is performed, the operator can (e.g., physically) hold a part of the tool (e.g., the handle), and can manually control the tool by moving the handle and / or pressing one or more input controls (e.g., buttons) on the tool (e.g., the handle of the tool). In another aspect, when controlled robotically, the surgical system can manipulate the surgical tool based on user input (e.g., received via user console 2 as described herein).

[0022] Generally speaking, a remote operator 9 (such as a surgeon or other operator) can use user console 2 to remotely manipulate arm 4 and / or the attached surgical tool 7, for example, during remote operation. User console 2 can be located in the same operating room as the rest of system 1, as Figure 1As shown. However, in other environments, the user console 2 may be located in an adjacent or nearby room, or it may be located at a remote location, e.g., in a different building, city, or country. The user console 2 may include one or more components, such as a seat 10, one or more foot controls (or foot pedals) 13, one or more (handheld) user input devices (UIDs) 14, and at least one display 15. The display is configured to display, for example, a view of the surgical site within the patient 6. The display may be configured to display image data (e.g., still images and / or video). In one aspect, the display can be any type of display, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED) display, a head-mounted display (HMD), etc. In some aspects, the display may be a 3D immersive display for displaying 3D (surgical) renderings. For example, during a surgical procedure, one or more endoscopes (e.g., an endoscope camera) may capture image data of the surgical site, and the display presents the image data in 3D to the user. In one aspect, the 3D display can be an autostereoscopic display that provides 3D perception to the user without the need for special glasses. As another example, the 3D display can be a stereoscopic display that provides 3D perception by using glasses (e.g., via active shutter or polarization).

[0023] In another aspect, the display 15 may be configured to display at least one graphical user interface (GUI), which may provide information and / or interactive content to assist the user in performing a surgical procedure through one or more instruments in the surgical system 1. For example, some of the displayed content may include image data captured by one or more endoscope cameras, as described herein. In another aspect, the GUI may include selectable UI items that, when manipulated by the user, may cause the system to perform one or more operations. For example, the GUI may include UI items as interactive content to switch control between robotic arms. In one aspect, to interact with the GUI, the system may include input devices (such as a keyboard, mouse, etc.). In another aspect, the user may use the UID 14 to interact with the GUI. For example, the user may manipulate the UID to navigate through the GUI (e.g., using a cursor), and making a selection may involve hovering the cursor over a UI item and manipulating the UID (e.g., selecting a control or button). In some aspects, the display may be a touch-sensitive display screen. In this case, the user may perform selections by navigating and selecting via touching the display. In some aspects, any method may be used to navigate and / or select UI items.

[0024] As shown, the remote operator 9 sits in the seat 10 and views the user display 15 while manipulating the foot control 13 and the handheld UID 14 to remotely control one or more of the robotic arm 4 and the surgical tool 7 (which is mounted on the distal end of the arm 4).

[0025] In some variations, the bedside operator 8 can also operate the system 1 in the "on-bed" mode, where the bedside operator (user) is now located on one side of the patient 6 and simultaneously manipulates the robot-driven tool (the end effector attached to the arm 4), for example, holding the handheld UID 14 and a manual laparoscopic tool with one hand. For example, the left hand of the bedside operator can manipulate the handheld UID to control the robotic components, while the right hand of the bedside operator can manipulate the manual laparoscopic tool. Thus, in these variations, the bedside operator can perform both robot-assisted minimally invasive surgery and manual laparoscopic surgery on the patient 6.

[0026] During an exemplary procedure (surgical operation), the patient 6 is prepared for surgery and draped in a sterile manner to achieve anesthesia. The initial access to the surgical site can be performed manually while the arm of the system 1 is in the stowed configuration or the retracted configuration (to facilitate access to the surgical site). Once the access is completed, the initial positioning or setup of the system 1 (including its arm 4) can be performed. Then, the surgical operation continues, where the remote operator 9 at the user console 2 uses the foot controls 13 and the UID 14 to manipulate the various end effectors and possibly the imaging system to perform the surgical operation. Manual assistance can also be provided by a bedside person (e.g., the bedside operator 8) wearing a sterile surgical gown at the operating table or surgical bed, and this bedside person can perform tasks on one or more of the robotic arms 4, such as retracting tissue, performing manual repositioning, and tool replacement. There can also be a non-sterile person to assist the remote operator 9 at the user console 2. When the procedure or surgical operation is completed, the system 1 and the user console 2 can be configured or set to a certain state to facilitate the completion of the postoperative procedure, such as cleaning or disinfection and entering or printing a health record via the user console 2.

[0027] In one aspect, a remote operator 9 holds and moves a UID 14 to provide an input command to drive (move) one or more robotic arm actuators 17 (or drive mechanisms) in the robotic system 1 for remote operation. The UID 14 may be communicatively coupled to the rest of the system 1 via, for example, a console computer system 16 (or host). The UID 14 may generate a spatial state signal corresponding to the movement of the UID 14, such as the position and orientation of the handheld housing of the UID, and the spatial state signal may be an input signal for controlling the movement of the robotic arm actuator 17. The system 1 may use a control signal derived from the spatial state signal to control the proportional movement of the actuator 17. In one aspect, a console processor of the console computer system 16 receives the spatial state signal and generates a corresponding control signal. Based on these control signals for how the actuator 17 is powered to drive a section or link of the arm 4, the movement of the corresponding surgical tool attached to the arm may mimic the movement of the UID 14. Similarly, the interaction between the remote operator 9 and the UID 14 may generate, for example, a grasping control signal that causes the jaws of the gripper of the surgical tool 7 to close and grasp the tissue of the patient 6.

[0028] The system 1 may include a number of UIDs 14, and corresponding control signals are generated for each UID to control the actuators and surgical tools (end effectors) of the respective arms 4. For example, the remote operator 9 may move a first UID 14 to control the movement of the actuator 17 located in the left robotic arm, where the actuator responds by moving linkages, gears, etc. in the arm 4. Similarly, the movement of a second UID 14 by the remote operator 9 controls the movement of another actuator 17, which in turn drives other linkages, gears, etc. of the system 1. The system 1 may include a right arm 4 fixed to a bed or table on the right side of the patient, and a left arm 4 located on the left side of the patient. The actuator 17 may include one or more motors that are controlled such that they drive the joints of the arm 4 to rotate to, for example, change the orientation of an endoscope or gripper of the surgical tool 7 attached to the arm relative to the patient. The movement of several actuators 17 in the same arm 4 may be controlled by a spatial state signal generated from a specific UID 14. The UID 14 may also control the movement of the corresponding surgical tool gripper. For example, each UID 14 may generate a corresponding grasping signal to control the movement of an actuator (e.g., a linear actuator) that opens or closes the jaws of the gripper at the distal end of the surgical tool 7 to grasp the tissue within the patient 6.

[0029] In some aspects, the communication between the surgical robot operating table 5 and the user console 2 can be through the control tower 3, which can translate the user commands received from the user console 2 (and more specifically from the console computer system 16) into robot control commands transmitted to the arm 4 on the surgical operating table 5. The control tower 3 can also transmit the status and feedback from the surgical operating table 5 back to the user console 2. The communication connections between the surgical operating table 5, the user console 2, and the control tower 3 can be via a wired link (e.g., optical fiber) and / or a wireless link, using any suitable data communication protocol among various wireless data communication protocols, such as the Bluetooth protocol. Any wired connection can optionally be built into the floor and / or walls or ceiling of the operating room. The system 1 can provide a video output to one or more displays, including the display in the operating room and remote displays accessible via the Internet or other networks. The video output or feed can also be encrypted to ensure privacy, and all or part of the video output can be saved to a server or an electronic health record system.

[0030] Figure 2 is a block diagram of a surgical system 1 including estimated position data within an image. The system includes one or more (e.g., electronic) components (or elements), such as a controller 20, a camera (e.g., an endoscope) 22, a sensor 23, a surgical instrument 24, a display 25, a speaker 29, and a memory 28. In one aspect, the system can include more or fewer elements, such as having one or more surgical instruments and / or having one or more (e.g., different) sensors 23. In another aspect, the surgical system 1 can include other elements not shown, such as having one or more robotic arms to which the surgical instrument 24 can be coupled.

[0031] In some aspects, at least some of the elements can be part of a single electronic device (or housed within the enclosure of a single electronic device). For example, the controller 20 and the memory 28 can be part of the control tower 3 of the surgical system 1. In another aspect, at least some of the elements can be separate electronic devices (or parts of separate electronic devices) relative to each other. For example, the sensor 23 can be a separate electronic device that can be positioned within the surgical site where the surgical system is located (or within a part of the surgical site). In one aspect, the elements of the surgical system can be communicatively coupled to the controller 20 (and / or to each other) to exchange digital data. For example, the controller 20 can be configured to receive sensor data from the sensor 23 via a wired (and / or wireless) connection.

[0032] In the case of a wireless connection, the controller may be configured to communicate wirelessly (e.g., to exchange data) with one or more elements (such as sensor 23) via a network. In one aspect, the device may communicate via any (computer) network, such as a wide area network (WAN) (e.g., the Internet), a local area network (LAN), etc., through which the devices may exchange data with each other and / or may exchange data with one or more other electronic devices (such as a remote electronic server). In another aspect, the network may be a wireless network, such as a wireless local area network (WLAN), a cellular network, etc., for exchanging digital data. With respect to the cellular network, the controller (e.g., via a network interface) may be configured to establish a wireless (e.g., cellular) call, where the cellular network may include one or more cell towers, and the one or more cell towers may be part of a communication network (e.g., a 4G Long Term Evolution (LTE) network) that supports data transmission (and / or voice calls) for electronic devices such as mobile devices (e.g., smart phones). In another aspect, the device may be configured to wirelessly exchange data via other networks (such as a wireless personal area network (WPAN) connection). For example, the controller may be configured to establish a wireless communication link (connection) with an element (e.g., an electronic device including sensor 23) via a wireless communication protocol (e.g., the Bluetooth protocol or any other wireless communication protocol). During the established wireless connection, the electronic device may transmit data (such as sensor data) to the controller 20 as data packets (e.g., Internet Protocol (IP) packets).

[0033] The camera 22 (e.g., a complementary metal oxide semiconductor (CMOS) image sensor) is an electronic device configured to capture video (and / or image) data (e.g., as a series of still images). In one aspect, the camera may be an endoscope designed to capture video of a surgical site within a patient during a surgical procedure. In one aspect, the camera may be a monocular camera that captures a single digital image (e.g., a still image) at a time (e.g., with a single camera sensor that captures a single digital image at a time) (e.g., to produce an endoscope video stream). In another aspect, the camera may be a stereo (stereoscopic) camera with two (or more) lenses, each lens having a separate camera sensor for capturing a single still image (e.g., for producing a separate video stream) in order to create a 3D video.

[0034] The surgical instrument (or tool) 24 can be any type of surgical instrument designed to be used during a surgical procedure and includes an end effector for performing one or more surgical tasks. For example, the surgical instrument can be a grasper for grasping and holding an object, an ultrasonic instrument that uses ultrasonic vibrations (e.g., at its tip) to rapidly generate heat for cutting and cauterizing tissue, a scalpel, etc. In one aspect, as described herein, the camera and the surgical instrument can be manually manipulated, robotically manipulated, or manipulated in both ways during a surgical procedure. For example, the surgical instrument 24 can include a handle (coupled to its proximal end) that is configured to be held by an operator and that allows the operator to manually control (e.g., the position, orientation, and configuration of) the distal end of the surgical instrument. Thus, the surgical instrument can be arranged to be manually manipulated by a user (e.g., a surgeon).

[0035] The sensor 23 can be any type of electronic device configured to detect (or sense) the environment (e.g., an operating room) and generate sensor data based on the environment. For example, the sensor 23 can include at least one microphone that can be configured to convert sound energy caused by the propagation of sound waves into an input microphone signal (or audio signal). In another aspect, the sensor can be a proximity sensor (e.g., an optical sensor) configured to detect the presence of one or more objects within the environment. In another aspect, the sensor can be a temperature sensor that senses the ambient temperature (e.g., within the room where the sensor is located) as sensor data.

[0036] In some aspects, the sensor can be a motion sensor (e.g., an inertial measurement unit (IMU)) designed to measure position and / or orientation. For example, the IMU can be coupled to the camera 22 (or a part of the camera) and can be configured to detect the movement of the camera (e.g., changes in the position and / or orientation of the camera) (e.g., due to an operator manipulating the camera to show different perspectives of a surgical site during a surgical procedure). In some aspects, the motion sensor can be a camera that captures images used by the controller 20 to perform motion tracking operations (e.g., based on changes in the captured images).

[0037] A memory (e.g., a non-transitory machine-readable storage medium) 28 can be any type of electronic storage device. For example, the memory can include read-only memory, random access memory, CD-ROM, DVD, magnetic tape, optical data storage devices, flash memory devices, and phase change memory. Although the memory is shown as separate from the controller 20, the memory can be part of the controller 20 (e.g., the internal memory of the controller). As shown, the memory 28 includes one or more 3D (e.g., computer-aided design (CAD)) models 27 of one or more surgical instruments 24 of the surgical system. Specifically, each model in the model can be a graphically mathematical coordinate-based representation of (at least a part of) a surgical instrument (e.g., as one or more basis (or B) splines, such as non-uniform rational basis splines (NURBS)). For example, when the instrument is a surgical clamp including a grasping / clamping distal portion coupled to a shaft, the 3D model can be a representation of the grasping / clamping distal portion. In one aspect, each model can include one or more different orientations of the corresponding instrument relative to a reference point within a 3D coordinate system (e.g., a Cartesian coordinate system). In some aspects, at least some of the models in the model can be predefined (e.g., provided to the surgical system 1 via a communication link with an electronic device (e.g., a remote server) that generates (and / or stores) the model). For example, one or more models in the model can be generated and provided by the manufacturer of the corresponding surgical instrument. In another aspect, one or more models in the model can be periodically updated (and / or added to the memory) in the memory 28 of the surgical system.

[0038] The controller 20 can be any type of electronic component configured to perform one or more computing operations. For example, the controller can be a dedicated processor, such as an application-specific integrated circuit (ASIC), a general microprocessor, a field-programmable gate array (FPGA), a digital signal controller, or a set of hardware logic structures (e.g., filters, arithmetic logic units, and dedicated state machines). The controller 20 is configured to receive image data (e.g., as a video stream) captured by the camera 22 and is configured to perform position data estimation, such as estimating the distance between two points in one or more images using one or more models 27 of one or more surgical instruments captured in the images. Such operations can allow an operator of the surgical system 1 to perform intraoperative position measurements using images captured by a monocular camera. More details about how the controller estimates position data are described herein.

[0039] In one aspect, the controller may be configured to receive user input via one or more input (electronic) devices (not shown). For example, the controller 20 may be coupled to one or more (e.g., peripheral computer) input devices, such as a keyboard or a mouse through which user input may be received. In another aspect, user input may be received via a touch-sensitive display (e.g., display 25), which may display a graphical user interface (GUI) having one or more user interface (UI) items, where the device may generate one or more control signals (as user input) based on an operator touching a part of the UI item being presented (of interest to the operator) on the touch-sensitive display. In some aspects, the touch-sensitive display may be part of an electronic device (such as a tablet computer, a laptop computer, or a smart phone).

[0040] Figures 3a - 3b , Figure 5 , and Figure 6 are flowcharts of processes for aspects of estimating position data. In one aspect, at least some of the operations of at least some of the processes in the process may be performed intraoperatively (e.g., when a surgeon performs a surgical (e.g., laparoscopic) procedure on a patient). In another aspect, at least some of the operations may be performed postoperatively (e.g., based on video captured during the surgical procedure). More details on how to perform the operations postoperatively are described herein. In some aspects, at least some of the operations of at least some of the processes in the process may be performed by the surgical system 1 described herein (e.g., the controller 20 of the surgical system).

[0041] Now turning to Figure 3a and Figure 3b , these show flowcharts of a process 30 for aspects of estimating the distance between two positions. The process 30 begins with the controller 20 calibrating the camera 22 (e.g., a monocular endoscope) of the surgical system 1 to determine one or more (e.g., internal) parameters of the camera (e.g., components such as the lens and / or image sensor of the camera) (at block 31). In one aspect, the parameters may be internal parameters of the camera associated with that particular camera (e.g., associated with the physical properties of the camera). In some aspects, these internal parameters may include at least one of the focal length of the camera (e.g., the lens of the camera), the principal point (or optical center) of the camera (e.g., the lens of the camera), the skew of the camera, and the distortion of the lens of the camera. In some aspects, to calibrate the camera to determine the parameters, the controller may execute a calibration algorithm that uses one or more images captured by the camera and one or more characteristics of the camera (e.g., external parameters such as position and / or orientation). In some aspects, the calibration algorithm may be based on the type of camera of the surgical system (e.g., whether the camera is a pinhole camera). In another aspect, the controller may execute any type of calibration (e.g., algorithm) to determine one or more parameters.

[0042] In one aspect, calibration may be performed by the manufacturer of the camera. In this case, the controller may perform the calibration and may retrieve one or more parameters. In another aspect, calibration may be performed "in the field", which means that calibration may be performed by the surgical system before (or during) a surgical procedure. In some aspects, calibration may be performed once, such as during the initial power-up of the camera when the camera is first connected to the surgical system 1. As another example, calibration may be performed when the camera is first coupled to the controller 20 and powered. Once calibration is complete, the controller may store (and use) one or more parameters to estimate position data, as described herein. In another aspect, the camera may be calibrated periodically (e.g., once a week).

[0043] The controller 20 displays the video captured by the camera on the display 25 (at block 32). In one aspect, the captured video may be displayed in real time. Specifically, the camera may capture video of the surgical site during a surgical procedure, and the surgical system may display the captured video for the surgeon to view during the procedure. The controller 20 receives (e.g., a first) image from the camera (e.g., the video captured by the camera), the image including a surgical instrument 24 (at block 33) at a (e.g., first) position within the field of view of the camera (e.g., touching an object at the first position). Specifically, the image includes the field of view of the camera and includes at least a portion of the surgical instrument (such as the distal portion that the surgeon may use to perform one or more surgical tasks), the field of view that may include a surgical site (at least a partial view of) (e.g., within the patient's cavity) having one or more objects (e.g., tissue, fluid, etc.). As described herein, the distal portion may be a gripper. In one aspect, the surgical instrument may have one or more degrees of freedom within the surgical space, such as having six degrees of freedom (6DOF), which allows the instrument to translate and / or rotate (e.g., manually) along at least one of three perpendicular axes.

[0044] In one aspect, the first image can be one of a series of images that are being captured (or have been captured) by a camera and are being displayed on display 25 (e.g., as described in block 32). In another aspect, the first image can be the first image captured by the camera when the camera is activated (e.g., when the camera is powered on). In some aspects, the first image can be received based on (e.g., in response to) a user input (received by controller 20). Specifically, the surgeon can move the surgical instrument to a first position such that the surgical instrument (e.g., a portion of the surgical instrument) is at (or adjacent to) and / or touching the first position (e.g., the position from which the surgeon wishes to estimate position data such as distance), and the controller can receive the first image in response to receiving the user input. For example, once moved to the position, an input can be received via an input device (e.g., the surgeon can press a (e.g., physical) button of an electronic device (e.g., a mouse) coupled to the controller, which generates a control signal based on the input and transmits it to the controller). Once the input is received, the controller can retrieve (draw) the first image captured by the camera. In another aspect, in response to the user input, the camera can capture the first image. In another aspect, the user input can be received in other ways, such as via a voice command. In this case, when the surgeon wishes for the surgical system to capture the first image, the surgeon can speak a phrase of one or more words. In response, a microphone (e.g., as sensor 23) can capture the phrase and perform a voice detection algorithm to detect the phrase contained therein. Upon detection of the phrase, the controller can receive (capture) the first image.

[0045] As described herein, a first image can be captured in response to a user input. In another aspect, the first image can be automatically captured (e.g., without user intervention). For example, the controller 20 can receive the first image based on an object recognition algorithm (which is being executed by the controller 20). Specifically, the controller 20 can determine whether to estimate position data based on the recognition of one or more objects (and / or a first position) within the first image. As described herein, the estimation of the position data can be performed during a surgical procedure. In this case, the procedure can include one or more surgical tasks known to the controller, such as the estimation of position data. The controller can be configured to receive the first image (for position estimation) based on determining that an object detected within the video stream captured by the camera is associated with a surgical task that includes the estimation of position data. In another aspect, the controller can be configured to receive the first image based on determining that the surgical instrument has been held at a first position for a period of time. As another example, the controller can be configured to receive the first image based on determining that the surgical instrument or more specifically the distal end of the surgical instrument is pressing against (contacting or touching) an object within the field of view of the camera 22. In one aspect, the controller can determine whether the surgical instrument is in contact with an object based on sensor data obtained by the sensor 23. For example, when the sensor is a pressure sensor coupled to the surgical instrument, it can generate sensor data indicating that the surgical instrument is pressing against an object. In another aspect, the controller can determine that the surgical instrument is touching an object based on object recognition (e.g., recognizing that the area of the object changes based on contact with the instrument).

[0046] The controller 20 matches the 3D model of the surgical instrument with at least a portion of the first image (at block 34). Specifically, the surgical instrument captured within the first image can have a particular position and / or orientation. The controller 20 can retrieve the 3D model 27 of the surgical instrument from the memory 28 and can project (e.g., align) the model onto the surgical instrument within the captured image such that the 3D model matches the surgical instrument (e.g., is superimposed over the surgical instrument) (e.g., within a tolerance threshold). In one aspect, the controller can manipulate the 3D model within the 3D model space (e.g., adjust the scale, orientation, etc.) to align (match) the model with the surgical instrument (e.g., to meet or exceed the tolerance threshold, as described herein). In another aspect, the model 27 can include one or more 3D models of the same surgical instrument but with different scales and / or orientations. In this case, the controller can be configured to determine the scale and / or orientation of the surgical instrument within the first image and then retrieve the 3D model that matches the scale and / or orientation of the instrument (e.g., up to the tolerance threshold) to select the matching 3D model. For example, the model 27 can include a table of one or more 3D models with different scales and / or orientations, and the controller can perform a table lookup on the data structure to select the stored 3D model with the matching scale and / or orientation.

[0047] In one aspect, the controller may retrieve a model based on one or more characteristics of the surgical instrument. For example, the memory 28 may include a table that includes a list of 3D models stored in the memory relative to one or more characteristics, such as a unique identifier of the instrument (e.g., serial number). The controller may determine one or more characteristics of the surgical instrument and may perform a table lookup on the table to identify the 3D model associated with the surgical instrument. Once identified, the controller may retrieve the 3D model and may perform a matching operation to match the 3D model with the instrument shown in the first image.

[0048] The controller 20 is configured to estimate a first six-dimensional (6D) pose of the surgical instrument at the first position (at block 35) based on one or more parameters (e.g., determined during calibration of the camera) and the matched 3D model of the surgical instrument in the first image. Specifically, the 3D model may represent the surgical instrument within a 3D model space. The controller uses one or more (e.g., internal) parameters of the camera to define the position and orientation of the surgical instrument relative to the position of the camera. In one aspect, the controller may apply the internal parameters and the 3D model to a 6D pose model (e.g., as inputs thereto), which produces a 6D pose as an output. Specifically, the 6D pose includes the orientation and position of the surgical instrument relative to the camera in a 3D coordinate system (such as a Cartesian coordinate system including an X-axis, a Y-axis, and a Z-axis) (e.g., at its origin). For example, the 6D pose includes rotations (pitch, yaw, and roll) between the X-axis, Y-axis, and Z-axis, and translations along the X-axis, Y-axis, and Z-axis from (or relative to) a reference point (such as the origin of the 3D coordinate system (e.g., which is the position of the camera)). In some aspects, the controller may use any known (or future) method (e.g., algorithm) to determine the 6D pose of an object in the image relative to the camera that captured the image.

[0049] The controller receives another (e.g., second) image from the camera (e.g., a video captured by the camera), which includes the surgical instrument 24 (at block 36) at another (e.g., second) location within the field of view of the camera (e.g., touching an object at the second location). Specifically, the second image may be captured after the surgical instrument (e.g., by the operator) has moved from a first position to a second position (e.g., after the first image) in order to estimate the position data associated with the two positions. For example, the position data may be the distance between the first (starting) position and the second (target or destination) position. In this case, both positions may be set on an object (at least a part of it) within the two received images. In one aspect, the second image may be received in a similar manner to the first image. For example, the second image may be received in response to receiving a user input (e.g., the operator presses a button on an input device for the camera to capture the second image). In another aspect, the second image may be received in response to determining that the surgical instrument is touching the second position of the object. The controller 20 matches the 3D model of the surgical instrument with the second image (e.g., at least a part of the second image) (at block 37). In one aspect, the controller may use the same 3D model used to match the surgical instrument in the first image to match the surgical instrument in the second image. In this case, the controller may adjust the 3D model (e.g., scale and / or rotate the 3D model) to match the surgical instrument within a tolerance threshold. In another aspect, the controller may be configured to match a different 3D model with the surgical instrument in the second image relative to the 3D model used for the first image.

[0050] The controller 20 determines whether the camera has moved (e.g., based on sensor data from the sensor 23) (at decision block 38). Specifically, the controller may determine whether the camera has moved from a first position where the camera captured the first image to a different second position where the camera captured the second image. For example, the sensor may be an IMU as described herein, which may be coupled to the camera 22 and may generate sensor (e.g., motion) data based on the movement of the camera. The controller may obtain the sensor data and determine whether the camera has moved after capturing the first image and before capturing the second image. In another aspect, the surgical system may include an external (e.g., 6D) tracking system and a marker attached to the camera. In this case, the tracking system may be another camera arranged to capture an image of the camera 22 including the marker, where the controller 20 may be configured to determine whether the camera 22 has moved based on the detected movement of the marker.

[0051] In another aspect, the controller 20 can determine whether the camera 22 has moved based on one or more images (e.g., a first image and a second image) captured and received from the camera 22. For example, the controller executes a camera motion tracking algorithm (e.g., a Simultaneous Localization and Mapping (SLAM) algorithm, a Visual Odometry (VO) algorithm, etc.) to track the movement of the camera based on the movement of one or more points within a sequence of one or more video frames (images) captured by the camera. In this case, the controller 20 can determine that the second image is captured at a perspective different from the perspective of the first image captured by the camera (e.g., with respect to one or more axes within the viewing space of the camera).

[0052] If so, the controller 20 estimates the second 6D pose of the surgical instrument at the second position (at block 39) based on one or more parameters, a matching 3D model of the surgical instrument in the second image, and / or the detected movement of the camera. As described herein, the one or more parameters can be the internal parameters of the camera. When the movement of the camera is detected, the controller can be configured to determine one or more external parameters of the camera, where the external parameters indicate changes in the position (e.g., translation) and / or orientation (e.g., rotation) of the camera within the environment (e.g., the real 3D world). Specifically, the controller 20 can apply the internal parameters, the external parameters, and the matching 3D model to a 6D pose model (e.g., the model used to determine the first 6D pose) to estimate the second 6D pose of the surgical instrument. Thus, the controller can determine the second 6D pose of the surgical instrument while taking into account (e.g., accounting for) the movement of the camera (e.g., which can occur due to an operator manipulating the camera).

[0053] As described herein, the camera can be manually manipulated (e.g., moved by an operator adjusting a handle coupled to the camera) to move the camera (e.g., to capture different perspectives of the surgical site). In this case, the surgical system can determine the movement based on sensor data from the sensor 23. In another aspect, when the camera is coupled to a robotic arm (e.g., Figure 1 the arm 4 in [reference]), the controller can determine the presence of camera movement based on the movement of one or more actuators of the arm. For example, the controller can receive one or more control signals generated from a spatial state signal (received from one or more UIDs of the system) and can determine how the robotic arm will move based on the generated control signals, which will be used to move the actuator 17 of the arm 4. In another aspect, the surgical system can include one or more (motion) sensors 23 coupled to the arm and can determine the arm movement based on the sensor data.

[0054] However, if the camera 22 has not moved yet (or the controller has not detected movement based on the sensor data), the controller estimates the second 6D pose of the surgical instrument at the second position (at block 40) based on one or more internal parameters of the camera and a matching 3D model of the surgical instrument in the second image.

[0055] Process 30 continues to Figure 3b where the controller determines whether a 3D reconstruction (e.g., a 3D physical representation) of one or more objects within one or more images captured by the camera 22 is available (at decision block 41). In some aspects, the 3D reconstruction may be a 3D physical representation of the surface of the objects within one or more captured images. In one aspect, the controller may determine that the 3D reconstruction is available (or can be generated or determined) based on whether the controller can generate such a reconstruction. Specifically, the controller may be configured to determine a 3D reconstruction of at least a portion of an object based on one or more images captured by a camera of the system. For example, the controller may use a SLAM (and / or VO) algorithm to generate the reconstruction based on one or more different images (e.g., when the second image is different from the first image (e.g., has a different viewing angle from the first image)). In another aspect, the determination may be based on whether the surgical system (e.g., the memory 28 of the surgical system) includes a (e.g., predefined) 3D reconstruction of the object.

[0056] If the 3D construction is not available, the controller estimates the (e.g., linear) distance between the first position and the second position based on the first 6D pose and the second 6D pose of the surgical instrument (at block 42). For example, given the 6D poses, the controller may be configured to determine the relative transformation between the two poses. For example, the controller may determine a transformation function (e.g., a transformation matrix) that, when applied to a matrix associated with the first 6D pose (e.g., the first position of the first 6D pose) of the surgical instrument in the first image, produces a matrix associated with the second 6D pose (e.g., the second position of the second 6D pose) of the surgical instrument in the second image. Based on the relative transformation, the controller may derive the distance between the two 6D poses (e.g., the two positions of the two 6D poses). In another aspect, the controller may use other known (or future) methods to determine the distance between the two positions.

[0057] The controller 20 displays the distance between a first position and a second position (at block 43) that is superimposed over the video of the object (e.g., on top of the video). For example, when performing at least some of the operations to estimate the position data, the surgical system may display a video of the surgical site captured by the camera 22. In this case, once the distance is estimated, the surgical system may display the distance between the two points such that the surgeon can see the distance and the (e.g., straight) line between the two points, and the surgeon can use the line as a guide for any surgical task (e.g., cutting).

[0058] However, if 3D reconstruction is available, the controller 20 estimates the distance between the first position and the second position as a path along the 3D reconstruction of the object (at block 44) based on the first 6D pose and the second 6D pose of the surgical instrument. As described herein, the first position and the second position within the first image and the second image can be part of (or on) a part of the object captured by the two images. For example, an operator of the surgical system may be trying to measure the distance between two positions on a tissue patch. However, in the case where the tissue patch is not flat (e.g., instead is curved) and / or textured (e.g., has hills and valleys), the operator may intend to measure along a path on the surface of the tissue patch between the two positions. Thus, the controller can be configured to use the 3D reconstruction and the relative transformation between the two 6D poses to determine the distance. For example, once the distance is determined using the relative transformation, the controller can adjust (modify) the distance based on the 3D reconstruction such that it conforms to the path along the 3D reconstruction.

[0059] Some aspects may perform variations of the process 30 described herein. For example, the particular operations of the process may not be performed in the exact order shown and described. A particular operation may not be performed in a series of consecutive operations, and different particular operations may be performed in different aspects. For example, the operations within the dashed boxes may be optional operations that may not be performed when executing at least a portion of the process 30. As an example, if calibration has been previously performed (e.g., during a previous execution of the process 30), the process may omit the operation to calibrate the camera performed at block 31. In this case, the controller can be configured to retrieve one or more parameters from the memory 28.

[0060] As described herein, the controller can determine whether the distance is along the path of the 3D reconstruction of the object or a linear distance based on whether the 3D reconstruction of the object is available. In another aspect, the controller can determine which distance to estimate and display based on user input (e.g., a user selection of a UI item on the GUI displayed on the display 25 that indicates which distance the user wishes to view).

[0061] Figure 4a and Figure 4bShows a number of stages 50 - 55, which show video 57 captured by camera 22 in display 25 with surgical instrument 24, where the distance between two positions where it is estimated that the surgical instrument touches an object (e.g., tissue 56) shown in video 57 is estimated. In one aspect, these figures show at least some of the operations described in the operation in Figure 3a and Figure 3b Process 30. The first stage 50 shows video 57 displayed on display 25 captured by camera 22, which includes a surgical instrument adjacent to object 56, and the surgical instrument is a grasping tool (e.g., having a distal grasping / clamping portion coupled to the surgical instrument) (e.g., showing the instrument entering the field of view of the camera).

[0062] The second stage 51 shows the surgical instrument at a first position 58 on the object. In one aspect, this stage shows a first image received from camera 22, as described in block 33 of process 30 herein in Figure 3a . In one aspect, this stage shows an image that may be received in response to receiving a user input (e.g., an operator selects a button indicating that camera 22 captures an image). In one aspect, the first position 58 may be a position where the surgical instrument (e.g., the distal end of the surgical instrument) has touched. In another aspect, the position may be on the surgical instrument (or a part of the surgical instrument). The third stage 52 shows a 3D model 59a aligned (e.g., matched) and superimposed over at least a portion of surgical instrument 24. Specifically, this stage shows the result of controller 20 retrieving the 3D model 59a associated with the surgical instrument and matching the 3D model with the surgical instrument (e.g., by manipulating the 3D model within the 3D model space).

[0063] Moving on to Figure 4b , the figure shows a fourth stage 53, where surgical instrument 24 moves to a second position 90. Specifically, in this stage, the surgeon may have moved the instrument to the second position in order to measure the distance between the first position 58 and the second position 90. In one aspect, this stage shows an image that the surgical system may capture in response to a user input, or may have automatically captured (e.g., a second image as described herein) (e.g., when it is detected that the surgical instrument has touched the second position 90). As described herein, the first position 58 may be a part of the surgical instrument (e.g., the distal end). In this case, the second position 90 may be the same part of the surgical instrument, but at a different position within the surgical site (e.g., within the field of view of the camera) (e.g., around object 56).

[0064] The fifth stage 54 shows the 3D model 59b aligned (e.g., matched) and superimposed over the surgical instrument 24. As described herein, the controller may match the 3D model 59b with the surgical instrument. In one aspect, the 3D model 59b may be the same as the 3D model 59a, but have a different position and / or orientation relative to the position and / or orientation of the 3D model 59a. In such a case, the controller may manipulate the model 59a to align with the position and / or orientation of the surgical instrument within the fifth stage 54, as described herein.

[0065] The sixth stage 55 shows the result of the controller estimating the distance between two positions by performing the position data estimation operation described herein (e.g., using the parameters of the camera and the aligned 3D model to estimate the 6D pose and estimating the distance based on those poses). Specifically, the figure shows two markers 92 and a notification 91, each at one of a first position 58 and a second position 90, the notification including a distance of 10 mm spanning between the two markers. Specifically, the notification includes a line spanning from the two markers 92 and text indicating the distance value, which in this case is 10 mm and may correspond to the length of the line. In one aspect, the notification (and / or marker) is a UI item (e.g., the second image used matches the 3D model with the surgical instrument at the second position shown in the video) overlaid on top of the video 57 (e.g., superimposed over the video). In one aspect, the UI item may be continuously displayed at the portion of the object being measured as long as the portion of the object is within the field of view of the camera 22. As shown herein, the distance is the linear distance between the two markers. In another aspect, when 3D reconstruction is available for the object, the distance may be a curved path along the surface of the object 56, as described herein.

[0066] Figure 5 is a flowchart of another aspect of a process 80 for estimating the distance between two positions using an image received from the endoscope 22 of the surgical system 1. The controller 20 receives a first image from the endoscope 22, the first image having a surgical instrument 24 at a first position within the field of view of the endoscope (at block 81). The controller matches the 3D model of the surgical instrument with the first image (at block 82). Specifically, the controller matches (aligns) the 3D model of the instrument with the instrument shown within the first image. The controller 20 receives a second image from the system from the endoscope, the second image having a surgical instrument at a second position within the field of view of the endoscope (at block 83). The controller matches the 3D model of the surgical instrument with the second image (at block 84). The controller estimates the distance between the first position and the second position based on both the match and one or more (e.g., internal) parameters of the endoscope (at block 85).

[0067] As described so far, the surgical system 1 can be configured to estimate position data based on captured images of surgical instruments, such as the distance between two different positions of a surgical instrument. In another embodiment, images including two or more surgical instruments can be used to estimate position data. Figure 6 FIG. 60 is a flow chart of a process 60 for aspects of using an image received from the endoscope 22 of the surgical system 1 to estimate the distance between two surgical instruments. The controller 20 receives an image (e.g., captured by the endoscope) from the endoscope that includes a first surgical instrument and a second surgical instrument (at block 61). For example, the first surgical instrument can be a grasping tool (e.g., similar to the Figure 4a and Figure 4b instrument shown), and the second surgical instrument can be a scalpel. In one aspect, each of the surgical instruments can be arranged to be individually manipulated by a user of the surgical system (e.g., when manually manipulated via a separate handle). In another aspect, at least one of the surgical instruments can be coupled to a robotic arm of the surgical system (e.g., the distal end of the robotic arm).

[0068] The controller 20 matches a first 3D model of the first surgical instrument and a second 3D model of the second surgical instrument to the image (at block 62). Specifically, the controller can retrieve the first 3D model associated with the first surgical instrument (from the model 27 in the memory 28), and retrieve the second 3D model associated with the second surgical instrument, and can align the two 3D models with their respective surgical instruments as described herein.

[0069] The controller 20 estimates position data (e.g., the distance between the first surgical instrument and the second surgical instrument) associated with the first surgical instrument and the second surgical instrument based on the match and one or more parameters of the endoscope (at block 63). For example, the controller can estimate the distance between two surgical instruments (e.g., two points associated with the two surgical instruments) by estimating the respective 6D poses of the two surgical instruments and then using the relative transformation between the 6D poses to determine the distance as described herein. In one aspect, the distance between specific points (e.g., the distal end) on each of the surgical instruments can be estimated. In another aspect, the distance along an object between positions where one or both of the surgical instruments have contacted the object can be estimated.

[0070] Thus, compared with Figure 3a and Figure 3bUnlike the estimation of the distance described in [[reference]], an image can be used to estimate the distance between two surgical instruments. On the other hand, when estimating position data with two (or more) surgical instruments, the controller can perform one or more operations described in those figures. For example, the controller can determine whether a 3D reconstruction of an object within the received image is available (e.g., based on whether one or more images have been received), and if so, the estimated distance between the two surgical instruments can be along the 3D reconstruction, as described herein. For example, the controller can receive an image that includes an object that is at least partially behind the two surgical instruments (e.g., where the surgeon can touch different locations of the object with each of the instruments). The controller can determine a 3D physical representation of the surface of the object (at least along the portion between the two instruments), and determine position data that includes an estimated surface distance along the 3D representation of the surface between the two instruments. In one aspect, the estimated surface distance can be the shortest distance between points of the two instruments relative to the physical characteristics of the surface.

[0071] Figure 7 Several stages 70 - 72 of a video 57 shown in a display 25 are shown, which include two surgical instruments 77 and 73 and an object (e.g., tissue) 56 (e.g., within the field of view of the camera), where the surgical system estimates the distance between the two instruments. The first stage 70 shows the video 57 captured by the camera 22, which is displayed within the display and includes a first surgical instrument 77 and a second surgical instrument 73, both of which are grasping tools. In one aspect, this stage shows how the surgeon positions the instruments around the object so that the system can estimate and display the distance between the instruments.

[0072] In one aspect, the controller can receive images of the two surgical instruments at this stage. For example, once the surgeon has placed the two instruments in their respective positions to estimate the distance between the instruments, the surgeon can provide user input (e.g., select a button, as described herein) to cause the controller to receive the image (and estimate the distance between the instruments). On the other hand, the controller can automatically receive images for distance estimation, as described herein.

[0073] The second stage 71 shows a first 3D model 74 of the first surgical instrument 77 that matches the first instrument and a second 3D model 75 of the second surgical instrument 75 that matches the first instrument (e.g., in the image of the first stage 70 received by the controller 20). As described herein, the controller can be configured to manipulate the two 3D models so that they are aligned (within a tolerance threshold) with their respective instruments. In one aspect, although shown as overlapping with their respective instruments (at least a portion), this may be for illustrative purposes only. In this case, the video 57 shown in the second stage 71 can be the same (or similar) to the video shown in the first stage 70.

[0074] In the third stage 72, the controller shows the result of estimating the distance between the two surgical instruments by showing a notification 76 that includes a line extending from the first (e.g., distal) end or position of the first instrument 77 to the second (e.g., distal) end or position of the second instrument 73, and the line includes the distance between the two positions (e.g., 10 mm). As described herein, the controller can estimate the distance by estimating the 6D pose of each respective instrument and then estimating the relative transformation between the two 6D poses (e.g., the transformation that produces one 6D pose when applied to one of the 6D poses).

[0075] As described herein, the distance 76 between the two instruments can be estimated. In one aspect, the controller can continuously update the distance based on the change of one or both of the instruments. For example, when the surgeon moves the second surgical instrument 73 away from the first surgical instrument 77, the controller can recalculate the 6D pose of the second instrument (and / or recalculate the 6D pose of the first instrument) based on the change of the alignment 3D model of the instruments, and display the new distance between the two instruments according to the estimate. Thus, the notification 76 can be updated as the surgeon moves at least one of the instruments to provide the surgeon with real-time position data updates.

[0076] As described so far, at least some of the operations performed by the surgical system can be performed intraoperatively. For example, the surgical instruments can estimate position data in real time, such as the distance between two surgical instruments, so as to provide the surgeon with real-time up-to-date measurements during the surgical procedure. In another aspect, at least some of the operations can be performed postoperatively. In this case, the controller 20 can be configured to receive a video stream (or one or more video streams) previously captured by the camera 22 (and stored in the memory 28), and can be configured to use the video stream to estimate position data. In this case, the controller can receive user input, and the controller can estimate position data according to the user input. As an example, referring to Figure 7 , when these operations are performed postoperatively, the controller can be configured to receive a user instruction to receive an image of the video stream in which the user is interested in its position data. For example, when the user is viewing the video stream, the scene shown in the first stage 70 can be presented. Once shown, the user may wish to see the distance between the surgical instruments. In response, the user can select a button, which can provide a control signal to the controller, indicating that the controller estimates the distance between the instruments, as described herein. Thus, the position data estimation operation performed by the controller can allow the user to view the data after performing the surgical procedure.

[0077] As described so far, an example of position data of one or more surgical instruments that can be estimated is distance (e.g., the distance between surgical instruments). Other types of position data are possible. For example, an estimation of the 6D pose of a surgical instrument can be performed one or more times (e.g., along the length of a surgical procedure). In such a case, the surgical system can estimate (and display) the travel path of the surgical instrument relative to camera 22 (e.g., along a surgical site). As another example, the position data can include one or more position statistics of a surgical instrument, such as how long the surgical instrument has remained at a particular position (or within a threshold distance of that position), how frequently the surgical instrument moves, and the average orientation of the surgical instrument (e.g., which can be based on the average 6D pose of the instrument over the entire surgical procedure). In one aspect, as described herein, the estimated position data can be displayed on video 57 (e.g., overlaid on top of the video), and / or can be provided within a report generated by the surgical system.

[0078] As described herein, the surgical system uses one or more 3D models to estimate the distance between at least two positions. In one aspect, each position can correspond to a point (or portion) of an object captured by a camera of the surgical system in one or more images. In another aspect, the position can be the point of an object that a portion (e.g., the distal end) of a surgical instrument captured in the image contacts. In another aspect, the position can be a point on an object that at least one surgical instrument has contacted. In another aspect, the position can be a point (or portion) of a surgical instrument (such as the distal end of the instrument) in one or more images captured by the system. In such a case, the position relative to the surgical instrument can be predefined.

[0079] Although certain aspects have been described and illustrated in the drawings, it should be understood that such aspects are merely illustrative of the invention and not limiting, and the invention is not limited to the specific constructions and arrangements shown and described, as various other modifications can be contemplated by those of ordinary skill in the art. Accordingly, this specification should be regarded as illustrative rather than restrictive.

[0080] To assist the Patent Office and any readers of any patent issued on this application in interpreting the appended claims, the applicant wishes to note that, unless the words "means for" or "step for" are expressly used in a particular claim, the applicant does not intend any of the appended claims or claim elements to invoke 35 U.S.C. 112(f).

[0081] As previously explained, one aspect of the present disclosure can be a non-transitory machine-readable medium (such as a microelectronic memory) having instructions stored thereon that program one or more data processing components (collectively referred to herein as "processors") to automatically (e.g., without user intervention) use one or more images to estimate location data (e.g., the distance between two locations), as described herein. In other aspects, some of these operations may be performed by specific hardware components that include hardwired logic. These operations may alternatively be performed by any combination of programmed data processing components and fixed hardwired circuit components.

[0082] Although certain aspects have been described and illustrated in the drawings, it should be understood that such aspects are merely illustrative of the present disclosure and not limiting thereof, and the present disclosure is not limited to the specific constructions and arrangements shown and described, as various other modifications may occur to those of ordinary skill in the art. Accordingly, this specification should be regarded as illustrative rather than restrictive.

[0083] In some aspects, the present disclosure may include language such as, for example, "[Element A] and at least one of [Element B]". This language can refer to one or more of the elements. For example, "at least one of A and B" can refer to "A", "B", or "A and B". Specifically, "at least one of A and B" can refer to "at least one of A and at least one of B" or "at least one of A or B". In some aspects, the present disclosure may include language such as, for example, "[Element A], [Element B], and / or [Element C]". This language can refer to any one of these elements or any combination thereof. For example, "A, B, and / or C" can refer to "A", "B", "C", "A and B", "A and C", "B and C", or "A, B, and C".

Claims

1. A method, comprising: Receiving a first image from an endoscope, the first image having a surgical instrument at a first position within the field of view of the endoscope; Matching a three-dimensional (3D) model of the surgical instrument with the first image; Receiving a second image from the endoscope, the second image having the surgical instrument at a second position within the field of view of the endoscope; Matching the 3D model of the surgical instrument with the second image; And Estimating a distance between the first position and the second position based on both the matching and one or more parameters of the endoscope.

2. The method according to claim 1, wherein, The one or more parameters of the endoscope include at least one of a focal length of a lens of the endoscope, a principal point associated with the lens, and a distortion of the lens.

3. The method according to claim 1, further comprising: Estimating a first six-dimensional (6D) pose of the surgical instrument at the first position based on the one or more parameters of the endoscope and the matched 3D model of the surgical instrument in the first image; And Estimating a second 6D pose of the surgical instrument at the second position based on the one or more parameters of the endoscope and the matched 3D model of the surgical instrument in the second image; Wherein the distance is estimated using the first 6D pose and the second 6D pose.

4. The method according to claim 3, wherein The second image is received after the first image, wherein the method further comprises determining that the endoscope has moved from a first position where the endoscope captured the first image to a different second position where the endoscope captured the second image, and wherein the second 6D pose is determined based on the movement of the endoscope.

5. The method according to claim 1, wherein The first position and the second position are on a part of an object within the first image and the second image, wherein the second image has a different perspective of the object than the first image, and wherein the method further comprises determining a 3D reconstruction of the part of the object based on the first image and the second image, and wherein the estimated distance has a path along the 3D reconstruction of the part of the object between the first position and the second position.

6. The method according to claim 1, wherein, The surgical instrument is arranged to be manually manipulated by a user.

7. The method according to claim 1, further comprising displaying: 1) a first marker at the first position and a second marker at the second position, and 2) the estimated distance overlaid on top of the second image.

8. A surgical system, comprising: A surgical instrument; An endoscope; A processor; And A memory having instructions stored therein that, when executed by the processor, cause the surgical system to: Receive a first image from the endoscope, the first image including the surgical instrument, Match a three-dimensional (3D) model of the surgical instrument with the first image, Receive a second image from the endoscope, the second image including the surgical instrument, Match the 3D model of the surgical instrument with the second image, and estimate position data associated with the surgical instrument based on both the match and one or more parameters associated with the endoscope.

9. The surgical system according to claim 8, wherein, The one or more parameters of the endoscope include at least one of the focal length of the lens of the endoscope, the principal point associated with the lens, and the distortion of the lens.

10. The surgical system according to claim 8, wherein, The memory has the following additional instructions: Estimate a first six-dimensional (6D) pose of the surgical instrument at the first position based on the one or more parameters of the endoscope and the matched 3D model of the surgical instrument in the first image; And Estimate a second 6D pose of the surgical instrument at the second position based on the one or more parameters of the endoscope and the matched 3D model of the surgical instrument in the second image; Wherein the distance is estimated using the first 6D pose and the second 6D pose.

11. The surgical system according to claim 10, wherein, The second image is received after the first image, wherein the memory has the following additional instructions: Determine that the endoscope has moved from a first position where the endoscope captured the first image to a different second position where the endoscope captured the second image, wherein the second 6D pose is determined based on the movement of the endoscope.

12. The surgical system according to claim 8, wherein, The first position and the second position are on a part of an object in the first image and the second image, wherein the second image has a different perspective of the object than the first image, wherein the memory has the following additional instructions: Determine a 3D reconstruction of the part of the object based on the first image and the second image, wherein the estimated distance has a path along the 3D reconstruction of the part of the object between the first position and the second position.

13. The surgical system according to claim 8, wherein, The surgical instrument is arranged to be manually manipulated by a user.

14. The surgical system according to claim 8, wherein, The memory has the following additional instructions: Display 1) a first marker at the first position and a second marker at the second position, and 2) the estimated distance overlaid on top of the second image.

15. A method, comprising: Receive an image from an endoscope, the image including a first surgical instrument and a second surgical instrument; Match a first three-dimensional (3D) model of the first surgical instrument and a second 3D model of the second surgical instrument with the image; And Estimate position data associated with the first surgical instrument and the second surgical instrument based on the match and one or more parameters of the endoscope.

16. The method according to claim 15, wherein, The one or more parameters of the endoscope include at least one of the focal length of the lens of the endoscope, the principal point associated with the lens, and the distortion of the lens.

17. The method according to claim 15, further comprising: Estimate a first six-dimensional (6D) pose of the first surgical instrument based on the one or more parameters of the endoscope and the matched first 3D model; And Estimate a second 6D pose of the second surgical instrument based on the one or more parameters of the endoscope and the matched second 3D model; The position data is estimated using the first 6D pose and the second 6D pose.

18. The method according to claim 15, wherein, The image is a first image including an object at least partially located behind the first surgical instrument and the second surgical instrument, and the method further includes: receiving, from the endoscope, a second image including the first surgical instrument, the second surgical instrument, and the object; and determining a 3D physical representation of the surface of the object based on the first image and the second image, wherein the position data includes an estimated surface distance along the 3D physical representation of the surface of the object between the first surgical instrument and the second surgical instrument.

19. The method according to claim 15, further comprising displaying the position data overlaid on top of the image.

20. The method according to claim 19, wherein, The position data includes a line from the first surgical instrument to the second surgical instrument and a distance indicating the length of the line.

Images