Computer assisted surgery of anatomical objects using virtual objects

Abstract

The present disclosure describes a computer system and method for adding a virtual reference object to a video of an anatomical structure. The computer system includes a memory and a processor communicatively coupled to the memory. The processor receives the video of the anatomical structure. The video shows a first view. The processor also receives a selection of a point and a vector in the first view, adds a first virtual geometry to the first view based on the point and the vector, detects a movement that causes the first view to change to a second view, and presents the first virtual geometry in the second view such that a pose of the first virtual geometry is maintained from the first view to the second view.

Description

COMPUTER ASSISTED SURGERY OF ANATOMICAL OBJECTS USING VIRTUAL OBJECTSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims benefit of co-pending United States provisional patent application Serial No. 63 / 597,576 filed November 9, 2023. The aforementioned related patent application is herein incorporated by reference in its entirety.TECHNICAL FIELD

[0002] The present disclosure relates generally to surgical systems. Specifically, the present disclosure relates to a surgical system that uses virtual objects to assist during surgery.BACKGROUND

[0003] Doctors use computer assisted surgical systems to perform operations on patients, even remotely. These surgical systems provide the doctors various views of surgical sites during the operations. The views, however, may provide a limited perspective of the anatomical object. As a camera capturing the views moves in the body or as a doctor removes portions of the anatomy, the doctor can become disoriented and lose track of the positioning of the camera and other medical instruments in the body.SUMMARY

[0004] The present disclosure describes a computer system and method for adding a virtual reference object to a video of an anatomical structure. According to an embodiment, the computer system includes a memory and a processor communicatively coupled to the memory. The processor receives the video of the anatomical structure. The video shows a first view. The processor also receives a selection of a point and a vector in the first view, adds a first virtual geometry to the first view based on the point and the vector, detects a movement that causes the first view to change to a second view, and presents the first virtual geometry in the second view such that a pose of the first virtual geometry is maintained from the first view to the second view.

[0005] According to another embodiment, a method includes receiving the video of the anatomical structure. The video shows a first view. The method also includes receiving a selection of a point and a vector in the first view, adding a first virtual geometry to the first view based on the point and the vector, detecting a movement that causes the first view to change to a second view, and presenting the first virtual geometry in the second view such that a pose of the first virtual geometry is maintained from the first view to the second view. Other embodiments includes a non-transitory machine-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method.

[0006] The foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 illustrates an example surgical system.

[0008] Figures 2A and 2B illustrate example components of the surgical system of Figure 1.

[0009] Figures 3A and 3B illustrate an example operation for adding and presenting a virtual geometry (e.g., a virtual plane) in the surgical system of Figure 1 .

[0010] Figures 3C, 3D, and 3E illustrate an example operation for adding and presenting a virtual geometry (e.g., a virtual plane) in the surgical system of Figure 1.

[0011] Figure 4 illustrates an example operation for adding and presenting a virtual geometry in the surgical system of Figure 1 .

[0012] Figure 5 illustrates an example operation for detecting proximity to a virtual geometry in the surgical system of Figure 1 .

[0013] Figures 6A and 6B illustrate example operations for adding and presenting virtual geometries (e.g., virtual cones and tubes) in the surgical system of Figure 1.

[0014] Figures 6C and 6D illustrate example operations for adding and presenting virtual geometries (e.g., virtual cones and tubes) in the surgical system of Figure 1.

[0015] Figure 7 illustrates an example operation for adding and presenting a virtual geometry in the surgical system of Figure 1 .

[0016] Figure 8 is a flowchart of an example method for adding and presenting a virtual geometry and performed by the surgical system of Figure 1 .

[0017] Figure 9 is a flowchart of an example method for adding and presenting a virtual geometry and performed by the surgical system of Figure 1 .DETAILED DESCRIPTION

[0018] Doctors use computer assisted surgical systems to perform operations on patients, even remotely. These surgical systems provide the doctors various views of surgical sites during the operations. For example, a surgical system includes an endoscope that is inserted into a patient’s body so that the endoscope captures video or an image stream of an anatomical object on which the doctor will be operating. The video or images, however, provide a limited view or perspective of the anatomical object. As the endoscope moves in the body or as the doctor removes portions of the anatomy, the doctor can become disoriented and lose track of the positioning of the endoscope and other medical instruments in the body. Additionally, it can be difficult for the doctor to properly judge depth while viewing the video or image stream, especially if certain medical conditions (e.g., cancer) mangle tissue planes that the doctor typically uses to judge depth. As a result, the doctor may unnecessarily cut too deeply into tissue.

[0019] Furthermore, some structures in the body are partially or fully covered by other anatomical objects. As a result, the doctor may have trouble seeing these structures, and the doctor may unintentionally cut or damage these structures during surgery. For example, the ureter may extend through a portion of the body, but the ureter may be blocked from view by other tissue or membranes along the length of the ureter. When the doctor is operating near the ureter, the doctor may unintentionally damage or cut the ureter if the doctor cannot see the ureter.

[0020] The present disclosure describes a computer system that assists the doctor during a surgical procedure by adding virtual geometries to the view provided to the doctor. For example, the computer system may add one or more virtual planes to the video or image stream. Each plane may serve as a reference plane during the procedure. A plane may be set with a particular orientation and at a certain depth in the body (e.g., a depth as viewed from the visual plane of the video or image stream). When the endoscope moves to a different position, the view provided by the video or image stream may change. The computer system may maintain the orientation and depth of the plane even though the view in the video or image stream has changed. As a result, the plane may continue to mark a particular orientation or depth in the body even though the view in the video or image stream has changed.

[0021] As another example, the computer system may add a virtual geometry to the video or image stream to indicate the location of an anatomical object that is blocked from view. In the example of the ureter, the computer system may detect an exposed portion of the ureter and may determine, from that exposed portion, the direction and depth of the ureter. The computer system may use the determined direction and depth to add a virtual geometry (e.g., a virtual cone, virtual tube, etc.) to the video to indicate the direction and depth of the portions of the ureter that are blocked from view. As a result, the virtual geometry indicates to the doctor the positioning of the portions of the ureter that the doctor cannot see in the video or image stream.

[0022] In certain embodiments, the computer system provides several technical advantages. For example, computer system adds virtual geometries (e.g., a virtual plane) to the video or image stream that helps the doctor stay oriented even when the view in the video or image stream changes. As another example, the computer system adds virtual geometries (e.g., virtual cone, virtual tube, etc.) to the video or image stream that indicates the position of anatomical objects that the doctor cannot see, which helps the doctor to avoid unintentionally cutting or damaging those anatomical objects.

[0023] The described computer system may be implemented as part of a surgical system (e.g., a computer-assisted surgical system). Figure 1 shows an examplecomputer-assisted surgical system 100 that implements some of the features described herein.

[0024] The surgical system 100 includes a manipulator assembly 102, a user control apparatus 104, and an auxiliary apparatus 106, all of which are communicatively coupled to each other. The surgical system 100 is utilized by a medical team to perform a computer-assisted medical procedure or other similar operation on a body of a patient 108 or on any other body as may serve a particular implementation. The medical team includes a first user 110-1 (such as a surgeon for a surgical procedure), a second user 110-2 (such as a patient-side assistant), a third user 110-3 (such as another assistant, a nurse, a trainee, etc.), and a fourth user 110- 4 (such as an anesthesiologist for a surgical procedure), all of whom are collectively referred to as users 110, and each of whom may control, interact with, or otherwise be a user of the surgical system 100. More, fewer, or alternative users may be present during a medical procedure as may serve a particular implementation. For example, team composition for different medical procedures, or for non-medical procedures, may differ and include users with different roles.

[0025] Although Figure 1 illustrates an ongoing minimally invasive medical procedure such as a minimally invasive surgical procedure, it will be understood that the surgical system 100 may similarly be used to perform open medical procedures or other types of operations. For example, operations such as exploratory imaging operations, mock medical procedures used for training purposes, and / or other operations may also be performed.

[0026] The manipulator assembly 102 includes one or more manipulator arms 112 (e.g., manipulator arms 112-1 through 112-4) to which one or more instruments may be coupled. The instruments are used for a computer-assisted surgical procedure on the patient 108 (e.g., by being at least partially inserted into the patient 108 and manipulated within the patient 108). While the manipulator assembly 102 is depicted and described herein as including four manipulator arms 112, the manipulator assembly 102 may include a single manipulator arm 112 or any other number of manipulator arms as may serve a particular implementation. Although the example of Figure 1 illustrates the manipulator arms 112 as robotic manipulator arms, one or more instruments may be partially or entirely manually controlled, such as by beinghandheld and controlled manually by a person. These partially or entirely manually controlled instruments are used in conjunction with, or as an alternative to, computer- assisted instrumentation that is coupled to the manipulator arms 112.

[0027] During the medical operation, the user control apparatus 104 facilitates teleoperational control by the user 110-1 of the manipulator arms 112 and instruments attached to the manipulator arms 112. To this end, the user control apparatus 104 provides the user 110-1 with imagery of an operational area associated with the patient 108 as captured by an imaging device. The manipulator arms 112 or any instruments coupled to the manipulator arms 112 mimic the dexterity of the hand, wrist, and fingers of the user 110-1 across multiple degrees of freedom of motion. In this manner, the user 110-1 intuitively performs a procedure (e.g., an incision procedure, a suturing procedure, etc.) using one or more of the manipulator arms 112 or any instruments coupled to the manipulator arms 112.

[0028] The auxiliary apparatus 106 includes one or more computing devices that perform auxiliary functions in support of the procedure, such as providing insufflation, electrocautery energy, illumination or other energy for imaging devices, image processing, or coordinating components of the surgical system 100. The auxiliary apparatus 106 includes a display monitor 114 that displays one or more user interfaces, or graphical or textual information in support of the procedure. In some instances, the display monitor 114 is a touchscreen display that provides user input functionality. Augmented content provided by a region-based augmentation system may be similar to, or differ from, content associated with the display monitor 114 or one or more display devices in the operation area (not shown).

[0029] The manipulator assembly 102, user control apparatus 104, and auxiliary apparatus 106 are communicatively coupled one to another in any suitable manner. The manipulator assembly 102, user control apparatus 104, and auxiliary apparatus 106 may be communicatively coupled by way of control lines 116, which represent any wired or wireless communication link as may serve a particular implementation. To this end, the manipulator assembly 102, user control apparatus 104, and auxiliary apparatus 106 may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, and so forth.

[0030] Figure 2A illustrates an example manipulator assembly 102. As seen in Figure 2A, the manipulator assembly 102 includes a base 118, a manipulator arm 112- 1 , a manipulator arm 112-2, a manipulator arm 112-3, and a manipulator arm 112-4. Each manipulator arm 112-1 , 112-2, 112-3, and 112-4 is pivotably coupled to the base 118. Although the base 118 may include casters to allow ease of mobility, in some embodiments, the manipulator assembly 102 is fixedly mounted to a floor, ceiling, operating table, structural framework, or the like.

[0031] In a typical procedure, two of the manipulator arms 112-1 , 112-2, 112-3, or 112-4 hold surgical instruments and a third holds a stereo endoscope. The remaining manipulator arms are available so that other instruments may be introduced at the work site. Alternatively, the remaining manipulator arms may be used for introducing another endoscope or another image capturing device, such as an ultrasound transducer, to the work site.

[0032] Each of the manipulator arms 112-1 , 112-2, 112-3, and 112-4 are formed of links that are coupled together and manipulated through actuatable joints. Each of the manipulator arms 112-1 , 112-2, 112-3, and 112-4 may include a setup arm and a device manipulator. The setup arm positions its held device so that a pivot point occurs at its entry aperture into the patient. The device manipulator may then manipulate its held device so that the held device may be pivoted about the pivot point, inserted into and retracted out of the entry aperture, and rotated about its shaft axis. Each of the manipulator arms 112-1 , 112-2, 112-3, and 112-4 may include sensors (e.g., joint sensors, position sensors, accelerometers, etc.) that detect or track movement of the manipulator arms 112-1 , 112-2, 112-3, and 112-4. For example, these sensors may detect how far or how quickly a manipulator arm 112-1 , 112-2, 112-3, or 112-4 moves in a certain direction.

[0033] Figure 2B illustrates an example user control apparatus 104. The user control apparatus 104 includes a stereo vision display 120 so that the user may view the surgical work site in stereo vision from images captured by the stereoscopic camera of the manipulator assembly 102. Left and right eyepieces 122 and 124 are provided in the stereo vision display 120 so that the user may view left and right display screens inside the display 120 respectively with the user's left and right eyes. While viewing typically an image of the surgical site on a suitable viewer or display, thesurgeon performs the surgical procedures on the patient by manipulating master control input devices, which in turn control the motion of robotic instruments.

[0034] The user control apparatus 104 also includes left and right input devices 126 and 128 that the user grasps respectively with his / her left and right hands to manipulate devices (e.g., surgical instruments) being held by the manipulator arms 112-1 , 112-2, 112-3, and 112-3 of the manipulator assembly 102 in preferably six or more degrees of freedom (“DOF”). Foot pedals 130 with toe and heel controls are provided on the user control apparatus 104 so the user may control movement and / or actuation of devices associated with the foot pedals.

[0035] A processing device 132 is provided in the user control apparatus 104 for control and other purposes. The processing device 132 performs various functions in the surgical system 100. One function performed by processing device 132 is to translate and transfer the mechanical motion of input devices 126 and 128 to actuate their corresponding joints in their associated manipulator arms 112-1 , 112-2, 112-3, and 112-4 so that the surgeon can effectively manipulate devices, such as the surgical instruments. Another function of the processing device 132 is to implement the methods, crosscoupling control logic, and controllers or processors described herein. The auxiliary apparatus 106 may include a processing device 132 that performs the functions or actions described herein. The processing device 132 includes a processor and a memory that perform the functions described herein.

[0036] The processor may include any electronic circuitry, including, but not limited to one or a combination of microprocessors, microcontrollers, application specific integrated circuits (ASIC), application specific instruction set processor (ASIP), and / or state machines, that communicatively couples to a memory and controls the operation of the user control apparatus 104 and / or the auxiliary apparatus 106. The processor may be 8-bit, 16-bit, 32-bit, 64-bit or of any other suitable architecture. The processor may include an arithmetic logic unit (ALU) for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components. The processor may include other hardware that operates software to control and process information. The processor executes software stored on amemory to perform any of the functions described herein. The processor controls the operation and administration of the user control apparatus 104 or the auxiliary apparatus 106 by processing information (e.g., information received from the user control apparatus 104, the manipulator assembly 102, the auxiliary apparatus 106, and / or a memory). The processor is not limited to a single processing device and may encompass multiple processing devices contained in the same device or computer or distributed across multiple devices or computers. The processor is considered to perform a set of functions or actions if the multiple processing devices collectively perform the set of functions or actions, even if different processing devices perform different functions or actions in the set.

[0037] Figures 3A, 3B, and 4-7 illustrate example operations in the surgical system 100 of Figure 1. A computer system (which may be implemented in the user control apparatus 104 and / or the auxiliary apparatus 106 of the surgical system 100 using the processing device 132) performs the operations. Generally, the computer system adds virtual reference geometries to videos of anatomical structures.

[0038] As seen in Figure 3A, the computer system presents a video showing a top- down view of an anatomical structure 302. The video may be captured by a camera positioned on or in the body of a patient. The camera may be directed at the anatomical structure 302 in the patient. The computer system receives the video and displays the video of the anatomical structure 302.

[0039] The computer system adds a virtual plane 304 to the video of the anatomical structure 302 such that the virtual plane 304 is displayed along with the anatomical structure 302. The virtual plane 304 intersects with the anatomical structure 302. In some instances, the virtual plane 304 may be set at a depth that intersects with a portion of the anatomical structure 302. As seen in Figure 3A, the virtual plane 304 intersects with certain portions of the anatomical structure 302 such that portions of the virtual plane 304 are positioned over the anatomical structure 302 and such that certain portions of the anatomical structure 302 extend above the virtual plane 304.

[0040] The virtual plane 304 is defined using a point 306 and a vector 308. The point 306 may be selected by an operator of the computer system. The point 306 may be selected or set at a particular depth in the video of the anatomical structure 302.The vector 308 may be a normal for the plane 304 (e.g., orthogonal to the plane 304). As a result, the point 306 may set the depth of the virtual plane 304, and the vector 308 may set the direction or orientation of the virtual plane 304. In some embodiments, the point 306 and / or the vector 308 are set automatically by the computer system. For example, the computer system may set the point 306 to be on the surface of the anatomical structure. As a result, the virtual plane 304 may be set tangential to a point on the surface of the anatomical structure. The computer system may set the vector 308 to extend parallel with the direction of view of the camera that captured the video of the anatomical structure 302. As a result, the computer system may automatically set the direction or orientation of the virtual plane 304 such that the virtual plane 304 faces the camera.

[0041] In some embodiments, the computer system determines the depth of the point 306 using a depth map of the anatomical structure 302. The depth map indicates the depths of different points on the anatomical structure 302. When the point 306 is selected, the computer system determines the location (e.g., the coordinates) of the point 306 on the anatomical structure 302. The computer system then determines the corresponding location on the depth map and determines the depth of that corresponding location (or the depth of a point on the depth map closest to that corresponding location). The computer system then uses that determined depth as the depth for the point 306.

[0042] In particular embodiments, the operator of the computer system adjusts the position, direction, or orientation of the virtual plane 304. For example, the operator of the system may use controllers (e.g., input devices 126 and 128) to adjust the depth of the point 306, which adjusts the depth of the virtual plane 304. As another example, the operator of the computer system may use the controllers to change the direction or orientation of the vector 308, which adjusts the direction or orientation of the virtual plane 304. In this manner, the operator of the system sets the virtual plane 304 at any depth and with any direction or orientation.

[0043] The virtual plane 304 serves as a reference plane that is fixed at the depth, direction, and orientation set for the virtual plane 304. As a result, when the camera that captures the video of the anatomical structure 302 moves so that the view provided by the camera changes, the virtual plane 304 is maintained at its depth,direction, and orientation to help the operator of the computer system from becoming disoriented. As seen in Figure 3B, the view of the anatomical structure 302 has changed from the top-down view shown in Figure 3A to a side view. This change in view may have occurred because the camera that captures the video of the anatomical structure 302 has rotated or moved to the side of the anatomical structure 302. In the example of Figure 3B, the virtual plane 304 remains at the depth, direction, and orientation set for the virtual plane 304 shown in Figure 3A. In this manner, the depth of the virtual plane 304 is maintained, even though the view provided by the camera has changed.

[0044] The computer system uses any suitable information or procedure to render the virtual plane 304 after the camera that captures the video has moved. For example, if the camera moves but the anatomical structure 302 remains in view, then the computer system may use a simultaneous localization and mapping (SLAM) process to generate a virtual map of the anatomical structure 302 and to place the virtual plane 304 in the virtual map. The computer system then renders the virtual plane 304 in the video of the anatomical structure 302 according to the virtual map, so that the virtual plane 304 is maintained at a particular depth and orientation, even though the camera has moved.

[0045] As another example, if the camera moves so that the anatomical structure 302 does not remain in view, then the computer system may use kinematic sensor information to render the virtual plane 304. For example, if the camera moves to another portion of the patient’s body, then the anatomical structure 302 may not remain in view, and a SLAM process may not be able to provide the computer system with sufficient information to render the plane 304 at the proper depth and orientation. In these instances, the computer system may use sensor information from the camera (e.g., information from kinematic sensors positioned on or near the camera) to determine how the camera has moved. For example, the sensor information may indicate a distance and direction traveled by the camera. The computer system uses the sensor information to determine the new location, orientation, and / or pose of the camera. The computer system then uses this information to render the virtual plane 304 at the proper depth and orientation in the new view provided by the camera, so that the virtual plane 304 is maintained at the same depth and orientation that was setfor the virtual plane 304. In this manner, the computer system continues to render a virtual plane 304 that the operator of the computer system uses as a reference plane to remain oriented, even when the camera moves and provides a different view. For example, the virtual plane 304 may help the operator of the computer system understand where a particular depth appears in a video, even when the view in the video changes due to movement of the camera.

[0046] Figures 3C, 3D, and 3E illustrate an example of adding a virtual geometry to a video 310. As seen in Figure 3C, the video 310 shows an anatomical structure 312 and a medical instrument 314. The computer system places a point 316 in the video 310. The point 316 may be positioned at any depth. For example, the point 316 may be positioned on the surface of the anatomical structure 312. As another example, the point 316 may be positioned beneath the surface of the anatomical structure 312 or above the surface of the anatomical structure 312. The computer system also extends a vector 318 from the point 316. The vector 318 may extend in any direction. For example, the vector 318 may extend towards or away from a camera capturing the video 310. In the example of Figure 3C, the vector 318 extends into the video 310 and away from the camera along a direction of view of the camera.

[0047] As seen in Figure 3D, the computer system adds a virtual plane 320 to the video 310. The virtual plane 320 is set at a depth according to the point 316. For example, the virtual plane 320 may be set at the same depth as the point 316. As another example, the virtual plane 320 may be set at a depth that is offset from the depth of the point 316 (e.g., the virtual plane 320 may be offset from the depth of the point 316 by one centimeter). Additionally, the orientation of the virtual plane 320 is set according to the vector 318. For example, the orientation of the virtual plane 320 may be set such that the vector 318 is normal to the virtual plane 320. Based on the depth and orientation of the virtual plane 320, the virtual plane 320 will intersect the anatomical structure 312 and the medical instrument 314 at certain points. As a result, some portions of the anatomical structure 312 and the medical instrument 314 will be positioned beneath the virtual plane 320, and some portions of the anatomical structure 312 and the medical instrument will be positioned above the virtual plane 320.

[0048] As seen in Figure 3E, when the view in the video 310 changes (e.g., due to movement of the camera capturing the video 310), the computer system continues presenting the virtual plane 320 at the same depth and orientation in the changed view. As a result, the virtual plane 320 serves as a reference plane that indicates a particular depth to the operator of the computer system. The operator may view the virtual plane 320 in the changed view to understand where a particular depth appears in the changed view.

[0049] Figure 4 illustrates an example operation 400 in the surgical system 100 of Figure 1. Generally, Figure 4 shows the computer system adding a virtual geometry (e.g., a virtual plane) to a video. The virtual geometry may be set at a particular depth or orientation in the video, and when the view in the video changes (e.g., due to movement of a camera capturing the video), the computer system continues to render and display the virtual geometry at the set depth and / or orientation. In this manner, the virtual geometry helps the operator of the computer system stay orientated when the view in the video changes.

[0050] The computer system receives the video 402. The video 402 is captured by a camera. In some instances, the video 402 shows an anatomical structure within the body of a patient. The camera captures the video 402 when the camera is positioned within the body of the patient next to the anatomical structure. The computer system displays the video 402 (e.g., using the stereo vision display 120 or the display system 210) so that the operator may view the anatomical structure while operating on or near the anatomical structure.

[0051] The computer system receives a point 404 and / or a vector 406. The operator of the computer system may select the point 404 using a controller (e.g., the input devices 126 and 128 or the operator input 206). For example, the operator of the computer system may use the controller to set or select a coordinate in the video 402 as the point 404. The operator may also set a depth for the point 404. The operator may also move the controller to move the point 404 after adding the point to the video 402. When the point 404 is at the desired position or location, the operator uses the controller to indicate selection of the point 404 at that position or location. The vector 406 may originate from the point 404. The operator of the computer system may also set a direction or orientation for the vector 406. For example, the operatorof the computer system may use the controller to set the direction or orientation of the vector 406 extending from the point 404. In some embodiments, the computer system automatically sets an initial direction or orientation for the vector 406. For example, the computer system may set the vector 406 to extend directly into the camera. The operator of the computer system may then adjust or change the direction or orientation of the vector 406.

[0052] The computer system then renders a virtual geometry 408 using the point 404 and the vector 406. For example, the computer system may render a virtual plane that is set at the depth for the point 404 and that is oriented with the vector 406 as the normal to the plane. The computer system then renders the virtual geometry 408 in the video 402 such that the virtual geometry 408 is displayed along with the video 402. In certain embodiments, the computer system uses a depth map to determine the depth of the point 404. The depth map indicates the depths of different points on the anatomical structure. When the point 404 is set, the computer system determines the location of the point 404 on the anatomical structure. The computer system then determines the corresponding location on the depth map and the depth of the corresponding location. The determined depth is then used as the depth of the point 404.

[0053] The virtual geometry 408 may be any shape. For example, the computer system may render a virtual cone or cylinder using the point 404 and the vector 406. The computer system may set the base of the virtual cone or cylinder at the depth for the point 404. The computer system may also orient the virtual cone or cylinder along the direction of the vector 406 (e.g., such that the vector 406 is oriented along the height of the virtual cone or cylinder). As another example, the computer system may render a virtual sphere using the point 404 and the vector 406. The computer system may set the center of the virtual sphere as the point 404. The computer system may also set the radius of the sphere as the vector 406. The computer system may display the virtual cone, cylinder, sphere, etc. along with the video 402.

[0054] In some embodiments, the controller that the operator of the computer system uses to set the point 404 and / or the vector 406 is the same controller that the operator of the computer system uses to move medical instruments in the body of the patient. To avoid moving the medical instruments while setting the point 404 and / orthe vector 406, the operator of the computer system enters a separate mode (e.g., a clutch mode) in which the operator of the computer system moves the controller without moving the medical instruments. In this manner, the operator of the computer system moves the controllers to set the point 404 and / or the vector 406 without moving the medical instruments in the body of the patient, thereby avoiding injury to the patient.

[0055] In particular embodiments, the computer system uses the positioning or location of a medical instrument to set the point 404 and / or the vector 406. For example, the operator of the computer system may use a controller to move a medical instrument in the video 402. The operator may then use the controller to indicate selection of the point 404 or the vector 406. The computer system may then set the point 404 and / or the vector 406 at the position or location of the medical instrument.

[0056] In certain embodiments, the computer system uses additional information to render the virtual geometry 408. For example, the computer system may receive an image 410, which may be an ultrasound image. The image 410 may reveal structures within or underneath the anatomical structure shown in the video 402. The computer system may determine the depth of these structures within or underneath the anatomical structure in the video 402. The computer system may then set the depth of the point 404 at the depth of the structure shown in the image 410, and the computer system may extend the vector 406 from the point 404 at that depth. The resulting virtual geometry 408 is then set at the depth of the anatomical structure within or beneath the anatomical structure shown in the video 402.

[0057] After the virtual geometry 408 is set or rendered, the operator of the computer system may continue to adjust the virtual geometry 408. For example, the operator of the computer system may operate the controller to make an adjustment 412 to the virtual geometry 408. The computer system may receive the adjustment 412 as the operator of the computer system operates the controller. The computer system may then make the corresponding adjustment to the virtual geometry 408. For example, the computer system may translate or rotate the virtual geometry 408 according to the movement of the controller. If the virtual geometry 408 is a virtual plane, the computer system may adjust the depth of the virtual plane, move the virtual plane, or rotate the virtual plane, which changes the direction or orientation of thevirtual plane. In this manner, the operator of the computer system may adjust the position, direction or orientation of the virtual geometry 408.

[0058] The operator of the computer system may add any number of virtual geometries 408 to the video 402. The virtual geometries 408 may be any desired shape. For example, the operator of the computer system may add virtual planes, virtual cones, virtual spheres, virtual cylinders, virtual tubes, etc. to the video 402. These virtual geometries 408 may also be set at different poses (e.g., different positions, directions, and / or orientations). The operator of the computer system may also set the position, direction, and orientation of these virtual geometries 408 using the controller. These virtual geometries 408 may help the operator of the computer system from becoming disoriented when the camera capturing the video 402 moves.

[0059] The computer system detects a movement 414. The movement 414 causes the view shown in the video 402 to change. For example the movement 414 may be a movement of the camera capturing the video 402. As another example, the movement 414 may be a movement of the patient or the anatomical structure in the video 402. The computer system tracks the movement 414 to determine the change in the view in the video 402. The video 402’ is the video 402 with the change in view. The computer system then renders the virtual geometry 408 in the video 402’ so that the virtual geometry 408 is maintained at the same pose (e.g., the same position, direction, and / or orientation) in the video 402’. For example, the position of the virtual geometry 408 in the video 402’ may still be set according to the point 404, and the direction or orientation of the virtual geometry in the video 402’ may still be set according to the vector 406. Because the point 404 and the vector 406 do not change positions, directions, or orientations as a result of the movement 414, the virtual geometry 408 is maintained at the same pose despite the movement 414.

[0060] As an example, the computer system may add a virtual plane to the video 402. The depth of the virtual plane may be set according to the point 404, and the direction or orientation of the virtual plane may be set according to the vector 406. The movement 414 may cause the view in the video 402 to change. The computer system may determine the movement 414 that occurred to determine the change in the view in the video 402’. For example, the computer system may use a SLAM process or kinematics information to determine the movement 414 and / or a change in the poseof the camera capturing the video 402. The computer system may then render the virtual plane in the video 402’ so that the virtual plane is maintained at the same pose that the virtual plane had prior to the movement 414. In this manner, the operator of the computer system may continue viewing the video 402’ to see the virtual plane. The virtual plane may continue to indicate the same depth, direction, and / or orientation that was set for the virtual plane despite the change in the view in the video 402’, which helps the operator of the computer system stay oriented.

[0061] The computer system may use any suitable process for tracking the movement 414. For example, the computer system may use a SLAM process to determine or track the movement 414 when the video 402’ continues to show at least a portion of the anatomical structure after the movement 414. As another example, the computer system may use sensor data (e.g., kinematic sensor information) if the movement 414 removes the anatomical structure from view in the video 402’. The sensor information may indicate to the computer system the distance and direction traveled by the camera. Additionally, the kinematic sensor information may indicate to the computer system the pose of the camera. The computer system may use this information to determine the change in view in the video 402’ and to render the virtual geometry 408 at the same depth, direction, and orientation.

[0062] During the SLAM process, the computer system may stitch together various frames of the video 402 to form a virtual map of the anatomical structure 302. The computer system may also add the virtual geometry 408 to the virtual map. When the movement 414 occurs, the computer system may compare the changed view in the video 402’ to the virtual map of the anatomical structure to determine the movement 414 or the updated pose of the camera. The computer system may then render the virtual geometry 408 in the video 402’ as indicated in the virtual map.

[0063] Figure 5 illustrates an example operation 500 in the surgical system 100 of Figure 1 . Generally, Figure 5 shows the computer system using the virtual geometry 408 to guide the movement of medical instruments. In this manner, the computer system improves the safety of a procedure.

[0064] The computer system receives or determines an instrument position 502.The instrument position 502 indicates the position of a medical instrument in the bodyof the patient. The instrument position 502 may include coordinates that indicate the position of the medical instrument. The computer system tracks the position or movement of the medical instrument using the instrument position 502. As the operator of the computer system moves the instrument (e.g., using the input devices 126 and 128), the instrument position 502 updates.

[0065] The computer system compares the distance between the instrument position 502 and the virtual geometry 408 to a margin 504. When the distance between the instrument position 502 and the virtual geometry 408 falls below the margin 504, the computer system determines that the instrument position 502 is too close to the virtual geometry 408. The computer system then generates and communicates an alert 506 indicating that the medical instrument is too close to the virtual geometry 408. In some embodiments, the computer system then prevents or resists the movement of the medical instrument closer to the virtual geometry 408. In an example, if the virtual geometry 408 is a virtual plane set at a depth beyond which the operator of the computer system does not wish for a medical instrument to be moved, then the computer system monitors the position of the medical instrument and generates and communicates the alert 506 when the medical instrument is too close to the virtual plane. The computer system also prevents or resists the movement of the medical instrument any closer towards the virtual plane. In some embodiments, the computer system also presents or displays the distance between the medical instrument and the virtual geometry 408, which indicates to the doctor how much further the medical instrument may be moved without reaching the virtual geometry 408. In this manner, the computer system improves the health and safety of the patient during a procedure.

[0066] The computer system may also use virtual geometries to indicate the locations and directions of anatomical structures that are not visible in the video. Figures 6A and 6B illustrate example operations 600 and 609 in the surgical system 100 of Figure 1. Generally, Figures 6A and 6B show the computer system adding virtual geometries to indicate the positioning and direction of portions of an anatomical structure (e.g., a ureter, a blood vessel, a nerve, a pedicle, etc.) that are not visible.

[0067] As seen in Figure 6A, the computer system receives a video in which a portion 602 of an anatomical structure 604 is visible. Other portions of the anatomicalstructure 604 may not be visible, because they are covered or obstructed from view. In the example of Figure 6A, the portion 602 is visible, and other portions of the anatomical structure 604 are covered or obstructed from view by another anatomical structure 606. The computer system analyzes the portion 602 to determine the positioning and direction of the portions of the anatomical structure 604.

[0068] The computer system then adds a virtual geometry 608 to the video to indicate the positioning and direction of the portions of the anatomical structure 604 that are obstructed from view. In the example of Figure 6A, the computer system analyzes the positioning and direction of the portion 602 to determine the positioning and direction of the virtual geometry 608. For example, the computer system determines a depth of the portion 602 (e.g., using a depth map) and sets a point 404 according to that depth. The point 404 may be set at a position where the anatomical structure 604 becomes obstructed from view (e.g., a point at the boundaries of the anatomical structure 604 and the anatomical structure 606). The computer system also determines a direction or orientation of the portion 602 and set a vector 406 extending from the point 404 along that direction or orientation. The computer system then renders the virtual geometry 608 according to the point 404 and the vector 406.

[0069] In the example of Figure 6A, the computer system renders a virtual cone with a tip set at the point 404 and with the vector 406 set as the center axis of the cone. In some embodiments, the radius of the virtual cone is set to indicate a range of uncertainty about the positioning and direction of the portion of the anatomical structure 604 that is obstructed from view. The computer system then displays the virtual cone along with the video to indicate a position and direction of the portion of the anatomical structure 604 that is obstructed from view by the anatomical structure 606. The video may then make it easier for the operator of the computer system to avoid accidentally cutting or damaging the portions of the anatomical structure 604 that are obstructed from view, which improves the health and safety of the patient.

[0070] In the example of Figure 6B, the computer system receives a video in which portions 610A and 610B of an anatomical structure 612 are visible. Other portions of the anatomical structure 612 may not be visible, because they are covered or obstructed from view by anatomical structures 614 and 616. The computer systemanalyzes the visible portions 610A and 61 OB to determine the positioning and direction of the portions of the anatomical structure 612 that are obstructed from view.

[0071] The computer system then adds virtual geometries 618A and 618B to the video to indicate the positioning and direction of the portions of the anatomical structure 612 that are obstructed from view. In the example of Figure 6B, the computer system analyzes the positioning and direction of the portions 610A and 61 OB to interpolate the positioning and direction of the portion of the anatomical structure 612 that extends between the portions 610A and 61 OB. The computer system then renders and adds the virtual geometry 618A such that the virtual geometry 618A connects to the portions 610A and 61 OB.

[0072] The computer system may analyze the positioning and direction of the portion 610B to determine a positioning and direction of the anatomical structure 612 that follows the portion 610B. The computer system may then add the virtual geometry 618B such that the virtual geometry 618B connects to the portion 610B. The operator of the computer system may view the video that shows the portions 610A and 610B and the virtual geometries 618A and 618B to gain an understanding of the positioning and direction of the anatomical structure 612, especially the portions of the anatomical structure 612 that are obstructed from view. The video may then make it easier for the operator of the computer system to avoid accidentally cutting or damaging the portions of the anatomical structure 612 that are obstructed from view, which improves the health and safety of the patient.

[0073] Figures 6C and 6D illustrate an example of adding a virtual geometry to a video 620. As seen in Figure 6C, the video 620 shows an anatomical structure 622, a medical instrument 624, and another anatomical structure 626. The anatomical structure 626 covers a portion of the anatomical structure 622. The computer system places a point 628 in the scene. The point 628 is positioned at a point where the anatomical structure 626 covers the anatomical structure 622 and at a depth of the anatomical structure 622. As a result, the anatomical structure 626 covers the point 628. The computer system also extends a vector 630 from the point 628. The vector 630 extends in the direction of the anatomical structure 622. In some embodiments, the computer system analyzes the video 620 to determine the depth and direction of the anatomical structure 622. The computer system then adds the point 628 at thedepth of the anatomical structure 622 and extends the vector 630 in the direction of the anatomical structure 622. Both the point 628 and the vector 630 may be covered by the anatomical structure 626.

[0074] As seen in Figure 6D, the computer system adds a virtual cone 632 to the video 620. The point 628 is the tip of the virtual cone 632, and the virtual cone 632 extends along the direction of the vector 630. As a result, the virtual cone 632 indicates a direction and orientation of a portion of the anatomical structure 622 that is beneath the anatomical structure 626. The operator of the computer system may view the virtual cone 632 to understand the positioning of the portion of the anatomical structure 622 even though the portion of the anatomical structure 622 is not visible in the video 620. The operator may then more easily avoid cutting or damaging the portion of the anatomical structure 622 during an operation.

[0075] Figure 7 illustrates an example operation 700 in the surgical system 100 of Figure 1 . The computer system receives a video 702, which may show an anatomical structure (e.g., a ureter, a blood vessel, a nerve, a pedicle, etc.). Specifically, the video 702 shows the portions 704A and 704B of the anatomical structure. Some portions of the anatomical structure may be obstructed from view in the video 702.

[0076] The computer system analyzes the video and, specifically, the portions 704A and 704B of the anatomical structure appearing in the video 702, to determine a direction 705 and a depth 706 of the anatomical structure in the video 702. For example, the computer system may determine the positions of the portions 704A and 704B to interpolate the direction 705 of the anatomical structure. The direction 705 of the anatomical structure may align with the portions 704A and 704B. As another example, the computer system may determine the depths of the portions 704A and 704B of the anatomical structure. The computer system then determines the depth 706 of the anatomical structure according to the determined depths of the portions 704A and 704B. The determined depth 706 may be the same as the determined depths of the portions 704A and 704B, or the depth 706 may be an average of the depth of the portions 704A and 704B.

[0077] In some embodiments, the computer system uses other information to determine the direction 705 and depth 706 of the anatomical structure. For example,the computer system may analyze a pre-operative scan of the anatomical structure to determine the direction 705 and depth 706 of the anatomical structure. The preoperative scan may show the internal structures of the anatomical structure or the portions of the anatomical structure that are obstructed from view, which may help in determining the direction 705 and the depth 706 of the anatomical structure. As another example, the computer system may use a depth map of the anatomical structure to determine the depth 706.

[0078] The computer system then renders and adds a virtual geometry 708 to the video 702. The virtual geometry 708 may be a virtual cone or a virtual tube that is consistent with the shape of the anatomical structure. The computer system positions the virtual geometry 708 at the depth 706. Additionally, the computer system orients or directs the virtual geometry 708 to be consistent with the direction 705. The virtual geometry 708 may connect to one or more of the portions 704A and 704B shown in the video 702. As a result, the virtual geometry 708 indicates the positioning and direction of a portion of the anatomical structure that is obstructed from view in the video 702. The virtual geometry 708 indicates to the operator of the computer system the positioning and direction of portions of the anatomical structure that are obstructed from view, which may help the operator avoid cutting or damaging these portions of the anatomical structure during an operation.

[0079] Figure 8 is a flowchart of example method 800 performed by a computer system in the surgical system 100 of Figure 1. By performing the method 800, the computer system adds a virtual geometry 408 to a video 402. The computer system maintains the depth, direction, and / or orientation of the virtual geometry 408, even when a camera capturing the video 402 moves and changes the view in the video 402. As a result, the virtual geometry 408 may help an operator from the computer system from becoming disoriented due to the movement of the camera.

[0080] In block 802, the computer system receives the video 402. The video 402 is captured by a camera positioned in a body of a patient. The video 402 shows an anatomical structure of the patient. The computer system displays the video 402 to the operator of the computer system during a medical procedure.

[0081] In block 804, the computer system adds the virtual geometry 408 to the video 402. The computer system receives a point 404 and / or a vector 406 that may be used to render and add the virtual geometry 408 to the video 402. The operator of the computer system may set the point 404 at a desired depth. Additionally, the operator of the computer system may set the direction or orientation of the vector 406 extending from the point 404. Using the point 404 and the vector 406, the computer system renders the virtual geometry 408 and adds the virtual geometry 408 to the video 402. The virtual geometry 408 may be set at the depth of the point 404 and may be directed or oriented according to the vector 406. For example, if the virtual geometry 408 is a virtual plane, then the virtual plane may be set at the depth of the point 404, and the virtual plane may have a normal vector oriented along the vector 406.

[0082] In some embodiments, the computer system determines the depth of the point 404 using a depth map of the anatomical structure. The depth map may indicate the depths of different points on the anatomical structure. When the point 404 is set, the computer system determines the location (e.g., the coordinates) of the point 404 on the anatomical structure. The computer system may then determine the corresponding location on the depth map and determine the depth of that corresponding location (or the depth of a point on the depth map closest to that corresponding location). The computer system may then use that determined depth as the depth for the point 404.

[0083] In block 806, the computer system detects a movement 414. The movement 414 may cause the view in the video 402 to change. For example, the movement 414 may change the perspective of the anatomical structure shown in the video 402, or the movement 414 may remove the anatomical structure from view. The computer system may detect and track this movement 414 using any suitable process. For example, the computer system may use a SLAM process to determine how the pose of the camera has changed and / or how the view in the video 402 has changed. As another example, the computer system may use kinematic sensor information to determine the distance and the direction moved by the camera, which the computer system may use to determine an updated pose for the camera.

[0084] In block 808, the computer system renders and presents the virtual geometry 408 in the new view in the video 402’. The computer system may use the detected movement 414 to determine how the virtual geometry 408 should be positioned, directed, and / or oriented in the video 402’. In certain embodiments, the computer system presents the virtual geometry 408 in the new view in the video 402’ such that the virtual geometry 408 maintains the same depth, direction, and / or orientation as the virtual geometry 408 had prior to the movement 414 of the camera. In this manner, when the operator of the computer system sees the new view in the video 402’, the operator may also see the virtual geometry 408, which is maintained at the same depth, direction, and orientation. Thus, the virtual geometry 408 may help prevent the operator from becoming disoriented as a result of the movement 414 of the camera and the change in view.

[0085] Figure 9 is a flowchart of an example method 900 performed by a computer system in the surgical system 100 of Figure 1. By performing the method 900, the computer system adds a virtual geometry 708 to a video 702 to indicate the positioning and directions of anatomical structures that are obstructed from view in the video 702.

[0086] In block 902, the computer system receives the video 702. The video 702 shows portions 704A and 704B of an anatomical structure. Other portions of the anatomical structure may be hidden or obstructed from view in the video 702.

[0087] In block 904, the computer system detects the portion 704A of the anatomical structure in the video 702. The computer system may use a computer vision process to detect the portion 704A of the anatomical structure in the video 702.

[0088] The computer system determines the direction 705 and the depth 706 of the anatomical structure in block 906. The computer system may analyze the detected portion 704A of the anatomical structure to determine the direction 705 and the depth 706 of the anatomical structure. For example, the direction 705 may be aligned with the direction of the portion 704A of the anatomical structure, and the depth 706 may be the same or within a margin of the depth of the portion 704A of the anatomical structure. In some embodiments, when multiple portions of the anatomical structure are shown in the video 702, the computer system may determine the direction 705 and the depth 706 according to the multiple portions. For example, the direction 705may be interpolated using the directions of the multiple portions, and the depth 706 may be the same or an average of the depths of the multiple portions of the anatomical structure.

[0089] In block 908, the computer system adds the virtual geometry 708 to the video 702. The computer system may render the virtual geometry 708 according to the determined direction 705 and depth 706. For example, the virtual geometry 708 may have the same direction 705 and the same depth 706. The computer system may then add the virtual geometry 708 with the direction 705 and the depth 706 to the video 702. When the operator of the computer system views the video 702, the virtual geometry 708 may indicate to the operator the positioning and direction of the portion of the anatomical structure that is hidden or obstructed from view. It may then be easier for the operator to avoid cutting or damaging the portion of the anatomical structure, which improves the health and safety of the patient.

[0090] In summary, the computer system assists the doctor during a surgical procedure by adding virtual geometries to the view provided to the doctor. For example, the computer system may add one or more virtual planes to the video or image stream. Each plane may serve as a reference plane during the procedure. A plane may be set with a particular orientation and at a certain depth in the body (e.g., a depth as viewed from the visual plane of the video or image stream). When the camera moves to a different position, the view provided by the video or image stream may change. The computer system may maintain the orientation and depth of the plane even though the view in the video or image stream has changed. As a result, the plane may continue to mark a particular orientation or depth in the body even though the view in the video or image stream has changed.

[0091] As another example, the computer system may add a virtual geometry to the video or image stream to indicate the location of an anatomical object that is blocked from view. In the example of the ureter, the computer system may detect an exposed portion of the ureter and may determine, from that exposed portion, the direction and depth of the ureter. The computer system may use the determined direction and depth to add a virtual geometry (e.g., a virtual cone, virtual tube, etc.) to the video to indicate the direction and depth of the portions of the ureter that are blocked from view. As a result, the virtual geometry indicates to the doctor thepositioning of the portions of the ureter that the doctor cannot see in the video or image stream.

[0092] This description and the accompanying drawings that illustrate aspects, embodiments, or modules should not be taken as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure other features. Like numbers in two or more figures represent the same or similar elements.

[0093] In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.

[0094] Further, the terminology in this description is not intended to be limiting. For example, spatially relative terms-such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like may be used to describe one element’s or feature’s relationship to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass different positions (i.e., locations) and orientations (i.e., rotational placements) of the elements or their operation in addition to the position and orientation shown in the figures. For example, if the content of one of the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features. Thus, the exemplary term “below” can encompass both positions and orientations of above and below. A device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.Likewise, descriptions of movement along and around various axes include various special element positions and orientations. In addition, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context indicates otherwise. And, the terms “comprises”, “comprising”, “includes”, and the like specify the presence of stated features, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

[0095] Elements described in detail with reference to one embodiment, or module may, whenever practical, be included in other embodiments, or modules in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, or application may be incorporated into other embodiments, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or embodiments non-functional, or unless two or more of the elements provide conflicting functions.

[0096] In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0097] This disclosure describes various devices, elements, and portions of computer-assisted devices and elements in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an element or a portion of an element in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian x-, y-, and z-coordinates). As used herein, the term “orientation” refers to the rotational placement of an element or a portion of an element (three degrees of rotational freedom - e.g., roll, pitch, and yaw). As used herein, the term “shape” refers to a set positions or orientations measured along an element. As used herein, and for a device with repositionable arms, the term “proximal” refers to adirection toward the base of the computer-assisted device along its kinematic chain and “distal” refers to a direction away from the base along the kinematic chain.

[0098] Aspects of this disclosure are described in reference to computer-assisted systems and devices, which may include systems and devices that are teleoperated, remote-controlled, autonomous, semiautonomous, robotic, and / or the like. Further, aspects of this disclosure are described in terms of an embodiment using a medical system, such as the DA VINCI SURGICAL SYSTEM or ION SYSTEM commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that aspects disclosed herein may be embodied and implemented in various ways, including robotic and, if applicable, non-robotic embodiments. Techniques described with reference to surgical instruments and surgical methods may be used in other contexts. Thus, the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperational systems. As further examples, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, and / or the like. Additional example applications include use for procedures on tissue removed from human or animal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.

[0099] Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. Thus, the scope of the disclosure should be limited only by the following claims, and it is appropriate that the claims be construed broadly and, in a manner, consistent with the scope of the embodiments disclosed herein.

Claims

WHAT IS CLAIMED IS:1 . A computer system for adding a virtual reference geometry to a video of an anatomical structure, the computer system comprising: a memory; and a processor communicatively coupled to the memory, the processor configured to: receive the video of the anatomical structure, wherein the video shows a first view; receive a selection of a point and a vector in the first view; add a first virtual geometry to the first view based on the point and the vector; detect a movement that causes the first view to change to a second view; and present the first virtual geometry in the second view such that a pose of the first virtual geometry is maintained from the first view to the second view.

2. The computer system of Claim 1 , wherein the processor is further configured to use a simultaneous localization and mapping process to determine a position of the first virtual geometry in the second view.

3. The computer system of Claim 1 , wherein the processor is further configured to use kinematics data to determine a position of the first virtual geometry in the second view.

4. The computer system of Claim 1 , wherein the processor is further configured to rotate the first virtual geometry based on a detected rotation of a controller.

5. The computer system of Claim 1 , wherein the processor is further configured to adjust a depth of the first virtual geometry based on input from a controller.

6. The computer system of Claim 1 , wherein the processor is further configured to add a second virtual geometry to the first view, wherein the second virtual geometry has a different pose than the first virtual geometry.

7. The computer system of Claim 1 , wherein the processor is further configured to determine a depth of the first virtual geometry based on an ultrasound image of the anatomical structure.

8. The computer system of Claim 7, wherein the video shows that a portion of the anatomical structure is obstructed from view, and wherein the ultrasound image shows the portion of the anatomical structure.

9. The computer system of Claim 7, wherein the depth of the first virtual geometry is beneath at least a portion an outer surface of the anatomical structure shown in the video.

10. The computer system of Claim 1 , wherein the processor is further configured to determine when a medical instrument has moved within a margin of the first virtual geometry.

11. The computer system of Claim 10, wherein the processor is further configured to present an alert in response to the medical instrument moving within the margin of the first virtual geometry.

12. The computer system of Claim 10, wherein the processor is further configured to present a distance between the medical instrument and the first virtual geometry.

13. The computer system of Claim 1 , wherein the first virtual geometry comprises a virtual plane, and wherein maintaining the pose of the first virtual geometry comprises maintaining a depth of the virtual plane from the first view to the second view.

14. The computer system of Claim 1 , wherein the point is positioned on a surface of the anatomical structure, and wherein the first virtual geometry is tangential to the surface of the anatomical structure.

15. The computer system of Claim 14, wherein the selection of the point is made based on a position of a camera that captured the video.

16. The computer system of any of Claim 15, wherein a normal of the first virtual geometry is directed according to a direction of the camera.

17. The computer system of Claim 14, wherein the selection of the point comprises moving the point based on movement of a controller and selecting the point based on input from the controller.

18. The computer system of Claim 17, wherein the processor is further configured to refrain from moving a medical instrument during the movement of the controller.

19. The computer system of Claim 17, wherein the selection of the point comprises moving a medical instrument based on movement of the controller and selecting the point based on the position of the medical instrument.

20. The computer system of Claim 1 , wherein presenting the first virtual geometry in the second view comprises maintaining the point and the vector in the second view and rendering the first virtual geometry in the second view based on the point and the vector.

21. The computer system of Claim 1 , wherein the processor is further configured to: detect a first portion of the anatomical structure in the video; and determine a direction of the anatomical structure based on the first portion of the anatomical structure in the video.

22. The computer system of Claim 21 , wherein the first virtual geometry comprises at least one of a virtual tube or a virtual cone, and wherein the first virtual geometry is added to the first view based on the determined direction of the first portion of the anatomical structure and based on a boundary at which the anatomical structure becomes obstructed from view in the video.

23. The computer system of Claim 21 , wherein the direction of the anatomical structure is determined further based on a pre-operative scan of the anatomical structure.

24. The computer system of Claim 21 , wherein the processor is further configured to detect a second portion of the anatomical structure in the video, wherein the direction of the anatomical structure is determined further based on the second portion of the anatomical structure in the video.

25. The computer system of Claim 21 , wherein the anatomical structure is at least one of a ureter, a blood vessel, a nerve, or a pedicle.

26. A method for adding a virtual reference geometry to a video of an anatomical structure, the method comprising: receiving the video of the anatomical structure, wherein the video shows a first view; receiving a selection of a point and a vector in the first view; adding a first virtual geometry to the first view based on the point and the vector; detecting a movement that causes the first view to change to a second view; and presenting the first virtual geometry in the second view such that a pose of the first virtual geometry is maintained from the first view to the second view.

27. The method of Claim 26, further comprising using a simultaneous localization and mapping process to determine a position of the first virtual geometry in the second view.

28. The method of Claim 26, further comprising using kinematics data to determine a position of the first virtual geometry in the second view.

29. The method of Claim 26, further comprising rotating the first virtual geometry based on a detected rotation of a controller.

30. The method of Claim 26, further comprising adjusting a depth of the first virtual geometry based on input from a controller.

31. The method of Claim 26, further comprising adding a second virtual geometry to the first view, wherein the second virtual geometry has a different pose than the first virtual geometry.

32. The method of Claim 26, further comprising determining a depth of the first virtual geometry based on an ultrasound image of the anatomical structure.

33. The method of Claim 32, wherein the video shows that a portion of the anatomical structure is obstructed from view, and wherein the ultrasound image shows the portion of the anatomical structure.

34. The method of Claim 32, wherein the depth of the first virtual geometry is beneath an outer surface of the anatomical structure shown in the video.

35. The method of Claim 26, further comprising determining when a medical instrument has moved within a margin of the first virtual geometry.

36. The method of Claim 35, further comprising presenting an alert in response to the medical instrument moving within the margin of the first virtual geometry.

37. The method of Claim 35, further comprising presenting a distance between the medical instrument and the first virtual geometry.

38. The method of Claim 26, wherein the first virtual geometry comprises a virtual plane, and wherein maintaining the pose of the first virtual geometry comprises maintaining a depth of the virtual plane from the first view to the second view.

39. The method of Claim 26, wherein the point is positioned on a surface of the anatomical structure, and wherein the first virtual geometry is tangential to the surface of the anatomical structure.

40. The method of Claim 39, wherein the selection of the point is made based on a position of a camera that captured the video.41 . The method of Claim 40, wherein a normal of the first virtual geometry is directed according to a direction of the camera.

42. The method of any Claim 39, wherein the selection of the point comprises moving the point based on movement of a controller and selecting the point based on input from the controller.

43. The method of Claim 42, further comprising refraining from moving a medical instrument during the movement of the controller.

44. The method of Claim 42, wherein the selection of the point comprises moving a medical instrument based on movement of the controller and selecting the point based on the position of the medical instrument.

45. The method of Claim 26, wherein presenting the first virtual geometry in the second view comprises maintaining the point and the vector in the second view and rendering the first virtual geometry in the second view based on the point and the vector.

46. The method of Claim 26, further comprising: detecting a first portion of the anatomical structure in the video; and determining a direction of the anatomical structure based on the first portion of the anatomical structure in the video.

47. The method of Claim 46, wherein the first virtual geometry comprises at least one of a virtual tube or a virtual cone, and wherein the first virtual geometry is added to the first view based on the determined direction of the first portion of the anatomical structure and based on a boundary at which the anatomical structure becomes obstructed from view in the video.

48. The method of Claim 46, wherein the direction of the anatomical structure is determined further based on a pre-operative scan of the anatomical structure.

49. The method of Claim 46, further comprising detecting a second portion of the anatomical structure in the video, wherein the direction of the anatomical structure is determined further based on the second portion of the anatomical structure in the video.

50. The method of Claim 46, wherein the anatomical structure is at least one of a ureter, a blood vessel, a nerve, or a pedicle.

51. A non-transitory machine-readable medium storing instructions for adjusting models of anatomical objects that, when executed by a processor, cause the processor to: perform the method of any of Claims 26 through 50.

Images