Computer-aided surgery using virtual anatomical objects

Abstract

This disclosure describes a computer system and method for adding virtual reference objects to a video of an anatomical structure. The computer system includes memory and a processor communicatively coupled to the memory. The processor receives a video of the anatomical structure. The video shows a first view. The processor also receives selections of points and vectors in the first view, adds a first virtual geometry to the first view based on the points and vectors, detects movement that changes the first view to a second view, and renders the first virtual geometry in the second view such that the pose of the first virtual geometry is maintained from the first view to the second view.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of co-pending U.S. Provisional Patent Application Serial No. 63 / 597,576, filed November 9, 2023. The aforementioned related patent application is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to surgical systems. Specifically, this disclosure relates to surgical systems that utilize virtual objects to assist during surgery. Background Technology

[0004] Doctors use computer-assisted surgical systems to perform procedures on patients, even remotely. These systems provide doctors with various views of the surgical site during the procedure. However, these views offer a limited perspective of the anatomical object. As the camera device capturing the view moves within the body, or as the doctor removes parts of the anatomical structure, the doctor may become disoriented and lose track of the camera device and other medical instruments' position within the body. Summary of the Invention

[0005] This disclosure describes a computer system and method for adding virtual reference objects to a video of an anatomical structure. According to one embodiment, the computer system includes a memory and a processor communicatively coupled to the memory. The processor receives a video of the anatomical structure. The video shows a first view. The processor also receives selections of points and vectors in the first view, adds a first virtual geometry to the first view based on the points and vectors, detects movement that changes the first view to a second view, and renders the first virtual geometry in the second view such that the pose of the first virtual geometry is maintained from the first view to the second view.

[0006] According to another embodiment, the method includes receiving a video of an anatomical structure. The video shows a first view. The method further includes: receiving a selection of points and vectors in the first view; adding a first virtual geometry to the first view based on the points and vectors; detecting movement that changes the first view to a second view; and presenting the first virtual geometry in the second view such that the pose of the first virtual geometry is maintained from the first view to the second view. Other embodiments include a non-transitory machine-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method.

[0007] The foregoing general description and the following detailed description are exemplary and illustrative in nature and are intended to provide an understanding of this disclosure without limiting its scope. In this regard, additional aspects, features, and advantages of this disclosure will be apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0008] Figure 1 An example surgical system is shown.

[0009] Figure 2A and Figure 2B It shows Figure 1 Example components of a surgical system.

[0010] Figure 3A and Figure 3B It shows the use of in Figure 1 Example operations for adding and rendering virtual geometry (e.g., virtual planes) in a surgical system.

[0011] Figure 3C , Figure 3D and Figure 3E It shows the use of in Figure 1 Example operations for adding and rendering virtual geometry (e.g., virtual planes) in a surgical system.

[0012] Figure 4 It shows the use of in Figure 1 Example operations for adding and rendering virtual geometry in the surgical system.

[0013] Figure 5 It shows the use of in Figure 1 Example operations for detecting proximity to virtual geometry in a surgical system.

[0014] Figure 6A and Figure 6B It shows the use of in Figure 1 Example operations for adding and rendering virtual geometry (e.g., virtual cones and virtual tubes) in a surgical system.

[0015] Figure 6C and Figure 6D It shows the use of in Figure 1 Example operations for adding and rendering virtual geometry (e.g., virtual cones and virtual tubes) in a surgical system.

[0016] Figure 7 It shows the use of in Figure 1 Example operations for adding and rendering virtual geometry in the surgical system.

[0017] Figure 8 It is used to add and render virtual geometry and is made by Figure 1 The flowchart shows an example method executed by the surgical system.

[0018] Figure 9 It is used to add and render virtual geometry and is made by Figure 1 The flowchart shows an example method executed by the surgical system. Detailed Implementation

[0019] Doctors use computer-assisted surgical systems to perform procedures on patients, even remotely. These systems provide doctors with various views of the surgical site during the procedure. For example, a surgical system may include an endoscope inserted into the patient's body, allowing the endoscope to capture a video or image stream of the anatomical object the doctor will be manipulating. However, the video or images provide a limited view or perspective of the anatomical object. As the endoscope moves through the body, or as the doctor removes parts of the anatomical structure, the doctor may become disoriented and lose track of the endoscope and other medical instruments' position within the body. Furthermore, it is difficult for doctors to accurately judge depth when viewing the video or image stream, especially in certain medical conditions (e.g., cancer) where the tissue plane that doctors typically use to judge depth is damaged. Therefore, the doctor may unnecessarily cut too deep into the tissue.

[0020] Furthermore, some structures in the body are partially or completely covered by other anatomical objects. Therefore, it can be difficult for doctors to see these structures, and they may unintentionally cut or damage them during surgery. For example, the ureter can extend through a part of the body, but it may be obscured by other tissues or membranes along its length. When a doctor operates near the ureter, if the ureter is not visible, the doctor may unintentionally damage or cut it.

[0021] This disclosure describes a computer system that assists a physician during surgery by adding virtual geometry to the view provided to the physician. For example, the computer system may add one or more virtual planes to a video or image stream. Each plane can be used as a reference plane during the procedure. The plane can be set to have a specific orientation and at a specific depth in the body (e.g., depth as viewed from the visual plane of the video or image stream). The view provided by the video or image stream may change as the endoscope moves to different positions. Even if the view in the video or image stream has changed, the computer system can maintain the orientation and depth of the plane. Therefore, the plane can continue to mark the depth or specific orientation in the body even if the view in the video or image stream has changed.

[0022] As another example, a computer system can add virtual geometry to a video or image stream to indicate the location of occluded and unseen anatomical objects. In the example of the ureter, the computer system can detect the exposed portion of the ureter and determine its orientation and depth based on that exposed portion. The computer system can then use the determined orientation and depth to add virtual geometry (e.g., a virtual cone, a virtual tube, etc.) to the video to indicate the orientation and depth of occluded and unseen portions of the ureter. Thus, the virtual geometry indicates to the physician the location of portions of the ureter that are not visible in the video or image stream.

[0023] In some implementations, the computer system offers several technical advantages. For example, adding virtual geometry (e.g., virtual planes) to a video or image stream helps doctors maintain their sense of orientation even when the view changes within the video or image stream. As another example, adding virtual geometry (e.g., virtual cones, virtual tubes, etc.) to a video or image stream indicates the location of anatomical objects that are not visible to the doctor, which helps the doctor avoid unintentionally cutting or damaging those anatomical objects.

[0024] The described computer system can be implemented as part of a surgical system (e.g., a computer-assisted surgical system). Figure 1 An example computer-assisted surgical system 100 implementing some of the features described herein is shown.

[0025] Surgical system 100 includes a manipulator assembly 102, a user control unit 104, and an auxiliary unit 106, all of which are communicatively coupled to each other. A medical team utilizes surgical system 100 to perform computer-assisted medical procedures or other similar operations on the body of patient 108 or on any other body, as may be suited to a particular implementation. The medical team includes a first user 110-1 (e.g., a surgeon for the surgical procedure), a second user 110-2 (e.g., a patient-side assistant), a third user 110-3 (e.g., another assistant, nurse, intern, etc.), and a fourth user 110-4 (e.g., an anesthesiologist for the surgical procedure), all of whom are collectively referred to as users 110, and each of these users can control surgical system 100, interact with surgical system 100, or otherwise become a user of surgical system 100. More, fewer, or alternative users may be present during the medical procedure, as may be suited to a particular implementation. For example, the team composition for different medical procedures or for non-medical procedures may differ and may include users with different roles.

[0026] although Figure 1The illustration depicts a minimally invasive medical procedure, such as a minimally invasive surgical procedure, but it will be understood that the surgical system 100 can be similarly used to perform open medical procedures or other types of operations. For example, it can also perform operations such as exploratory imaging, simulated medical procedures for training purposes, and / or other operations.

[0027] Manipulator assembly 102 includes one or more instruments that can be coupled to one or more manipulator arms 112 (e.g., manipulator arms 112-1 to 112-4). These instruments are used for computer-assisted surgical procedures performed on patient 108 (e.g., by being at least partially inserted into and manipulated within patient 108). Although manipulator assembly 102 is depicted and described herein as comprising four manipulator arms 112, manipulator assembly 102 may include a single manipulator arm 112 or any other number of manipulator arms as may serve a particular implementation. Although Figure 1 The example shows manipulator arm 112 as a robotic manipulator arm, but one or more instruments may be partially or fully manually controlled, for example by being held and manually controlled by a person. These partially or fully manually controlled instruments may be used in conjunction with, or as an alternative to, computer-aided instruments coupled to manipulator arm 112.

[0028] During medical procedures, the user control device 104 facilitates remote operation control of the manipulator arm 112 and instruments attached to it by the user 110-1. To this end, the user control device 104 provides the user 110-1 with images of the operation area associated with the patient 108, such as those captured by an imaging device. The manipulator arm 112 or any instrument coupled to it mimics the dexterity of the user 110-1's hand, wrist, and fingers across multiple degrees of freedom of motion. In this way, the user 110-1 intuitively performs procedures (e.g., incisions, sutures, etc.) using any instrument coupled to the manipulator arm 112 or one or more of the manipulator arms 112.

[0029] The auxiliary device 106 includes one or more computing devices that perform auxiliary functions supporting the process, such as providing pneumoperitoneum, electrosurgical energy, illumination or other energy to imaging equipment, image processing, or components coordinating the surgical system 100. The auxiliary device 106 includes a display monitor 114 that displays one or more user interfaces, or graphical or textual information supporting the process. In some cases, the display monitor 114 is a touchscreen display providing user input functionality. Enhanced content provided by the area-based enhancement system may be similar to or different from the content associated with the display monitor 114 or one or more display devices in the operating area (not shown).

[0030] The controller assembly 102, user control device 104, and auxiliary device 106 are communicatively coupled to each other in any suitable manner. The controller assembly 102, user control device 104, and auxiliary device 106 may be communicatively coupled via control line 116, which represents any wired or wireless communication link that can serve a particular implementation. For this purpose, the controller assembly 102, user control device 104, and auxiliary device 106 may each include one or more wired or wireless communication interfaces, such as one or more LAN interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0031] Figure 2A An example manipulator component 102 is shown. (See example...) Figure 2A As seen, the manipulator assembly 102 includes a base 118, manipulator arms 112-1, 112-2, 112-3, and 112-4. Each manipulator arm 112-1, 112-2, 112-3, and 112-4 is pivotally coupled to the base 118. Although the base 118 may include casters to allow for easy movement, in some embodiments, the manipulator assembly 102 is fixedly mounted on the floor, ceiling, console, structural frame, etc.

[0032] In a typical procedure, two of the manipulator arms 112-1, 112-2, 112-3, or 112-4 hold surgical instruments, and a third manipulator arm holds a stereoscopic endoscope. The remaining manipulator arms are available to allow the introduction of other instruments at the work site. Alternatively, the remaining manipulator arms can be used to introduce another endoscope or another image-capturing device, such as an ultrasound transducer, into the work site.

[0033] Each of manipulator arms 112-1, 112-2, 112-3, and 112-4 is formed by links coupled together and actuated by joints. Each of manipulator arms 112-1, 112-2, 112-3, and 112-4 may include a mounting arm and a device manipulator. The mounting arm positions the device it holds such that a pivot point appears at its entrance port to the patient. The device manipulator can then manipulate the device it holds such that the held device can pivot about the pivot point, be inserted into and retracted from the entrance port, and rotate about its axis. Each of manipulator arms 112-1, 112-2, 112-3, and 112-4 may include sensors (e.g., joint sensors, positioning sensors, accelerometers, etc.) for detecting or tracking movement of manipulator arms 112-1, 112-2, 112-3, and 112-4. For example, these sensors can detect how far or how fast the manipulator arms 112-1, 112-2, 112-3, or 112-4 move in a certain direction.

[0034] Figure 2B An example user control device 104 is shown. User control device 104 includes a stereoscopic display 120, allowing a user to stereoscopically observe the surgical site from images captured by a stereoscopic camera device of manipulator assembly 102. A left eyepiece 122 and a right eyepiece 124 are provided in the stereoscopic display 120, allowing the user to view the left and right displays within the display 120 with their left and right eyes, respectively. While typically observing images of the surgical site on a suitable observer or display, the surgeon performs the surgical procedure by manipulating a main control input device, which in turn controls the movement of robotic instruments.

[0035] User control device 104 also includes a left input device 126 and a right input device 128, which the user grasps with his / her left and right hands, respectively, to manipulate devices (e.g., surgical instruments) held by manipulator arms 112-1, 112-2, 112-3, and 112-3 of manipulator assembly 102, preferably in six or more degrees of freedom (“DOF”). A foot pedal 130 with toe and heel controls is provided on user control device 104, allowing the user to control the movement and / or actuation of devices associated with the foot pedal.

[0036] A processing device 132 is provided in the user control device 104 for control and other purposes. The processing device 132 performs various functions in the surgical system 100. One function performed by the processing device 132 is to convert and transmit the mechanical movements of input devices 126 and 128 to actuate their corresponding joints in their associated manipulator arms 112-1, 112-2, 112-3, and 112-4, enabling the surgeon to effectively manipulate devices such as surgical instruments. Another function of the processing device 132 is to implement the methods, cross-coupled control logic, and controllers or processors described herein. The auxiliary device 106 may include the processing device 132 performing the functions or actions described herein. The processing device 132 includes a processor and memory for performing the functions described herein.

[0037] The processor may include any electronic circuit system, including but not limited to one or a combination of a microprocessor, microcontroller, application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), and / or state machine, communicatively coupled to memory and controlling the operation of user control device 104 and / or auxiliary device 106. The processor may be 8-bit, 16-bit, 32-bit, 64-bit, or any other suitable architecture. The processor may include an arithmetic logic unit (ALU) for performing arithmetic and logical operations, processor registers for supplying operands to the ALU and storing the results of ALU operations, and control units for fetching instructions from memory and executing them by directing the coordinated operation of the ALU, registers, and other components. The processor may include other hardware with operating software to control and process information. The processor executes software stored in memory to perform any of the functions described herein. The processor controls the operation and management of user control device 104 or auxiliary device 106 by processing information, such as information received from user control device 104, manipulator assembly 102, auxiliary device 106, and / or memory. A processor is not limited to a single processing device and can include multiple processing devices contained in the same device or computer or distributed across multiple devices or computers. If multiple processing devices jointly perform a set of functions or actions, the processor is considered to perform that set of functions or actions, even if different processing devices perform different functions or actions within that set.

[0038] Figure 3A , Figure 3B as well as Figures 4 to 7 It shows Figure 1 Example operations in surgical system 100. A computer system (implemented in the user control unit 104 and / or auxiliary unit 106 of surgical system 100, possibly using processing device 132) performs the operations. Generally, the computer system adds virtual reference geometry to a video of the anatomical structure.

[0039] like Figure 3A As seen, the computer system presents a video showing a top view of anatomical structure 302. This video can be captured by a camera device positioned on or inside the patient's body. The camera device can be aimed at anatomical structure 302 within the patient. The computer system receives the video and displays the video of anatomical structure 302.

[0040] The computer system adds a virtual plane 304 to the video of the anatomical structure 302, so that the virtual plane 304 is displayed together with the anatomical structure 302. The virtual plane 304 intersects with the anatomical structure 302. In some cases, the virtual plane 304 can be positioned at a depth where it intersects with a portion of the anatomical structure 302. For example... Figure 3AAs can be seen, the virtual plane 304 intersects with certain parts of the anatomical structure 302, such that a part of the virtual plane 304 is positioned on the anatomical structure 302, and that certain parts of the anatomical structure 302 extend above the virtual plane 304.

[0041] A virtual plane 304 is defined using point 306 and vector 308. Point 306 can be selected by the operator of the computer system. Point 306 can be selected or set at a specific depth in the video of the anatomical structure 302. Vector 308 can be a normal to plane 304 (e.g., orthogonal to plane 304). Therefore, point 306 can set the depth of virtual plane 304, and vector 308 can set the direction or orientation of virtual plane 304. In some embodiments, point 306 and / or vector 308 are set automatically by the computer system. For example, the computer system can set point 306 on the surface of the anatomical structure. Therefore, virtual plane 304 can be set to be tangent to a point on the surface of the anatomical structure. The computer system can set vector 308 to extend parallel to the viewing direction of the camera device capturing the video of the anatomical structure 302. Therefore, the computer system can automatically set the direction or orientation of virtual plane 304 such that virtual plane 304 faces the camera device.

[0042] In some implementations, the computer system uses a depth map of the anatomical structure 302 to determine the depth of point 306. The depth map indicates the depth of different points on the anatomical structure 302. When point 306 is selected, the computer system determines the location (e.g., coordinates) of point 306 on the anatomical structure 302. The computer system then determines the corresponding location on the depth map and determines the depth of that corresponding location (or the depth of the point on the depth map closest to that corresponding location). The computer system then uses the determined depth as the depth of point 306.

[0043] In a particular implementation, the operator of the computer system adjusts the positioning, orientation, or orientation of the virtual plane 304. For example, the operator can use a controller (e.g., input devices 126 and 128) to adjust the depth of point 306, which adjusts the depth of the virtual plane 304. As another example, the operator can use a controller to change the orientation or orientation of vector 308, which adjusts the orientation or orientation of the virtual plane 304. In this way, the operator can position the virtual plane 304 at any depth and with any orientation.

[0044] Virtual plane 304 serves as a reference plane fixed at a depth, orientation, and orientation defined for it. Therefore, when the camera capturing video of the anatomical structure 302 moves, causing a change in the view provided by the camera, virtual plane 304 remains at its depth, orientation, and orientation to help the computer system operator avoid becoming disoriented. Figure 3B As seen, the view of anatomical structure 302 has been changed from... Figure 3A The top view shown has changed to a side view. This change of view may have occurred because the camera device capturing the video of the anatomical structure 302 has been rotated or moved to one side of the anatomical structure 302. Figure 3B In the example, virtual plane 304 remains as Figure 3A The depth, direction, and orientation of the virtual plane 304 are shown. In this way, the depth of the virtual plane 304 is maintained even if the view provided by the camera device changes.

[0045] After the camera capturing the video has moved, the computer system uses any suitable information or process to render the virtual plane 304. For example, if the camera moves but the anatomical structure 302 remains in view, the computer system can use Simultaneous Localization and Mapping (SLAM) processing to generate a virtual map of the anatomical structure 302 and place the virtual plane 304 in the virtual map. The computer system then renders the virtual plane 304 in the video of the anatomical structure 302 based on the virtual map, such that the virtual plane 304 remains at a specific depth and orientation, even though the camera has moved.

[0046] As another example, if the camera device moves such that the anatomical structure 302 is not kept in the view, the computer system can use kinematic sensor information to render the virtual plane 304. For example, if the camera device moves to another part of the patient's body, the anatomical structure 302 may not remain in the view, and the SLAM processing may not provide the computer system with enough information to render the plane 304 at the appropriate depth and orientation. In these cases, the computer system can use sensor information from the camera device (e.g., information from kinematic sensors positioned on or near the camera device) to determine how the camera device has moved. For example, the sensor information may indicate the distance and direction of travel of the camera device. The computer system uses the sensor information to determine the new position, orientation, and / or pose of the camera device. The computer system then uses this information to render the virtual plane 304 at the appropriate depth and orientation in the new view provided by the camera device, such that the virtual plane 304 remains at the same depth and orientation set for the virtual plane 304. In this way, even as the camera moves and provides different views, the computer system continues to render the virtual plane 304, which the operator of the computer system uses as a reference plane to maintain a sense of orientation. For example, even when the view in the video changes due to the movement of the camera, the virtual plane 304 can help the operator of the computer system understand where a particular depth appears in the video.

[0047] Figure 3C , Figure 3D and Figure 3E An example of adding virtual geometry to video 310 is shown. Figure 3C As seen in video 310, anatomical structure 312 and medical device 314 are shown. The computer system places point 316 within video 310. Point 316 can be positioned at any depth. For example, point 316 can be positioned on the surface of anatomical structure 312. As another example, point 316 can be positioned below or above the surface of anatomical structure 312. The computer system also extends vector 318 from point 316. Vector 318 can extend in any direction. For example, vector 318 can extend toward or away from the camera device capturing video 310. Figure 3C In the example, vector 318 extends into video 310 and moves away from the camera device along the direction of observation of the camera device.

[0048] like Figure 3DAs seen, the computer system adds a virtual plane 320 to the video 310. The virtual plane 320 is positioned at a depth based on point 316. For example, the virtual plane 320 can be positioned at the same depth as point 316. As another example, the virtual plane 320 can be positioned at a depth offset from the depth of point 316 (e.g., the virtual plane 320 can be offset by one centimeter from the depth of point 316). Furthermore, the orientation of the virtual plane 320 is set according to vector 318. For example, the orientation of the virtual plane 320 can be set such that vector 318 is perpendicular to the virtual plane 320. Based on the depth and orientation of the virtual plane 320, the virtual plane 320 will intersect with the anatomical structure 312 and the medical device 314 at certain points. Therefore, some portions of the anatomical structure 312 and the medical device 314 will be positioned below the virtual plane 320, and some portions of the anatomical structure 312 and the medical device will be positioned above the virtual plane 320.

[0049] like Figure 3E As can be seen, when the view in video 310 changes (e.g., due to movement of the camera capturing video 310), the computer system continues to present the virtual plane 320 with the same depth and orientation in the changed view. Therefore, the virtual plane 320 serves as a reference plane to indicate a specific depth to the operator of the computer system. The operator can observe the virtual plane 320 in the changed view to understand where the specific depth appears in the changed view.

[0050] Figure 4 It shows Figure 1 Example operation 400 in surgical system 100. Generally, Figure 4 A computer system is illustrated that adds virtual geometry (e.g., a virtual plane) to video. The virtual geometry can be positioned at a specific depth or orientation within the video, and the computer system continues to render and display the virtual geometry at the set depth and / or orientation as the view in the video changes (e.g., due to movement of the camera capturing the video). In this way, the virtual geometry helps the operator of the computer system maintain a sense of orientation as the view in the video changes.

[0051] The computer system receives video 402. Video 402 is captured by a camera device. In some cases, video 402 shows anatomical structures within the patient's body. The camera device captures video 402 when it is positioned within the patient's body, close to an anatomical structure. The computer system displays video 402 (e.g., using a stereoscopic display 120 or display system 210) so that an operator can observe the anatomical structure while working on or near it.

[0052] The computer system receives point 404 and / or vector 406. An operator of the computer system can use a controller (e.g., input devices 126 and 128 or operator input 206) to select point 404. For example, the operator can use the controller to set or select coordinates in video 402 as point 404. The operator can also set the depth of point 404. After adding a point to video 402, the operator can also move the controller to move point 404. When point 404 is at a desired location or position, the operator uses the controller to indicate the selection of point 404 at that location or position. Vector 406 may originate from point 404. The operator of the computer system can also set the direction or orientation of vector 406. For example, the operator can use the controller to set the direction or orientation of vector 406 extending from point 404. In some embodiments, the computer system automatically sets the initial direction or orientation of vector 406. For example, the computer system may set vector 406 to extend directly into the camera device. Then, the operator of the computer system can adjust or change the direction or orientation of vector 406.

[0053] The computer system then renders virtual geometry 408 using point 404 and vector 406. For example, the computer system can render a virtual plane set at the depth of point 404 and oriented with vector 406 as the plane's normal. The computer system then renders virtual geometry 408 in video 402, so that virtual geometry 408 is displayed along with video 402. In some embodiments, the computer system uses a depth map to determine the depth of point 404. A depth map indicates the depth of different points on an anatomical structure. When point 404 is set, the computer system determines the position of point 404 on the anatomical structure. The computer system then determines the corresponding position and the depth of the corresponding position on the depth map. The determined depth is then used as the depth of point 404.

[0054] The virtual geometry 408 can be of any shape. For example, a computer system can use point 404 and vector 406 to render a virtual cone or cylinder. The computer system can set the base of the virtual cone or cylinder at the depth of point 404. The computer system can also orient the virtual cone or cylinder along the direction of vector 406 (e.g., orient vector 406 along the height of the virtual cone or cylinder). As another example, a computer system can use point 404 and vector 406 to render a virtual sphere. The computer system can set the center of the virtual sphere to point 404. The computer system can also set the radius of the sphere to vector 406. The computer system can display virtual cones, cylinders, spheres, etc., together with video 402.

[0055] In some implementations, the controller used by the operator of the computer system to set point 404 and / or vector 406 is the same controller used by the operator of the computer system to move the medical device in the patient's body. To avoid moving the medical device when setting point 404 and / or vector 406, the operator of the computer system enters a separate mode (e.g., clutch mode), in which the operator moves the controller without moving the medical device. In this way, the operator of the computer system moves the controller to set point 404 and / or vector 406 without moving the medical device in the patient's body, thereby avoiding harm to the patient.

[0056] In a particular implementation, the computer system uses the positioning or location of the medical device to set point 404 and / or vector 406. For example, an operator of the computer system can use a controller to move the medical device in video 402. The operator can then use the controller to indicate the selection of point 404 or vector 406. The computer system can then set point 404 and / or vector 406 at the positioning or location of the medical device.

[0057] In some implementations, the computer system uses additional information to render the virtual geometry 408. For example, the computer system may receive an image 410, which may be an ultrasound image. Image 410 may reveal structures within or beneath the anatomical structure shown in video 402. The computer system may determine the depth of these structures within or beneath the anatomical structure in video 402. The computer system may then set the depth of point 404 at the depth of the structure shown in image 410, and the computer system may extend vector 406 from point 404 at that depth. The resulting virtual geometry 408 may then be set at the depth of the anatomical structure within or beneath the anatomical structure shown in video 402.

[0058] After setting or rendering the virtual geometry 408, the operator of the computer system can continue to adjust the virtual geometry 408. For example, the operator can manipulate the controller to adjust the virtual geometry 408 412. When the operator manipulates the controller, the computer system can receive the adjustment 412. The computer system can then make corresponding adjustments to the virtual geometry 408. For example, the computer system can translate or rotate the virtual geometry 408 based on the movement of the controller. If the virtual geometry 408 is a virtual plane, the computer system can adjust the depth of the virtual plane, move the virtual plane, or rotate the virtual plane, which changes the direction or orientation of the virtual plane. In this way, the operator of the computer system can adjust the positioning, orientation, or orientation of the virtual geometry 408.

[0059] The operator of the computer system can add any number of virtual geometries 408 to the video 402. The virtual geometries 408 can be of any desired shape. For example, the operator can add virtual planes, virtual cones, virtual spheres, virtual cylinders, virtual tubes, etc., to the video 402. These virtual geometries 408 can also be set to different poses (e.g., different positions, orientations, and / or orientations). The operator of the computer system can also use a controller to set the position, orientation, and orientation of these virtual geometries 408. These virtual geometries 408 can help the operator of the computer system avoid becoming disoriented when the camera device capturing the video 402 moves.

[0060] The computer system detects movement 414. Movement 414 causes a change in the view shown in video 402. For example, movement 414 could be movement of a camera device capturing video 402. As another example, movement 414 could be movement of a patient or anatomical structure in video 402. The computer system tracks movement 414 to determine the change in view in video 402. Video 402' is the video 402 in which the view has changed. The computer system then renders virtual geometry 408 in video 402' such that virtual geometry 408 maintains the same pose (e.g., the same location, orientation, and / or orientation) in video 402'. For example, the location of virtual geometry 408 in video 402' can still be set based on point 404, and the orientation or orientation of virtual geometry in video 402' can still be set based on vector 406. Because point 404 and vector 406 do not change their location, orientation, or orientation as a result of movement 414, virtual geometry 408 maintains the same pose despite the presence of movement 414.

[0061] As an example, the computer system can add a virtual plane to video 402. The depth of the virtual plane can be set according to point 404, and the orientation or orientation of the virtual plane can be set according to vector 406. Movement 414 can cause a change in the view in video 402. The computer system can determine the movement 414 that occurred to determine the change in view in video 402'. For example, the computer system can use SLAM processing or kinematic information to determine the movement 414 and / or the change in pose of the camera capturing video 402. The computer system can then render the virtual plane in video 402' such that the virtual plane is maintained in the same pose that the virtual plane had before movement 414. In this way, the operator of the computer system can continue to observe video 402' to see the virtual plane. Although the view in video 402' changes, the virtual plane can continue to indicate the same depth, orientation, and / or orientation set for the virtual plane, which helps the operator of the computer system maintain a sense of orientation.

[0062] The computer system can use any suitable processing for tracking movement 414. For example, if video 402' continues to show at least a portion of an anatomical structure after movement 414, the computer system can use SLAM processing to determine or track movement 414. As another example, if movement 414 removes an anatomical structure from the view of video 402', the computer system can use sensor data (e.g., kinematic sensor information). Sensor information can indicate to the computer system the distance and direction of the camera's movement. Additionally, kinematic sensor information can indicate to the computer system the camera's pose. The computer system can use this information to determine changes in view in video 402' and render virtual geometry 408 with the same depth, orientation, and orientation.

[0063] In SLAM processing, the computer system can stitch together various frames of video 402 to form a virtual map of the anatomical structure 302. The computer system can also add virtual geometry 408 to the virtual map. When movement 414 occurs, the computer system can compare the changed view in video 402' with the virtual map of the anatomical structure to determine the updated pose or movement 414 of the camera device. The computer system can then render the virtual geometry 408 in video 402' as indicated in the virtual map.

[0064] Figure 5 It shows Figure 1 Example operation 500 in surgical system 100. Generally, Figure 5 A computer system is shown that uses virtual geometry 408 to guide the movement of a medical device. In this way, the computer system improves the safety of the process.

[0065] The computer system receives or determines device positioning 502. Device positioning 502 indicates the location of the medical device in the patient's body. Device positioning 502 may include coordinates indicating the location of the medical device. The computer system uses device positioning 502 to track the location or movement of the medical device. As the operator of the computer system moves the device (e.g., using input devices 126 and 128), device positioning 502 is updated.

[0066] The computer system compares the distance between the device positioning 502 and the virtual geometry 408 with a margin 504. When the distance between the device positioning 502 and the virtual geometry 408 is less than the margin 504, the computer system determines that the device positioning 502 is too close to the virtual geometry 408. The computer system then generates and transmits an alarm 506 indicating that the medical device is too close to the virtual geometry 408. In some embodiments, the computer system then prevents or resists movement of the medical device closer to the virtual geometry 408. In an example, if the virtual geometry 408 is a virtual plane set at a certain depth, and the operator of the computer system does not want the medical device to move beyond that depth, the computer system monitors the positioning of the medical device and generates and transmits an alarm 506 when the medical device is too close to the virtual plane. The computer system also prevents or resists any further movement of the medical device toward the virtual plane. In some embodiments, the computer system also presents or displays the distance between the medical device and the virtual geometry 408, which indicates to the physician how far the medical device can move without reaching the virtual geometry 408. In this way, computer systems improve the health and safety of patients during the procedure.

[0067] Computer systems can also use virtual geometry to indicate the location and orientation of anatomical structures that are not visible in the video. Figure 6A and Figure 6B It shows Figure 1 Example operations 600 and 609 in the surgical system 100. Generally, Figure 6A and Figure 6B A computer system is shown that adds virtual geometry to indicate the location and orientation of invisible parts of anatomical structures (e.g., ureter, blood vessels, nerves, pedicles, etc.).

[0068] like Figure 6A As seen, the computer system receives video, in which a portion 602 of the anatomical structure 604 is visible. Other portions of the anatomical structure 604 may be invisible because they are covered or occluded. Figure 6A In the example, part 602 is visible, and other parts of the anatomical structure 604 are covered or obscured by another anatomical structure 606 and are not visible. The computer system analyzes part 602 to determine the location and orientation of the parts of the anatomical structure 604.

[0069] The computer system then adds virtual geometry 608 to the video to indicate the location and orientation of occluded and unseen portions of the anatomical structure 604. Figure 6AIn the example, the computer system analyzes the location and orientation of part 602 to determine the location and orientation of virtual geometry 608. For example, the computer system determines the depth of part 602 (e.g., using a depth map) and sets point 404 based on that depth. Point 404 may be set at locations where anatomical structure 604 becomes occluded and invisible (e.g., a point at the boundary between anatomical structure 604 and anatomical structure 606). The computer system also determines the orientation or orientation of part 602 and sets a vector 406 extending from point 404 along that orientation or orientation. The computer system then renders virtual geometry 608 based on point 404 and vector 406.

[0070] exist Figure 6A In the example, the computer system renders a virtual cone with a vertex set at point 404 and a vector 406 set as the central axis of the cone. In some implementations, the radius of the virtual cone is set to indicate a range of uncertainty regarding the location and orientation of the occluded and unseen portion of the anatomical structure 604. The computer system then displays the virtual cone along with video to indicate the location and orientation of the occluded and unseen portion of the anatomical structure 604. This video can then make it easier for the operator of the computer system to avoid accidentally cutting or damaging the occluded and unseen portion of the anatomical structure 604, thus improving patient health and safety.

[0071] exist Figure 6B In the example, the computer system receives video in which portions 610A and 610B of the anatomical structure 612 are visible. Other portions of the anatomical structure 612 may be invisible because they are covered or occluded by anatomical structures 614 and 616. The computer system analyzes the visible portions 610A and 610B to determine the location and orientation of the occluded portions of the anatomical structure 612.

[0072] The computer system then adds virtual geometries 618A and 618B to the video to indicate the location and orientation of occluded and unseen portions of the anatomical structure 612. Figure 6B In the example, the computer system analyzes the location and orientation of parts 610A and 610B to interpolate the location and orientation of the portion of the anatomical structure 612 extending between parts 610A and 610B. The computer system then renders and adds virtual geometry 618A, such that virtual geometry 618A is connected to parts 610A and 610B.

[0073] The computer system can analyze the location and orientation of part 610B to determine the location and orientation of the anatomical structure 612 following part 610B. The computer system can then add virtual geometry 618B, connecting it to part 610B. The operator of the computer system can view a video showing parts 610A and 610B, as well as virtual geometry 618A and 618B, to gain an understanding of the location and orientation of anatomical structure 612, particularly any obscured or unseen portions of anatomical structure 612. This video can then make it easier for the operator to avoid accidentally cutting or damaging the obscured or unseen portions of anatomical structure 612, thus improving patient health and safety.

[0074] Figure 6C and Figure 6D An example of adding virtual geometry to a 620-pixel video is shown. Figure 6C As seen in video 620, an anatomical structure 622, a medical device 624, and another anatomical structure 626 are shown. Anatomical structure 626 covers a portion of anatomical structure 622. The computer system places a point 628 in the scene. Point 628 is positioned at the point where anatomical structure 626 covers anatomical structure 622, and is positioned at the depth of anatomical structure 622. Therefore, anatomical structure 626 covers point 628. The computer system also extends a vector 630 from point 628. Vector 630 extends in the direction of anatomical structure 622. In some embodiments, the computer system analyzes video 620 to determine the depth and direction of anatomical structure 622. The computer system then adds point 628 at the depth of anatomical structure 622 and extends vector 630 in the direction of anatomical structure 622. Both point 628 and vector 630 can be covered by anatomical structure 626.

[0075] like Figure 6D As seen, the computer system adds a virtual cone 632 to the video 620. Point 628 is the vertex of the virtual cone 632, and the virtual cone 632 extends along the direction of vector 630. Therefore, the virtual cone 632 indicates the orientation and orientation of a portion of the anatomical structure 622 below the anatomical structure 626. Even if this portion of the anatomical structure 622 is not visible in the video 620, the operator of the computer system can observe the virtual cone 632 to understand the location of that portion of the anatomical structure 622. The operator can then more easily avoid cutting or damaging portions of the anatomical structure 622 during operation.

[0076] Figure 7 It shows Figure 1Example operation 700 in surgical system 100. The computer system receives video 702, which may show anatomical structures (e.g., ureter, blood vessels, nerves, pedicles, etc.). Specifically, video 702 shows portions 704A and 704B of the anatomical structures. Some portions of the anatomical structures may be obscured and not visible in video 702.

[0077] The computer system analyzes the video, and specifically portions 704A and 704B of the anatomical structures appearing in video 702, to determine the orientation 705 and depth 706 of the anatomical structures in video 702. For example, the computer system can determine the location of portions 704A and 704B to interpolate the orientation 705 of the anatomical structures. The orientation 705 of the anatomical structures can be aligned with portions 704A and 704B. As another example, the computer system can determine the depth of portions 704A and 704B of the anatomical structures. The computer system then determines the depth 706 of the anatomical structures based on the determined depths of portions 704A and 704B. The determined depth 706 can be the same as the determined depths of portions 704A and 704B, or the depth 706 can be the average of the depths of portions 704A and 704B.

[0078] In some implementations, the computer system uses additional information to determine the orientation 705 and depth 706 of the anatomical structure. For example, the computer system may analyze a preoperative scan of the anatomical structure to determine its orientation 705 and depth 706. The preoperative scan may show the internal structures of the anatomical structure or obscured portions of the anatomical structure, which can help determine its orientation 705 and depth 706. As another example, the computer system may use a depth map of the anatomical structure to determine its depth 706.

[0079] The computer system then renders a virtual geometry 708 and adds it to video 702. The virtual geometry 708 can be a virtual cone or tube conforming to the shape of the anatomical structure. The computer system positions the virtual geometry 708 at depth 706. Additionally, the computer system orients or directs the virtual geometry 708 to align with direction 705. The virtual geometry 708 can be connected to one or more of the portions 704A and 704B shown in video 702. Therefore, the virtual geometry 708 indicates the location and orientation of occluded and unseen portions of the anatomical structure in video 702. The virtual geometry 708 indicates the location and orientation of these occluded and unseen portions of the anatomical structure to the operator of the computer system, which can help the operator avoid cutting or damaging these portions of the anatomical structure during the process.

[0080] Figure 8 It is by Figure 1The flowchart illustrates an example method 800 performed by the computer system in the surgical system 100. By performing method 800, the computer system adds virtual geometry 408 to video 402. Even when the camera capturing video 402 moves and changes the view in video 402, the computer system maintains the depth, orientation, and / or orientation of the virtual geometry 408. Therefore, the virtual geometry 408 can help the operator from the computer system avoid becoming disoriented due to movement of the camera.

[0081] In block 802, the computer system receives video 402. Video 402 is captured by a camera device positioned within the patient's body. Video 402 shows the patient's anatomical structures. The computer system displays video 402 to the operator of the computer system during the medical procedure.

[0082] In block 804, the computer system adds virtual geometry 408 to video 402. The computer system receives point 404 and / or vector 406, which can be used to render virtual geometry 408 and add it to video 402. The operator of the computer system can set point 404 at a desired depth. Additionally, the operator of the computer system can set the direction or orientation of vector 406 extending from point 404. Using point 404 and vector 406, the computer system renders virtual geometry 408 and adds it to video 402. Virtual geometry 408 can be set at the depth of point 404 and can be oriented or oriented according to vector 406. For example, if virtual geometry 408 is a virtual plane, the virtual plane can be set at the depth of point 404, and the virtual plane can have a normal vector oriented along vector 406.

[0083] In some implementations, the computer system uses a depth map of the anatomical structure to determine the depth of point 404. The depth map can indicate the depth of different points on the anatomical structure. When point 404 is set, the computer system determines the location (e.g., coordinates) of point 404 on the anatomical structure. The computer system can then determine the corresponding location on the depth map and the depth of that corresponding location (or the depth of the point on the depth map closest to that corresponding location). The computer system can then use the determined depth as the depth of point 404.

[0084] In block 806, the computer system detects movement 414. Movement 414 can cause a change in the view in video 402. For example, movement 414 can change the perspective of an anatomical structure shown in video 402, or movement 414 can remove an anatomical structure from the view. The computer system can use any suitable processing to detect and track this movement 414. For example, the computer system can use SLAM processing to determine how the pose of the camera device has changed and / or how the view in video 402 has changed. As another example, the computer system can use kinematic sensor information to determine the distance and direction of movement by the camera device, which the computer system can use to determine the updated pose of the camera device.

[0085] In block 808, the computer system renders and presents virtual geometry 408 in a new view of video 402'. The computer system can use the detected movement 414 to determine how virtual geometry 408 should be positioned, guided, and / or oriented in video 402'. In some embodiments, the computer system presents virtual geometry 408 in a new view of video 402' such that virtual geometry 408 maintains the same depth, orientation, and / or orientation as it had before the movement 414 of the camera device. In this way, when the operator of the computer system sees the new view in video 402', the operator can also see virtual geometry 408 maintaining the same depth, orientation, and orientation. Therefore, virtual geometry 408 can help prevent the operator from becoming disoriented as a result of the movement 414 of the camera device and the change of view.

[0086] Figure 9 It is by Figure 1 The flowchart illustrates an example method 900 performed by the computer system in the surgical system 100. By performing method 900, the computer system adds virtual geometry 708 to video 702 to indicate the location and orientation of anatomical structures that are occluded and not visible in video 702.

[0087] In block 902, the computer system receives video 702. Video 702 shows portions 704A and 704B of the anatomical structure. Other portions of the anatomical structure may be hidden or obscured and not visible in video 702.

[0088] In block 904, the computer system detects portion 704A of the anatomical structure in video 702. The computer system can use computer vision processing to detect portion 704A of the anatomical structure in video 702.

[0089] In block 906, the computer system determines the orientation 705 and depth 706 of the anatomical structure. The computer system can analyze detected portions 704A of the anatomical structure to determine the orientation 705 and depth 706. For example, the orientation 705 can be aligned with the orientation of portion 704A of the anatomical structure, and the depth 706 can be the same as or within the range of the depth of portion 704A of the anatomical structure. In some embodiments, when multiple portions of the anatomical structure are shown in video 702, the computer system can determine the orientation 705 and depth 706 based on the multiple portions. For example, the orientation 705 can be interpolated using the orientations of the multiple portions, and the depth 706 can be the same as or the average depth of the multiple portions of the anatomical structure.

[0090] In block 908, the computer system adds virtual geometry 708 to video 702. The computer system can render virtual geometry 708 based on a determined orientation 705 and depth 706. For example, virtual geometry 708 can have the same orientation 705 and the same depth 706. The computer system can then add virtual geometry 708 with orientation 705 and depth 706 to video 702. When an operator of the computer system views video 702, virtual geometry 708 can indicate to the operator the location and orientation of hidden or occluded parts of the anatomical structure. The operator can then more easily avoid cutting or damaging parts of the anatomical structure, which improves patient health and safety.

[0091] In summary, computer systems assist surgeons during procedures by adding virtual geometry to the view provided to the surgeon. For example, a computer system can add one or more virtual planes to a video or image stream. Each plane can be used as a reference plane during the procedure. The planes can be set to have a specific orientation and at a certain depth within the body (e.g., the depth as viewed from the visual plane of the video or image stream). As the camera moves to different positions, the view provided by the video or image stream may change. Even if the view in the video or image stream has changed, the computer system can maintain the orientation and depth of the planes. Therefore, even if the view in the video or image stream has changed, the planes can continue to mark a specific orientation or depth within the body.

[0092] As another example, a computer system can add virtual geometry to a video or image stream to indicate the location of occluded and unseen anatomical objects. In the example of the ureter, the computer system can detect the exposed portion of the ureter and determine its orientation and depth from that exposed portion. The computer system can use the determined orientation and depth to add virtual geometry (e.g., a virtual cone, a virtual tube, etc.) to the video to indicate the orientation and depth of occluded and unseen portions of the ureter. Thus, the virtual geometry indicates to the physician the location of portions of the ureter that are not visible in the video or image stream.

[0093] The aspects, embodiments, or modules illustrated in this specification and the accompanying drawings should not be considered limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this specification and the claims. In some cases, well-known circuits, structures, or techniques have not been shown or described in detail so as not to obscure other features. Similar reference numerals in two or more figures denote the same or similar elements.

[0094] In this specification, specific details are set forth in relation to some embodiments consistent with this disclosure. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are illustrative and not restrictive. Those skilled in the art will recognize that other elements, though not specifically described herein, are within the scope and spirit of this disclosure. Furthermore, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless otherwise specifically described or if one or more features would render the embodiment inoperable.

[0095] Furthermore, the terminology used in this specification is not intended to be restrictive. For example, spatially relative terms such as “below,” “under,” “lower,” “above,” “upper,” “proximal,” “farthest,” etc., may be used to describe the relationship of one element or feature to another element or feature as shown in the figures. In addition to the positioning and orientation shown in the figures, these spatially relative terms are intended to cover different positioning (i.e., location) and orientation (i.e., rotational placement) of elements or their operation. For example, if one of the contents in the figures is flipped, it is described as an element that is “below” or “under” other elements or features and then “above” or “on” other elements or features. Thus, the exemplary term “below” can cover both above and below positioning and orientation. Devices may be oriented in other ways (rotated 90 degrees or in other orientations), and the spatially relative descriptors used herein are interpreted accordingly. Similarly, descriptions of movement along and about various axes include various specific element positioning and orientations. Furthermore, unless the context otherwise indicates, the singular forms “a,” “an,” and “the” are also intended to include the plural forms. Furthermore, the terms "comprising," "including," "including," etc., specify the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

[0096] Where feasible, elements described in detail with reference to one embodiment or module may be included in other embodiments or modules in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment but not with reference to a second embodiment, that element may still be required to be included in the second embodiment. Therefore, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment or application may be incorporated into other embodiments or aspects unless otherwise specifically described, unless one or more elements would render one or more embodiments inoperable, or unless two or more of the elements provide conflicting functionality.

[0097] In some cases, well-known methods, processes, components, and circuits are not described in detail to avoid unnecessarily obscuring aspects of the implementation.

[0098] This disclosure describes various devices, elements, and portions of computer-aided devices and components in three-dimensional space based on their state. As used herein, the term "position" refers to the location of an element or portion of an element in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x-, y-, and z-coordinates). As used herein, the term "orientation" refers to the rotational placement of an element or portion of an element (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term "shape" refers to a set of positions or orientations measured along the element. As used herein, and for devices with repositionable arms, the term "proximal end" refers to a direction along its kinematic chain toward the base of the computer-aided device, and "distal end" refers to a direction along the kinematic chain away from the base.

[0099] The aspects of this disclosure are described with reference to computer-aided systems and devices, which may include teleoperated, remotely controlled, autonomous, semi-automatic, robotic, and other systems and devices. Furthermore, aspects of this disclosure are described according to embodiments using medical systems, such as the da Vinci Surgical System or the ION System, commercially available from Intuitive Surgical, Sunnyvale, California. However, those skilled in the art will understand that the aspects disclosed herein can be implemented and carried out in various ways, including robotic and non-robotic implementations (where applicable). The techniques described with reference to surgical instruments and methods can be used in other situations. Therefore, the instruments, systems, and methods described herein can be used for humans, animals, parts of human or animal anatomy, industrial systems, general-purpose robotic or teleoperation systems. As another example, the instruments, systems, and methods described herein can be used for non-medical purposes, including industrial applications, general-purpose robotic applications, sensing or manipulating non-tissue artifacts, cosmetic enhancements, imaging of human or animal anatomy, collecting data from human or animal anatomy, installing or disassembling systems, training medical or non-medical personnel, etc. Additional example applications include procedures for the removal of tissue from human or animal anatomy (with or without return to the anatomy) and procedures for human or animal carcasses. Furthermore, these techniques can also be used in medical treatments or diagnostic procedures, with or without surgical involvement.

[0100] Although illustrative embodiments have been shown and described, a wide range of modifications, alterations, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be employed without the need for corresponding use of other features. Many variations, substitutions, and modifications will be recognized by those skilled in the art. Therefore, the scope of this disclosure should be limited only by the appended claims, and it is appropriate that the claims be interpreted broadly and in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A computer system for adding virtual reference geometry to a video of an anatomical structure, the computer system comprising: Memory; as well as A processor communicatively coupled to the memory, the processor being configured to: Receive the video of the anatomical structure, wherein the video shows a first view; Receive selections of points and vectors in the first view; A first virtual geometry is added to the first view based on the points and the vectors; Detect the movement that transforms the first view into the second view; and The first virtual geometry is presented in the second view such that the pose of the first virtual geometry is maintained from the first view to the second view.

2. The computer system according to claim 1, wherein, The processor is also configured to use simultaneous localization and mapping processing to determine the location of the first virtual geometry in the second view.

3. The computer system according to claim 1, wherein, The processor is also configured to use kinematic data to determine the location of the first virtual geometry in the second view.

4. The computer system according to claim 1, wherein, The processor is also configured to rotate the first virtual geometry based on the detected rotation of the controller.

5. The computer system according to claim 1, wherein, The processor is also configured to adjust the depth of the first virtual geometry based on input from the controller.

6. The computer system according to claim 1, wherein, The processor is also configured to add a second virtual geometry to the first view, wherein the second virtual geometry has a different pose than the first virtual geometry.

7. The computer system according to claim 1, wherein, The processor is also configured to determine the depth of the first virtual geometry based on ultrasound images of the anatomical structure.

8. The computer system according to claim 7, wherein, The video shows a portion of the anatomical structure obscured and not visible, and the ultrasound image shows that portion of the anatomical structure.

9. The computer system according to claim 7, wherein, The depth of the first virtual geometry is below at least a portion of the outer surface of the anatomical structure shown in the video.

10. The computer system according to claim 1, wherein, The processor is also configured to determine when the medical device has moved within the margin of the first virtual geometry.

11. The computer system according to claim 10, wherein, The processor is also configured to generate an alarm in response to the medical device moving within the margin of the first virtual geometry.

12. The computer system according to claim 10, wherein, The processor is also configured to present the distance between the medical device and the first virtual geometry.

13. The computer system according to claim 1, wherein, The first virtual geometry includes a virtual plane, and wherein maintaining the pose of the first virtual geometry includes maintaining the depth of the virtual plane from the first view to the second view.

14. The computer system according to claim 1, wherein, The point is located on the surface of the anatomical structure, and wherein the first virtual geometry is tangent to the surface of the anatomical structure.

15. The computer system according to claim 14, wherein, The selection of the point is based on the positioning of the camera device that captures the video.

16. The computer system according to any one of claims 15, wherein, The normals of the first virtual geometry are oriented according to the orientation of the camera device.

17. The computer system according to claim 14, wherein, The selection of the point includes moving the point based on the movement of the controller, and selecting the point based on input from the controller.

18. The computer system according to claim 17, wherein, The processor is also configured to prevent the medical device from moving during the movement of the controller.

19. The computer system according to claim 17, wherein, The selection of the point includes moving the medical device based on the movement of the controller, and selecting the point based on the positioning of the medical device.

20. The computer system according to claim 1, wherein, Rendering the first virtual geometry in the second view includes maintaining the points and the vectors in the second view, and rendering the first virtual geometry in the second view based on the points and the vectors.

21. The computer system according to claim 1, wherein, The processor is also configured to: Detecting the first part of the anatomical structure in the video; and The orientation of the anatomical structure is determined based on the first portion of the anatomical structure in the video.

22. The computer system according to claim 21, wherein, The first virtual geometry includes at least one of a virtual tube or a virtual cone, and wherein the first virtual geometry is added to the first view based on the orientation determined by the first portion of the anatomical structure and the boundary of the anatomical structure that becomes occluded and invisible in the video.

23. The computer system according to claim 21, wherein, The orientation of the anatomical structure is also determined based on preoperative scans of the anatomical structure.

24. The computer system according to claim 21, wherein, The processor is also configured to detect a second portion of the anatomical structure in the video, wherein the orientation of the anatomical structure is further determined based on the second portion of the anatomical structure in the video.

25. The computer system according to claim 21, wherein, The anatomical structure is at least one of the ureter, blood vessel, nerve, or pedicle.

26. A method for adding virtual reference geometry to a video of an anatomical structure, the method comprising: Receive the video of the anatomical structure, wherein the video shows a first view; Receive selections of points and vectors in the first view; A first virtual geometry is added to the first view based on the points and the vectors; Detect the movement that transforms the first view into the second view; and The first virtual geometry is presented in the second view such that the pose of the first virtual geometry is maintained from the first view to the second view.

27. The method of claim 26, further comprising using simultaneous positioning and mapping processing to determine the position of the first virtual geometry in the second view.

28. The method of claim 26, further comprising using kinematic data to determine the positioning of the first virtual geometry in the second view.

29. The method of claim 26, further comprising rotating the first virtual geometry based on the detected rotation of the controller.

30. The method of claim 26, further comprising adjusting the depth of the first virtual geometry based on input from the controller.

31. The method of claim 26, further comprising adding a second virtual geometry to the first view, wherein, The second virtual geometry has a different pose than the first virtual geometry.

32. The method of claim 26, further comprising determining the depth of the first virtual geometry based on an ultrasound image of the anatomical structure.

33. The method according to claim 32, wherein, The video shows a portion of the anatomical structure obscured and not visible, and the ultrasound image shows that portion of the anatomical structure.

34. The method according to claim 32, wherein, The depth of the first virtual geometry is below the outer surface of the anatomical structure shown in the video.

35. The method of claim 26, further comprising determining when the medical device has moved within the margin of the first virtual geometry.

36. The method of claim 35, further comprising presenting an alarm in response to movement of the medical device within the margin of the first virtual geometry.

37. The method of claim 35, further comprising presenting the distance between the medical device and the first virtual geometry.

38. The method according to claim 26, wherein, The first virtual geometry includes a virtual plane, and wherein maintaining the pose of the first virtual geometry includes maintaining the depth of the virtual plane from the first view to the second view.

39. The method according to claim 26, wherein, The point is located on the surface of the anatomical structure, and wherein the first virtual geometry is tangent to the surface of the anatomical structure.

40. The method according to claim 39, wherein, The selection of the point is based on the positioning of the camera device that captures the video.

41. The method according to claim 40, wherein, The normals of the first virtual geometry are oriented according to the orientation of the camera device.

42. The method according to claim 39, wherein, The selection of the point includes moving the point based on the movement of the controller, and selecting the point based on input from the controller.

43. The method of claim 42, further comprising preventing the medical device from moving during the movement of the controller.

44. The method according to claim 42, wherein, The selection of the point includes moving the medical device based on the movement of the controller, and selecting the point based on the positioning of the medical device.

45. The method according to claim 26, wherein, Rendering the first virtual geometry in the second view includes maintaining the points and the vectors in the second view, and rendering the first virtual geometry in the second view based on the points and the vectors.

46. ​​The method of claim 26, further comprising: Detecting a first portion of the anatomical structure in the video; as well as The orientation of the anatomical structure is determined based on the first portion of the anatomical structure in the video.

47. The method according to claim 46, wherein, The first virtual geometry includes at least one of a virtual tube or a virtual cone, and wherein the first virtual geometry is added to the first view based on the orientation determined by the first portion of the anatomical structure and the boundary of the anatomical structure that becomes occluded and invisible in the video.

48. The method according to claim 46, wherein, The orientation of the anatomical structure is also determined based on preoperative scans of the anatomical structure.

49. The method of claim 46, further comprising detecting a second portion of the anatomical structure in the video, wherein, The orientation of the anatomical structure is also determined based on the second part of the anatomical structure in the video.

50. The method of claim 46, wherein, The anatomical structure is at least one of the ureter, blood vessel, nerve, or pedicle.

51. A non-transitory machine-readable medium storing instructions for adjusting a model of an anatomical object, the instructions causing the processor, when executed by a processor, to: Perform the method according to any one of claims 26 to 50.

Images