Computer assisted dissection

Abstract

The present disclosure describes a computer-assisted robotic system and a method for performing a dissection. The system includes a memory and a controller communicatively coupled to the memory. The controller receives, from an ultrasound sensor, first ultrasound images showing a first anatomical structure and generates, based on the first ultrasound images, a first model of the first anatomical structure. The controller also moves, based on a first user input, a first surgical instrument and moves, based on the first model and the movement of the first surgical instrument, at least one of the ultrasound sensor or a camera.

Description

COMPUTER ASSISTED DISSECTIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims benefit of co-pending United States provisional patent application Serial No. 63 / 610,469 filed December 15, 2023. The aforementioned related patent application is herein incorporated by reference in its entirety.TECHNICAL FIELD

[0002] The present disclosure relates generally to medical systems (e.g., surgical systems). Specifically, the present disclosure relates to a medical system that provides computer assistance or guidance during dissection.BACKGROUND

[0003] Doctors use computer assisted medical systems to perform operations on patients, even remotely. These medical systems provide the doctors various views of surgical sites during the operations. For many operations, the doctors first removes outer layers of fat or tissue before reaching the target anatomy. This procedure is referred to as dissection. Dissection, however, may take a significant amount of time, which increases the duration of the procedure and doctor fatigue, increasing risk to the health and safety of the patient. For example, during dissection (and other surgical procedures), the doctor may spend time and energy repositioning cameras and other sensors to achieve a better view of the surgical site.SUMMARY

[0004] The present disclosure describes a computer-assisted robotic system and a method for performing a dissection. According to an embodiment, a computer- assisted robotic system includes a memory and a controller communicatively coupled to the memory. The controller receives, from an ultrasound sensor, first ultrasound images showing a first anatomical structure and generates, based on the first ultrasound images, a first model of the first anatomical structure. The controller also moves, based on a first user input, a first surgical instrument and moves, based on the first model and the movement of the first surgical instrument, at least one of the ultrasound sensor or a camera.

[0005] According to another embodiment, a method includes receiving, from an ultrasound sensor, first ultrasound images showing the first anatomical structure and generating, based on the first ultrasound images, a first model of the first anatomical structure. The method also includes moving, based on a first user input, a first surgical instrument and moving, based on the first model and the movement of the first surgical instrument, at least one of the ultrasound sensor or a camera. Other embodiments includes a non-transitory machine-readable medium storing instructions that, when executed by a controller, cause the controller to perform the method.

[0006] The foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 illustrates an example medical system.

[0008] Figures 2A and 2B illustrate example components of the medical system of Figure 1.

[0009] Figure 3 illustrates an example operation for performing a dissection in the medical system of Figure 1 .

[0010] Figure 4 illustrates an example operation for updating a model in the medical system of Figure 1 .

[0011] Figure 5 illustrates an example operation for moving an ultrasound sensor or camera in the medical system of Figure 1.

[0012] Figure 6 illustrates an example operation for moving an ultrasound sensor in the medical system of Figure 1 .

[0013] Figure 7 illustrates an example operation for moving a camera in the medical system of Figure 1 .

[0014] Figure 8 illustrates an example operation for adjusting energy output in the medical system of Figure 1 .

[0015] Figure 9 illustrates an example operation for monitoring distances in the medical system of Figure 1 .

[0016] Figure 10 illustrates an example operation for updating a model in the medical system of Figure 1 .

[0017] Figure 11 is a flowchart of an example method for performing a dissection and performed by the medical system of Figure 1 .

[0018] Figure 12 is a flowchart of an example method for adjusting energy output and performed by the medical system of Figure 1.

[0019] Figure 13 is a flowchart of an example method for monitoring distances and performed by the medical system of Figure 1.DETAILED DESCRIPTION

[0020] Doctors use computer assisted medical systems (e.g., surgical systems) to perform operations on patients, even remotely. These medical systems provide the doctors various views of surgical sites during the operations. To perform the operations, the doctors may move user input devices to control medical instruments (e.g., surgical instruments) at the surgical sites. For many operations, the doctors first remove outer layers of fat or tissue before reaching the target anatomy. This procedure is referred to as dissection. Dissection, however, may take a significant amount of time, which increases the duration of the procedure and doctor fatigue, increasing risk to the health and safety of the patient. For example, not only does removing the outer layers of fat or tissue take time, but as the procedure continues, the doctor may also spend time changing the pose of a camera and other sensors to achieve a better view of the surgical site or target site, which further increases the duration of the dissection and the operation.

[0021] The present disclosure describes a computer-assisted robotic system that assists doctors when performing dissections. Generally, the system uses ultrasound images to generate a model (e.g., a three-dimensional model) of the anatomical structures at the target site, which serves as a map of the target site. A doctor maymove a controller to move a surgical instrument to perform the dissection. The system may then automatically move an ultrasound sensor (e.g., that generated the ultrasound images) and / or a camera (e.g., that generates a video of the anatomical structure) based on the model and the movement of the surgical instrument. The system may update the model of the anatomical structure based on ultrasound images and / or video captured by the ultrasound sensor and / or camera after moving the ultrasound sensor and / or camera.

[0022] In certain embodiments, the computer-assisted robotic system provides several technical advantages. For example, by automatically moving the ultrasound sensor and / or camera based on the model and the movement of the surgical instrument, the system provides an unobstructed or clearer view of the area being dissected. The surgeon may not need to move the ultrasound sensor and / or camera, which reduces the duration of the dissection and improves the health and safety of the patient. As another example, by updating the model of the anatomical structure, the system provides updated information to the doctor during the dissection, which further improves the health and safety of the patient. Additionally, the updated model allows the system to perform more precise movements to provide clearer or unobstructed views of the area being dissected, which also improves the health and safety of the patient.

[0023] Figure 1 shows an example computer-assisted surgical system 100 (which may also be referred to as a computer-assisted robotic system) that implements some of the features described herein.

[0024] The surgical system 100 includes a manipulator assembly 102, a user control apparatus 104, and an auxiliary apparatus 106, all of which are communicatively coupled to each other. The surgical system 100 is utilized by a medical team to perform a computer-assisted medical procedure or other similar operation on a body of a patient 108 or on any other body as may serve a particular implementation. The medical team includes a first user 110-1 (such as a surgeon for a surgical procedure), a second user 110-2 (such as a patient-side assistant), a third user 110-3 (such as another assistant, a nurse, a trainee, etc.), and a fourth user 110- 4 (such as an anesthesiologist for a surgical procedure), all of whom are collectively referred to as users 110, and each of whom may control, interact with, or otherwise bea user of the surgical system 100. More, fewer, or alternative users may be present during a medical procedure as may serve a particular implementation. For example, team composition for different medical procedures, or for non-medical procedures, may differ and include users with different roles.

[0025] Although Figure 1 illustrates an ongoing minimally invasive medical procedure such as a minimally invasive surgical procedure, it will be understood that the surgical system 100 may similarly be used to perform open medical procedures or other types of operations. For example, operations such as exploratory imaging operations, mock medical procedures used for training purposes, and / or other operations may also be performed.

[0026] The manipulator assembly 102 includes one or more manipulator arms 112 (e.g., manipulator arms 112-1 through 112-4) to which one or more instruments may be coupled. The instruments are used for a computer-assisted surgical procedure on the patient 108 (e.g., by being at least partially inserted into the patient 108 and manipulated within the patient 108). While the manipulator assembly 102 is depicted and described herein as including four manipulator arms 112, the manipulator assembly 102 may include a single manipulator arm 112 or any other number of manipulator arms as may serve a particular implementation. Although the example of Figure 1 illustrates the manipulator arms 112 as robotic manipulator arms, one or more instruments may be partially or entirely manually controlled, such as by being handheld and controlled manually by a person. These partially or entirely manually controlled instruments are used in conjunction with, or as an alternative to, computer- assisted instrumentation that is coupled to the manipulator arms 112.

[0027] During the medical operation, the user control apparatus 104 facilitates teleoperational control by the user 110-1 of the manipulator arms 112 and instruments attached to the manipulator arms 112. To this end, the user control apparatus 104 provides the user 110-1 with imagery of an operational area associated with the patient 108 as captured by an imaging device. The manipulator arms 112 or any instruments coupled to the manipulator arms 112 mimic the dexterity of the hand, wrist, and fingers of the user 110-1 across multiple degrees of freedom of motion. In this manner, the user 110-1 intuitively performs a procedure (e.g., an incision procedure, a suturingprocedure, etc.) using one or more of the manipulator arms 112 or any instruments coupled to the manipulator arms 112.

[0028] The auxiliary apparatus 106 includes one or more computing devices that perform auxiliary functions in support of the procedure, such as providing insufflation, electrocautery energy, illumination or other energy for imaging devices, image processing, or coordinating components of the surgical system 100. The auxiliary apparatus 106 includes a display monitor 114 that displays one or more user interfaces, or graphical or textual information in support of the procedure. In some instances, the display monitor 114 is a touchscreen display that provides user input functionality. Augmented content provided by a region-based augmentation system may be similar to, or differ from, content associated with the display monitor 114 or one or more display devices in the operation area (not shown).

[0029] The manipulator assembly 102, user control apparatus 104, and auxiliary apparatus 106 are communicatively coupled one to another in any suitable manner. The manipulator assembly 102, user control apparatus 104, and auxiliary apparatus 106 may be communicatively coupled by way of control lines 116, which represent any wired or wireless communication link as may serve a particular implementation. To this end, the manipulator assembly 102, user control apparatus 104, and auxiliary apparatus 106 may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, and so forth.

[0030] Figure 2A illustrates an example manipulator assembly 102. As seen in Figure 2A, the manipulator assembly 102 includes a base 118, a manipulator arm 112- 1 , a manipulator arm 112-2, a manipulator arm 112-3, and a manipulator arm 112-4. Each manipulator arm 112-1 , 112-2, 112-3, and 112-4 is pivotably coupled to the base 118. Although the base 118 may include casters to allow ease of mobility, in some embodiments, the manipulator assembly 102 is fixedly mounted to a floor, ceiling, operating table, structural framework, or the like.

[0031] In a typical procedure, two of the manipulator arms 112-1 , 112-2, 112-3, or 112-4 hold surgical instruments and a third holds a stereo endoscope. The remaining manipulator arms are available so that other instruments may be introduced at thework site. Alternatively, the remaining manipulator arms may be used for introducing another endoscope or another image capturing device, such as an ultrasound transducer, to the work site.

[0032] Each of the manipulator arms 112-1 , 112-2, 112-3, and 112-4 are formed of links that are coupled together and manipulated through actuatable joints. Each of the manipulator arms 112-1 , 112-2, 112-3, and 112-4 may include a setup arm and a device manipulator. The setup arm positions its held device so that a pivot point occurs at its entry aperture into the patient. The device manipulator may then manipulate its held device so that the held device may be pivoted about the pivot point, inserted into and retracted out of the entry aperture, and rotated about its shaft axis. Each of the manipulator arms 112-1 , 112-2, 112-3, and 112-4 may include sensors (e.g., kinematics sensors, position sensors, accelerometers, etc.) that detect or track movement of the manipulator arms 112-1 , 112-2, 112-3, and 112-4. For example, these sensors may detect how far or how quickly a manipulator arm 112-1 , 112-2, 112-3, or 112-4 moves in a certain direction.

[0033] Figure 2B illustrates an example user control apparatus 104. The user control apparatus 104 includes a stereo vision display 120 so that the user may view the surgical work site in stereo vision from images captured by the stereoscopic camera of the manipulator assembly 102. Left and right eyepieces 122 and 124 are provided in the stereo vision display 120 so that the user may view left and right display screens inside the display 120 respectively with the user's left and right eyes. While viewing typically an image of the surgical site on a suitable viewer or display, the surgeon performs the surgical procedures on the patient by manipulating control input devices, which in turn control the motion of robotic instruments.

[0034] The user control apparatus 104 also includes left and right input devices 126 and 128 that the user grasps respectively with his / her left and right hands to manipulate devices (e.g., surgical instruments) being held by the manipulator arms 112-1 , 112-2, 112-3, and 112-3 of the manipulator assembly 102 in preferably six or more degrees of freedom (“DOF”). Foot pedals 130 with toe and heel controls are provided on the user control apparatus 104 so the user may control movement and / or actuation of devices associated with the foot pedals.

[0035] A processing device 132 is provided in the user control apparatus 104 for control and other purposes. The processing device 132 performs various functions in the surgical system 100. One function performed by processing device 132 is to translate and transfer the mechanical motion of input devices 126 and 128 to actuate their corresponding joints in their associated manipulator arms 112-1 , 112-2, 112-3, and 112-4 so that the surgeon can effectively manipulate devices, such as the surgical instruments. Another function of the processing device 132 is to implement the methods, crosscoupling control logic, and controllers or processors described herein. The auxiliary apparatus 106 includes a processing device 132 that performs the functions or actions described herein. The processing device 132 may include a controller and a memory that perform the functions described herein. The controller may include one or more processors.

[0036] The controller may include any electronic circuitry, including, but not limited to one or a combination of microcontrollers, microcontrollers, application specific integrated circuits (ASIC), application specific instruction set controller (AS IP), and / or state machines, that communicatively couples to a memory and controls the operation of the user control apparatus 104 and / or the auxiliary apparatus 106. The controller may be 8-bit, 16-bit, 32-bit, 64-bit or of any other suitable architecture. The controller may include an arithmetic logic unit (ALU) for performing arithmetic and logic operations, controller registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components. The controller may include other hardware that operates software to control and process information. The controller executes software stored on a memory to perform any of the functions described herein. The controller controls the operation and administration of the user control apparatus 104 or the auxiliary apparatus 106 by processing information (e.g., information received from the user control apparatus 104, the manipulator assembly 102, the auxiliary apparatus 106, and / or a memory). The controller is not limited to a single processing device and may encompass multiple processing devices contained in the same device or computer or distributed across multiple devices or computers. The controller is considered to perform a set of functions or actions if the multiple processing devices collectively perform the set offunctions or actions, even if different processing devices perform different functions or actions in the set.

[0037] Figures 3 through 10 illustrate example operations performed by a computer-assisted robotic system (e.g., the surgical system 100 of Figure 1 ). Generally, the system provides different control modes in which an operator (e.g., a doctor, surgeon, or other medical professional) of the system may move a medical instrument or interact with certain user interface elements.

[0038] Figure 3 illustrates an example operation 300 for performing a medical procedure (e.g., a dissection) in the surgical system 100 of Figure 1. In particular embodiments, the computer-assisted robotic system performs the operation 300. Generally, the system uses ultrasound images 302 to build a model 304 of an anatomical structure. The system moves repositionable structures 308A, 308B, 308C and 308D (e.g., the manipulator arms 12-1 , 12-2, 12-3 and 12-4) to perform the procedure. The system moves the repositionable structures 308A and 308D to move surgical instruments 310A and 310B. The system uses the model 304 to automatically move the repositionable structures 308B and 308C to move an ultrasound sensor 312 and a camera 314 held by the repositionable structures 308B and 308C. The ultrasound sensor 312 captures ultrasound images 302 of the target site, and the camera 314 captures video 316 of the target site. In this manner, the system allows a user to manually control the repositionable structures 308A and 308D to move the instruments 310A and 310B, while automatically repositioning the ultrasound sensor 312 and / or the camera 314 held by the repositionable structures 308B and 308C, which may provide the user a better view of the target site while reducing the number of repositionable structures 308 that the user manages.

[0039] The system receives ultrasound images 302 of one or more anatomical structures. These anatomical structures may be present at the target site. For example, the anatomical structures may include a mesentery or an omentum that is to be removed during a dissection. The anatomical structures may also include other tissue, organs, or structures that are not the targets of the dissection. The ultrasound sensor 312 captures the ultrasound images 302 by sweeping the target site. Each ultrasound image 302 may show a slice of the anatomical structures, which mayinclude portions of the anatomical structures that are beneath the surface of the anatomical structures and that may not be visible to the naked eye.

[0040] The system then uses computer vision techniques and a simultaneous localization and mapping (SLAM) process to generate a three-dimensional model 304 of the anatomical structures, which serves as a three-dimensional map of the target site. The system uses the computer vision techniques to identify the anatomical structures in the different ultrasound images 302. In some instances, the system may also use the computer vision techniques to analyze the video 316 to identify the anatomical structures and / or the instrument 310A or 310B in the video 316. The system then uses the SLAM process to layer or stitch the ultrasound images 302 together to form the model 304. The SLAM process may also use the views of the anatomical structures in the video 316 to form the model 304. The model 304 provides a three-dimensional view of the anatomical structures at the target site, and the model 304 shows the positioning of the anatomical structures with respect to each other. Because the ultrasound images 302 may show portions of the anatomical structures that are hidden or obstructed from view, the model 304 may also show portions of the anatomical structure that are hidden or obstructed from view (e.g., in the video 316). The system may also determine, using the SLAM process, the positions of the instrument 310A or 310B, the ultrasound sensor 312, and the camera 314 with respect to the anatomical structures. The model 304 thus maps the instrument 301 A or 310B, the ultrasound sensor 312, and the camera 314, as well as the anatomical structures, providing a three-dimensional map of the target site.

[0041] The system receives user input 306 from a user of the system. For example, the user may operate the input devices 126 and 128 to provide the user input 306. The system moves the repositionable structure 308A according to the user input 306, which also moves the instrument 310A held by the repositionable structure 308A. The system may also move the repositionable structure 308D, which holds the instrument 310B, according to the user input 306. By moving the repositionable structure 308A, the repositionable structure 308D, and / or the instruments 310A and 310B, the system performs a procedure (e.g., a dissection). For example, moving the instruments 310A and 310B may cause the instruments to cut and remove portions of an anatomical structure (e.g., a mesentery or omentum).

[0042] The instruments 310A and 31 OB may be energy based instruments that output energy to perform surgical procedures (e.g., cutting or sealing tissue). For example, the instruments 310A and 31 OB may use electric current, ultrasonic waves, microwaves, etc. to operate on the tissue. The energy output of the instruments 310A and 31 OB may be adjusted depending on the desired effect on the tissue (e.g., a desired depth of a cut, a thickness of tissue to be sealed or cut, etc.).

[0043] The system automatically moves the repositionable structures 308B and 308C, which hold the ultrasound sensor 312 and the camera 314, based on the model 304 and / or the movement of the repositionable structure 308A or 308D. In some embodiments, the system moves the repositionable structures 308B and 308C while the user provides the user input 306 that moves the repositionable structure 308A or 308D. As a result, the ultrasound sensor 312 and / or the camera 314 may be moved concurrent with the instrument 310A or 310B.

[0044] As an example, as portions of the anatomical structure are cut or removed, the system may determine the positioning and movement of the instrument 310A or 310B with respect to the anatomical structures in the model 304. The system then changes the poses (e.g., positions and orientations) of the ultrasound sensor 312 and the camera 314, such that the ultrasound sensor 312 sweeps the portion of the anatomical structure that was cut or removed and such that the camera 314 captures video 316 of that portion of the anatomical structure. The system then uses the ultrasound images 302 and the video 316 of the portion of the anatomical structure that was cut or removed in the computer vision technique and / or the SLAM process to update the model 304 to account for the portions that were cut or removed.

[0045] As another example, as the instrument 310A or 310B is moved, the system may track the position and movement of the instrument 310A or 310B with respect to the anatomical structures in the model 304. The system may determine, from the model 304, whether the movement of the instrument 310A or 310B causes the instrument 310A or 310B to be in a position that obstructs the view of the ultrasound sensor 312 or the camera 314. For example, the system may determine whether the instrument 310A or 310B is positioned along a line of sight of the ultrasound sensor 312 or the camera 314. If the instrument 310A or 310B obstructs the view of the ultrasound sensor 312 or the camera 314, the system moves the ultrasound sensor312 or the camera 314 to change the pose (e.g., position and orientation) of the ultrasound sensor 312 or the camera 314 so that the ultrasound sensor 312 or the camera 314 has an unobstructed view of the anatomical structure. In this manner, the system automatically repositions and redirects the ultrasound sensor 312 or the camera 314 to provide unobstructed views of the target site.

[0046] As yet another example, as the instrument 310A or 31 OB is moved to perform the medical procedure (e.g., dissection), the system may automatically move ultrasound sensor 312 to capture ultrasound images of other parts of the anatomical structure or of the target site. The system may even move the ultrasound sensor 312 out of view of the camera 314 and the video 316 to capture ultrasounds images of parts of the anatomical structure or of the target site that are out of view. The system then uses the computer vision technique and the SLAM process to analyze these ultrasound images and to update the model 304. In this manner, the system automatically sweeps parts of anatomical structures that are out of view to update the model 304.

[0047] Although the system may automatically move the repositionable structures 308B and 308C, the ultrasound sensor 312, and the camera 314, the system also allows the user to manually move the repositionable structures 308B and 308C, the ultrasound sensor 312, and the camera 314. The user may operate the user input device 126 or 128 after selecting or transferring control to one of the repositionable structures 308B or 308C to move the repositionable structure 308B or 308C. When the user wants to move the repositionable structure 308A or 308D, the user may select or transfer control to the repositionable structure 308A or 308D.

[0048] Figure 4 illustrates an example operation 400 for updating a model 304 in the surgical system 100 of Figure 1 . In particular embodiments, the computer-assisted robotic system performs the operation 400. By performing the operation 400, the system updates the model 304 so that the model 304 more accurately reflects the anatomical structure at the surgical site.

[0049] The system detects a movement 402 at the surgical site. For example, the system may use computer vision techniques to analyze the video 316 and / or the ultrasound images 302 to detect a movement 402 of the anatomical structure. Themovement 402 may cause the anatomical structure to change in shape. For example, the movement 402 may occur as part of the normal functioning of the anatomical structure (e.g., a beating or pulsing of the anatomical structure). As another example, the movement 402 may be a portion of the anatomical structure being cut or removed.

[0050] The system then updates the model 304 to reflect the change in shape of the anatomical structure. For example, the system may use the SLAM process to analyze the video 316 and / or the ultrasound images 302 and to update the model 304 to account for the change in shape. As a result, the model 304 reflects the anatomical structure with the changed shape. The user of the system may view the model 304 to more accurately understand the shape of the anatomical structure.

[0051] Figure 5 illustrates an example operation 500 for moving an ultrasound sensor 312 or camera 314 in the surgical system 100 of Figure 1. In particular embodiments, the computer-assisted robotic system performs the operation 500. By performing the operation 500, the system automatically moves the ultrasound sensor 312 or the camera 314 to provide updated information to the user.

[0052] The system receives the user input 502. The user of the system may provide the user input 502 by operating a user input device 126 or 128. The user input 502 indicates a portion 504 of the anatomical structure. The user input 502 may move an instrument 310A or 310B or the user input 502 may move a cursor on a user interface to indicate the portion 504 of the anatomical structure. For example, the instrument 310A or 310B or the cursor may be moved close to or near the portion 504.

[0053] The system automatically moves the repositionable structure 308B or 308C holding the ultrasound sensor 312 or the camera 314 based on the indicated portion 504 of the anatomical structure. The system may move the ultrasound sensor 312 or the camera 314 to change the pose of the ultrasound sensor 312 or the camera 314 to direct the ultrasound sensor 312 or the camera 314 at the indicated portion 504 of the anatomical structure. After the system directs the ultrasound sensor 312 or the camera 314 at the portion 504 of the anatomical structure, the ultrasound sensor 312 or the camera 314 capture updated views of the portion 504 of the anatomical structure. For example, after the system moves the ultrasound sensor 312, the ultrasound sensor 312 may capture ultrasound images 506. The ultrasound images506 show the indicated portion of 504 of the anatomical structure. In this manner, the system automatically sweeps the ultrasound sensor 312 over the indicated portion 504. As another example, after the system moves the camera 314, the camera 314 captures video that shows the indicated portion 504 of the anatomical structure.

[0054] The system updates the model 304 according to the ultrasound images 506. For example, the ultrasound images 506 may show parts of the anatomical structure that were previously not seen when generating the model 304. After receiving the ultrasound images 506, the system updates the model 304 to account for the parts of the anatomical structure shown in the ultrasound images 506. As a result, the model 304 more accurately depicts the shape of the anatomical structure.

[0055] As an example operation, the user notice that a part of the model 304 does not accurately reflect the anatomical structure (e.g., due to changes to the anatomical structure or due to a portion of the anatomical structure not being swept by the ultrasound sensor 312 when the model 304 was generated). The user may provide the user input 502 to indicate a portion 504 of the anatomical structure that is shown in that part of the model 304. The system then moves the ultrasound sensor 312 and / or the camera 314 to direct the ultrasound sensor 312 and / or the camera 314 at the portion 504 of the anatomical structure. For example, the ultrasound sensor 312 may sweep the portion 504 of the ultrasound sensor to produce the ultrasound images 506. The system then uses computer vision techniques and the SLAM process to analyze the ultrasound images 506 and to update the part of the model 304. In this manner, the system automatically updates the model 304 in response to user indication of a part of the model 304 to update.

[0056] Figure 6 illustrates an example operation for moving an ultrasound sensor 312 in the surgical system 100 of Figure 1. In particular embodiments, the computer- assisted robotic system performs the operation 600. By performing the operation 600, the system moves the ultrasound sensor 312 based on information gleaned from the updated model 304. The model 304 may have been updated according to the operation 400 or the operation 500.

[0057] The system identifies a portion 602 of the anatomical structure. For example, a user of the system may provide user input that indicates the portion 602 ofthe anatomical structure. The portion 602 may be different from the portion 504 of the anatomical structure that was used to update the model 304. The system moves the repositionable structure 308B to move the ultrasound sensor 312 based on the portion 602 of the anatomical structure. For example, the system may move the repositionable structure 308B to change the pose of the ultrasound sensor 312 so that the ultrasound sensor 312 sweeps the portion 602 of the anatomical structure. The ultrasound sensor 312 may then produce ultrasound images that show the portion 602 of the anatomical structure. In some embodiments, the system updates the model 304 using these ultrasound images. In this manner, the system continues to sweep different portions of the anatomical structure using the ultrasound sensor 312 to provide ultrasound images of different portions of the anatomical structure.

[0058] Figure 7 illustrates an example operation for moving a camera 314 in the surgical system 100 of Figure 1. In particular embodiments, the computer-assisted robotic system performs the operation 700. By performing the operation 700, the system adjust the pose of the camera 314 to maintain views of the ultrasound sensor 312 or the instrument 310A.

[0059] The system monitors or tracks a position 702 and a movement 704 of the repositionable structure 308B or the ultrasound sensor 312. The system also tracks or maintains a position 706 and a movement 708 of the repositionable structure 308A or the instrument 310A. For example, as the system moves the repositionable structure 308B or the ultrasound sensor 312, the system tracks the movement 704 of the ultrasound sensor 312 and updates the changing position 702 of the ultrasound sensor 312. As the repositionable structure 308A or the instrument 310A moves, the system tracks the movement 708 of the instrument 310A and updates the changing position 706 of the instrument 310A.

[0060] The movement of the instrument 310A or the ultrasound sensor 312 may cause the instrument 310A or the ultrasound sensor 312 to move out of view in the video 316 captured by the camera 314. As a result, the user of the system may lose sight of the instrument 310A or the ultrasound sensor 312. The system automatically moves the repositionable structure 308C or the camera 314 to keep the instrument 310A or the ultrasound sensor 312 in view. For example, the system may detect, based on the position 702 or the movement 704 of the ultrasound sensor 312, that theultrasound sensor 312 has moved out of view in the video 316. As another example, the system may determine from the position 706 or the movement 708, that the instrument 310A has moved out of view in the video 316. In response, the system moves the repositionable structure 308C or the camera 314 to change the pose of the camera 314 (e.g., the position and / or orientation of the camera 314). After moving the camera 314, the camera 314 captures an updated video 316. The updated video 316 may show the ultrasound sensor 312 and / or the instrument 310A.

[0061] For example, if the ultrasound sensor 312 and the instrument 310A move apart from each other, such that one or more of the ultrasound sensor 312 or the instrument 310A move out of view of the video 316, the system may move the camera 314 to capture a wider or larger view of the target site such that the video 316 shows both the ultrasound sensor 312 and the instrument 310A. In this manner, the system automatically moves or changes the pose of the camera 314 so that the ultrasound sensor 312 and / or the instrument 310A remain visible to the user.

[0062] Figure 8 illustrates an example for adjusting energy output in the surgical system 100 of Figure 1. In particular embodiments, the computer-assisted robotic system performs the operation 800. By performing the operation 800, the system automatically adjusts the energy output of the instrument 310A based on information from the video 316 or the ultrasound images 302.

[0063] The system receives the video 316 and the ultrasound images 302 from the camera 314 and the ultrasound sensor 312. The video 316 and the ultrasound images 302 may show an anatomical structure (e.g., a mesentery or an omentum) on which the operation is being performed. The system determines a characteristic 802 of the anatomical structure from the video 316 and / or the ultrasound images 302. For example, the system may determine from the video 316 and / or the ultrasound images 302 a thickness of the anatomical structure.

[0064] The system then adjusts an energy output 804 of the instrument 310A according to the determined characteristic 802. The instrument 310A and 310B may be For example, if the characteristic 802 is a thickness of the anatomical structure, then the system may increase or decrease the energy output 804 of the instrument 310A appropriate for the thickness. If the anatomical structure is thick, then the systemmay increase the energy output 804 so that the instrument 310A cuts deeper into the anatomical structure. If the anatomical structure is thin, then the system may decrease or reduce the energy output 804 so that the instrument 31 OA cuts shallower into the anatomical structure. In this manner, the system automatically adjusts the energy output 804 so that the instrument 31 OA does not cut or dissect too deep into the anatomical structure.

[0065] In some embodiments, the system receives haptic output 806 from a haptic sensor. The haptic sensor may be attached to the ultrasound sensor 312. When the ultrasound sensor 312 is pressed against the anatomical structure (e.g., to generate ultrasound images), the haptic sensor detects a pressure or force exerted on the ultrasound sensor 312 by the anatomical structure. The haptic sensor generates the haptic output 806 that indicates the amount of force or pressure on the ultrasound sensor 312.

[0066] The system uses the haptic output 806, the video 316, and / or the ultrasound images 302 to reference into a database 808 to determine a tissue type 810 of the anatomical structure. For example, the system may use the view of the anatomical structure shown in the video 316 and the pressure exerted on the haptic sensor by the anatomical structure indicated by the haptic output 806 to reference into the database 808. The database 808 may include a table that links tissue types to certain views of anatomical structures and haptic outputs. By referencing into the database 808, the database 808 may return the tissue type 810 for the anatomical structure. The system includes the tissue type 810 in the characteristic 802.

[0067] The system then adjusts the energy output 804 based (at least partially) on the determined tissue type 810. For example, the system may increase or reduce the energy output 804 so that the instrument 310A cuts into the tissue type 810 to a desired depth. If the tissue type 810 indicates that the tissue is easier to cut, then the system may reduce the energy output 804. If the tissue type 810 indicates that the tissue is tougher or more difficult to cut, then the system may increase the energy output 804. In this manner, the system adjusts the energy output 804 so that the instrument 310A appropriately handles the determined tissue type 810.

[0068] Figure 9 illustrates an example operation 900 for monitoring distances in the surgical system 100 of Figure 1. In particular embodiments, the computer-assisted robotic system performs the operation 900. By performing the operation 900, the system helps keep the instrument 310A from damaging or injuring other anatomical structures that are not the target of the operation.

[0069] As discussed previously, the system receives the ultrasound images 302 generated by the ultrasound sensor 312. The ultrasound images 302 may show multiple anatomical structures. Some of these anatomical structures may not be the target of the operation. As a result, it may not be desirable for the instrument 310A to approach too close to these anatomical structures. The model 304 serves as a three- dimensional map of the surgical site. The model 304 may include anatomical structures that are targets of the operation and anatomical structures that are not.

[0070] The system uses the model 304 to monitor or track a distance 904 between the instrument 310A and the anatomical structure. For example, the system may use a SLAM process to analyze a video from the camera 314 to determine the position of the instrument 310A with respect to the model 304 as the instrument 310A moves. Specifically, the system may determine the distance 904 between the instrument 310A and an anatomical structure that is shown in the model 304 and that is not the target of the operation.

[0071] The system compares the distance 904 with a threshold 906 to determine whether the instrument 310A is positioned too close to the anatomical structure. If the distance 904 exceeds the threshold 906, then the system determines that the instrument 310A is sufficiently far from the anatomical structure. If the distance 904 falls below the threshold 906, the system determines that the instrument 310A is positioned too close to the anatomical structure and that the instrument 310A may pose a danger to the anatomical structure. In response, the system generates and communicates an alert 908 to the user of the system. The alert 908 indicates that the instrument 310A is too close to the anatomical structure and that the instrument 310A should be moved further away from the anatomical structure. In this manner, the system helps keep the instrument 310A from being positioned too close to the anatomical structure and potentially damaging or injuring the anatomical structure.

[0072] In some embodiments, the system moves the repositionable structure 308B or 308C to change the pose of the ultrasound sensor 312 or the camera 314 when the instrument 310A is positioned too close to the anatomical structure. For example, the system may change the pose of the ultrasound sensor 312 or the camera 314 to direct the ultrasound sensor 312 or the camera 314 at a region where the instrument 310A is nearing or within the threshold 906 distance of the anatomical structure. In this manner, the system provides the user a view that shows how close the instrument 310A is to the other anatomical structure, which may help the user understand how and where to move the instrument 310A further away from the other anatomical structure.

[0073] Figure 10 illustrates an example operation 1000 for updating a model 304 in the surgical system 100 of Figure 1. In particular embodiments, the computer- assisted robotic system performs the operation 1000. By performing the operation 1000, the system moves the ultrasound sensor 312 to update the model 304.

[0074] As discussed previously, the system moves the repositionable structure 308A and the instrument 310A according to user input 306. As the system moves, the instrument 310A, the system determines and tracks the position 1002 of the instrument 310A. The system may change the pose of the camera 314 as the instrument 310A moves so that the instrument 310A remains in view in the video 316.

[0075] The system determines a portion 1004 of the model 304 based on the position of 1002 of the instrument 310A. For example, the user may move the instrument 310A to a position 1002 near a part of an anatomical structure corresponding to the portion 1004 of the model 304. The system then determines that the user has indicated the portion 1004 of the model 304 by moving the instrument 310A near the portion 1004.

[0076] The system moves the repositionable structure 308B and the ultrasound sensor 312 to change a pose of the ultrasound sensor 312. The ultrasound sensor 312 is then directed to the part of the anatomical structure that corresponds to the portion 1004 of the model 304. The ultrasound sensor 312 then sweeps that part of the anatomical structure to generate ultrasound images 1006. The ultrasound images1006 show the part of the anatomical structure that correspond to the portion 1004 of the model 304.

[0077] The system then updates the model 304 using the ultrasound images 1006. For example, the ultrasound images 1006 may show the shape or structure of the part of the anatomical structure corresponding to the portion 1004 of the model 304. The system updates the portion 1004 of the model 304 such that the portion 1004 matches the size and shape of the part of the anatomical structure shown in the ultrasound images 1006 (e.g., using computer vision techniques and / or a SLAM process). In this manner, the system allows the user to indicate the portion 1004 of the model 304 that should be updated, and the system automatically moves the ultrasound sensor 312 and updates the model 304 using the ultrasound images 1006 from the ultrasound sensor 312.

[0078] As an example, the user may determine a part of the anatomical structure is critical or important to the operation. The user may move the instrument 310A near this part of the anatomical structure to indicate that this part is critical or important. The system then determines the portion 1004 of the model 304 that shows this part of the anatomical structure. The system may then understand from the model 304 how to move the ultrasound sensor 312 to sweep this part of the anatomical structure. The system uses the ultrasound sensor 312 to sweep this part of the anatomical structure and to generate ultrasound images 1006 of this part of the anatomical structure. The system then uses computer vision techniques and / or the SLAM process to update the model 304. As a result, the model 304 updates and shows the important or critical part of the anatomical structure.

[0079] Figure 11 is a flowchart of an example method 1100 for performing a dissection and performed by the surgical system 100 of Figure 1. In particular embodiments, the computer-assisted robotic system performs the method 1100. By performing the method 1100, the system automatically moves an ultrasound sensor 312 or camera 314 as a user moves or controls the instrument 310A.

[0080] In block 1102, the system receives ultrasound images 302. The ultrasound images 302 are produced by the ultrasound sensor 312 sweeping a target site that includes one or more anatomical structures (e.g., a mesentery or omentum). Theultrasound images 302 may show portions of the anatomical structures that are beneath the surface of the anatomical structures and that are not visible in the video 316 from the camera 314.

[0081] In block 1104, the system generates the model 304 using the ultrasound images 302. The system may use a computer vision technique to identify the anatomical structures in the ultrasound images 302. The system uses a SLAM process to stitch or layer the ultrasound images 302 together to form the model 304. The SLAM process may also determine the positioning of the instrument 310A, the ultrasound sensor 312, and the camera 314 with respect to the anatomical structures shown in the model 304. As a result, the model 304 serves as a three-dimensional map of the target site.

[0082] In block 1106, the system moves the instrument 310A. For example, the user of the system may operate the user input device 126 or 128 to provide the user input 306. The system moves the repositionable structure 308A according to the user input 306, which moves the instrument 310A. Moving the instrument 310A may cut or remove portions of an anatomical structure at the target site.

[0083] In block 1108, the system moves the ultrasound sensor 312 or the camera 314 according to the model 304 and the movement of the instrument 310A. For example, the system may move the ultrasound sensor 312 or the camera 314 to direct the ultrasound sensor 312 or the camera 314 towards a portion of the anatomical structure indicated by the instrument 310A. As another example, the system may move the ultrasound sensor 312 or the camera 314 to maintain a view of the instrument 310A as the instrument 310A moves. As another example, the system may move the ultrasound sensor 312 or the camera 314 to provide an unobstructed view of the anatomical structure. In this manner, the system reduces the number of tasks and responsibilities that the user oversees during the operation.

[0084] Figure 12 is a flowchart of an example method 1200 for adjusting the energy output 804 of the instrument 310A and performed by the surgical system 100 of Figure 1. In particular embodiments, the computer-assisted robotic system performs the method 1200. By performing the method 1200, the system adjusts the energy output804 of the instrument 310A such that the energy output 804 is more suitable for a characteristic 802 of the anatomical structure.

[0085] In block 1202, the system determines the characteristic 802 of the anatomical structure. For example, the system may analyze the video 316 and the ultrasound images 302 of the anatomical structure to determine a thickness of the anatomical structure. As another example, the system may use the video 316, the ultrasound images 302, and / or the haptic output 806 from a haptic sensor attached to the ultrasound sensor 312 to determine a tissue type 810 of the anatomical structure.

[0086] In block 1204, the system adjusts the energy output 804 of the instrument 310A according to the characteristic 802 of the anatomical structure. For example, if the anatomical structure is thick, the system may increase the energy output 804 of the instrument 310A so that the instrument 310A cuts deeper into the anatomical structure. As another example, if the tissue type 810 indicates that the anatomical structure has a tougher tissue, then the system may increase the energy output 804 so that the instrument 310A may more suitably cut the tougher tissue.

[0087] Figure 13 is a flowchart of an example method 1300 for monitoring distances and performed by the surgical system 100 of Figure 1 . In particular embodiments, the computer-assisted robotic system performs the method 1300. By performing the method 1300, the system monitors the distance between the instrument 310A and an anatomical structure.

[0088] In block 1302, the system monitors the position of the instrument 310A. When the user moves the instrument 310A, the system updates the position of the instrument 310A. In block 1304, the system determines that the instrument 310A is within the threshold 906 distance of an anatomical structure. For example, the system may use the position of the instrument 310A and the model 304 to determine the distance 904 between the instrument 310A and the anatomical structure. The system may compare the distance 904 to the threshold 906. If the distance 904 falls below the threshold 906, then the system determines that the instrument 310A is within the threshold 906 distance of the anatomical structure.

[0089] In block 1306, the system communicates the alert 908 indicating that the instrument 310A is too close to the anatomical structure. The alert 908 may indicatethat the instrument 31 OA should be moved further away from the anatomical structure to avoid damaging or injuring the anatomical structure. The user may move the instrument 31 OA further away from the anatomical structure after seeing the alert 908.

[0090] In summary, the computer-assisted robotic system assists doctors when performing medical procedures (e.g., dissections). Generally, the system uses ultrasound images to generate a model (e.g., a three-dimensional model) of an anatomical structure. A doctor may move a controller to move a surgical instrument to perform the procedure. The system then automatically moves an ultrasound sensor (e.g., that generated the ultrasound images) and / or a camera (e.g., that generates a video of the anatomical structure) based on the model and the movement of the surgical instrument. The system may update the model of the anatomical structure based on ultrasound images and / or video captured by the ultrasound sensor and / or camera after moving the ultrasound sensor and / or camera.

[0091] This description and the accompanying drawings that illustrate aspects, embodiments, or modules should not be taken as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure other features. Like numbers in two or more figures represent the same or similar elements.

[0092] In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.

[0093] Further, the terminology in this description is not intended to be limiting. For example, spatially relative terms-such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like may be used to describe one element’s or feature’s relationship to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass different positions (i.e., locations) and orientations (i.e., rotational placements) of the elements or their operation in addition to the position and orientation shown in the figures. For example, if the content of one of the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features. Thus, the exemplary term “below” can encompass both positions and orientations of above and below. A device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. Likewise, descriptions of movement along and around various axes include various special element positions and orientations. In addition, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context indicates otherwise. And, the terms “comprises”, “comprising”, “includes”, and the like specify the presence of stated features, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

[0094] Elements described in detail with reference to one embodiment, or module may, whenever practical, be included in other embodiments, or modules in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, or application may be incorporated into other embodiments, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or embodiments non-functional, or unless two or more of the elements provide conflicting functions.

[0095] In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0096] This disclosure describes various devices, elements, and portions of computer-assisted devices and elements in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an element or a portion of an element in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian x-, y-, and z-coordinates). As used herein, the term “orientation” refers to the rotational placement of an element or a portion of an element (three degrees of rotational freedom - e.g., roll, pitch, and yaw). As used herein, the term “shape” refers to a set positions or orientations measured along an element. As used herein, and for a device with repositionable arms, the term “proximal” refers to a direction toward the base of the computer-assisted device along its kinematic chain and “distal” refers to a direction away from the base along the kinematic chain.

[0097] Aspects of this disclosure are described in reference to computer-assisted systems and devices, which may include systems and devices that are teleoperated, remote-controlled, autonomous, semiautonomous, robotic, and / or the like. Further, aspects of this disclosure are described in terms of an embodiment using a medical system, such as the DA VINCI SURGICAL SYSTEM or ION SYSTEM commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that aspects disclosed herein may be embodied and implemented in various ways, including robotic and, if applicable, non-robotic embodiments. Techniques described with reference to surgical instruments and surgical methods may be used in other contexts. Thus, the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperational systems. As further examples, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, and / or the like. Additional example applications include use for procedures on tissue removed from human oranimal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.

[0098] Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. Thus, the scope of the disclosure should be limited only by the following claims, and it is appropriate that the claims be construed broadly and, in a manner, consistent with the scope of the embodiments disclosed herein.

Claims

WHAT IS CLAIMED IS:1 . A computer-assisted robotic system comprising: a memory; and a controller communicatively coupled to the memory, the controller configured to: receive, from an ultrasound sensor, first ultrasound images showing a first anatomical structure; generate, based on the first ultrasound images, a first model of the first anatomical structure; move, based on a first user input, a first surgical instrument; and move, based on the first model and the movement of the first surgical instrument, at least one of the ultrasound sensor or a camera.

2. The computer-assisted robotic system of Claim 1 , further comprising: a first repositionable structure arranged to hold the first surgical instrument, wherein moving the first surgical instrument comprises moving the first repositionable structure; a second repositionable structure arranged to hold the camera; a third repositionable structure arranged to hold the ultrasound sensor, wherein moving at least one of the ultrasound sensor or the camera comprises moving at least one of the second repositionable structure or the third repositionable structure; and an input device arranged to receive the first user input.

3. The computer-assisted robotic system of Claim 2, further comprising a fourth repositionable structure arranged to hold a second surgical instrument, wherein the controller is further configured to move the fourth repositionable structure based on user input.

4. The computer-assisted robotic system of Claim 2, wherein moving at least one of the ultrasound sensor or the camera is concurrent with moving the first surgical instrument.

5. The computer-assisted robotic system of Claim 1 , wherein the first model shows a portion of the first anatomical structure that is hidden from view in a video captured by the camera.

6. The computer-assisted robotic system of Claim 1 , wherein the controller is further configured to update the first model based on a movement of the first anatomical structure in a video captured by the camera.

7. The computer-assisted robotic system of Claim 1 , wherein the controller is further configured to receive a second user input indicating a first portion of the first anatomical structure, wherein the first ultrasound images show the first portion of the first anatomical structure.

8. The computer-assisted robotic system of Claim 7, wherein moving the ultrasound sensor is further based on the first portion of the first anatomical structure.

9. The computer-assisted robotic system of Claim 8, wherein the controller is further configured to: receive second ultrasound images after moving the ultrasound sensor; and update the model based on the second ultrasound images.

10. The computer-assisted robotic system of Claim 9, wherein the controller is further configured to: determine a second portion of the first anatomical structure after updating the model; and move the ultrasound sensor based on the second portion of the first anatomical structure.11 . The computer-assisted robotic system of Claim 7, wherein moving the camera is further based on the portion of the first anatomical structure.

12. The computer-assisted robotic system of Claim 1 , wherein moving the camera is based on the movement or position of the ultrasound sensor.

13. The computer-assisted robotic system of Claim 12, wherein moving the camera is further based on a movement or position of the first surgical instrument.

14. The computer-assisted robotic system of Claim 13, wherein moving the camera causes both the ultrasound sensor and the first surgical instrument to be visible in a video captured by the camera.

15. The computer-assisted robotic system of Claim 1 , wherein the controller is further configured to: determine, based on the first ultrasound images or a video captured by the camera, a characteristic of the first anatomical structure; and adjust an energy output of the first surgical instrument based on the characteristic of the first anatomical structure.

16. The computer-assisted robotic system of Claim 15, wherein determining the characteristic of the first anatomical structure is further based on an output of a haptic sensor of the ultrasound sensor.

17. The computer-assisted robotic system of Claim 15, wherein determining the characteristic comprises determining a tissue type for the first anatomical structure, wherein the characteristic is determined based on the tissue type.

18. The computer-assisted robotic system of Claim 17, wherein determining the characteristic comprises referencing a database using the tissue type.

19. The computer-assisted robotic system of Claim 1 , wherein the first ultrasound images show a second anatomical structure, and wherein the controller is further configured to generate a second model of the second anatomical structure based on the first ultrasound images.

20. The computer-assisted robotic system of Claim 19, wherein the controller is further configured to: monitor a position of the first surgical instrument in relation to the second anatomical structure based on the second model; and generate an alert based on determining that the first surgical instrument is within a threshold distance of the second anatomical structure.

21. The computer-assisted robotic system of Claim 20, wherein moving at least one of the ultrasound sensor or the camera directs at least one of the ultrasound sensor or the camera at a region where the first surgical instrument is nearing or within the threshold distance of the second anatomical structure.

22. The computer-assisted robotic system of Claim 1 , wherein the controller is further configured to move at least one of the ultrasound sensor or the camera based on a second user input.

23. The computer-assisted robotic system of Claim 1 , wherein moving the ultrasound sensor causes the ultrasound sensor to (i) move out of a field of view of the camera and (ii) take additional ultrasound images of a portion of the anatomical structure in the field of view.

24. The computer-assisted robotic system of Claim 1 , wherein moving at least one of the ultrasound sensor or the camera based on the model and the movement of the first surgical instrument comprises: determining a position of the first surgical instrument after moving the first surgical instrument; and moving the camera so that the first surgical instrument remains in view in a video captured by the camera.

25. The computer-assisted robotic system of Claim 24, wherein moving at least one of the ultrasound sensor or the camera based on the model and the movement of the first surgical instrument further comprises: determining a portion of the model based on the position of the first surgical instrument; and moving the ultrasound sensor to capture second ultrasound images corresponding to the portion of the model.

26. The computer-assisted robotic system of Claim 25, wherein the controller is further configured to: receive the second ultrasound images; and update the portion of the model based on the second ultrasound images.

27. The computer-assisted robotic system of Claim 1 , wherein moving at least one of the ultrasound sensor or camera changes a pose of at least one of the ultrasound sensor or camera.

28. The computer-assisted robotic system of Claim 1 , wherein moving the first surgical instrument dissects the first anatomical structure.

29. A method comprising: receiving, from an ultrasound sensor, first ultrasound images showing a first anatomical structure; generating, based on the first ultrasound images, a first model of the first anatomical structure; moving, based on a first user input, a first surgical instrument; and moving, based on the first model and the movement of the first surgical instrument, at least one of the ultrasound sensor or a camera.

30. The method of Claim 29, further comprising: holding, by a first repositionable structure, the first surgical instrument, wherein moving the first surgical instrument comprises moving the first repositionable structure; holding, by a second repositionable structure, the camera; holding, by a third repositionable structure, the ultrasound sensor, wherein moving at least one of the ultrasound sensor or the camera comprises moving at least one of the second repositionable structure or the third repositionable structure; and receiving, by an input device, the first user input.31 . The method of Claim 30, further comprising: holding, by a fourth repositionable structure, a second surgical instrument; and moving the fourth repositionable structure based on user input.

32. The method of Claim 30, wherein moving at least one of the ultrasound sensor or the camera is concurrent with moving the first surgical instrument.

33. The method of Claim 29, wherein the first model shows a portion of the first anatomical structure that is hidden from view in a video captured by the camera.

34. The method of Claim 29, further comprising updating the first model based on a movement of the first anatomical structure in a video captured by the camera.

35. The method of Claim 29, further comprising receiving a second user input indicating a first portion of the first anatomical structure, wherein the first ultrasound images show the first portion of the first anatomical structure.

36. The method of Claim 35, wherein moving the ultrasound sensor is further based on the first portion of the first anatomical structure.

37. The method of Claim 36, further comprising: receiving second ultrasound images after moving the ultrasound sensor; and updating the model based on the second ultrasound images.

38. The method of Claim 37, further comprising: determining a second portion of the first anatomical structure after updating the model; and moving the ultrasound sensor based on the second portion of the first anatomical structure.

39. The method of Claim 35, wherein moving the camera is further based on the portion of the first anatomical structure.

40. The method of Claim 29, wherein moving the camera is based on the movement or position of the ultrasound sensor.41 . The method of Claim 40, wherein moving the camera is further based on a movement or position of the first surgical instrument.

42. The method of Claim 41 , wherein moving the camera causes both the ultrasound sensor and the first surgical instrument to be visible in a video captured by the camera.

43. The method of Claim 29, further comprising: determining, based on the first ultrasound images or a video captured by the camera, a characteristic of the first anatomical structure; and adjusting an energy output of the first surgical instrument based on the characteristic of the first anatomical structure.

44. The method of Claim 43, wherein determining the characteristic of the first anatomical structure is further based on an output of a haptic sensor of the ultrasound sensor.

45. The method of Claim 43, wherein determining the characteristic comprises determining a tissue type for the first anatomical structure, wherein the characteristic is determined based on the tissue type.

46. The method of Claim 45, wherein determining the characteristic comprises referencing a database using the tissue type.

47. The method of Claim 29, wherein the first ultrasound images show a second anatomical structure, and wherein the method further comprises generating a second model of the second anatomical structure based on the first ultrasound images.

48. The method of Claim 47, further comprising: monitoring a position of the first surgical instrument in relation to the second anatomical structure based on the second model; and generating an alert based on determining that the first surgical instrument is within a threshold distance of the second anatomical structure.

49. The method of Claim 48, wherein moving at least one of the ultrasound sensor or the camera directs at least one of the ultrasound sensor or the camera at a region where the first surgical instrument is nearing or within the threshold distance of the second anatomical structure.

50. The method of Claim 29, further comprising moving at least one of the ultrasound sensor or the camera based on a second user input.51 . The method of Claim 29, wherein moving the ultrasound sensor causes the ultrasound sensor to (i) move out of a field of view of the camera and (ii) take additional ultrasound images of a portion of the first anatomical structure in the field of view.

52. The method of Claim 29, wherein moving at least one of the ultrasound sensor or the camera based on the model and the movement of the first surgical instrument comprises: determining a position of the first surgical instrument after moving the first surgical instrument; andmoving the camera so that the first surgical instrument remains in view in a video captured by the camera.

53. The method of Claim 52, wherein moving at least one of the ultrasound sensor or the camera based on the model and the movement of the first surgical instrument further comprises: determining a portion of the model based on the position of the first surgical instrument; and moving the ultrasound sensor to capture second ultrasound images of the portion of the model.

54. The method of Claim 53, further comprising: receiving the second ultrasound images; and updating the portion of the model based on the second ultrasound images.

55. The method of Claim 29, wherein moving at least one of the ultrasound sensor or camera changes a pose of at least one of the ultrasound sensor or camera.

56. A non-transitory machine-readable medium storing instructions for adjusting models of anatomical objects that, when executed by a controller, cause the controller to: perform the method of any of Claims 29 through 55.

Images