Method for guiding manual movement of medical system

Abstract

The invention relates to a method of guiding manual movement of a medical system. A computer-assisted medical system includes a user device wearable by an operator. The user device includes a display device configured to present an image superimposed in an environment of the manipulator assembly, and a sensor configured to detect one or more landmarks in the environment. The medical system includes a controller configured to execute instructions to perform operations. The operations include receiving position or orientation information of the one or more landmarks in the environment from the sensor, and directing manual movement of a portion of the manipulator assembly by causing the display device to present an image superimposed in the environment based on the received position or orientation information.

Description

[0001] This application is a divisional application of Chinese patent application 2019800045484 (PCT / US2019 / 016549) filed on February 4, 2019 and entitled “Method for guiding manual movement of a medical system”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Provisional Application Serial No. 62 / 642,299, filed on March 13, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present specification relates to methods of guiding manual movement of a medical system. Background Art

[0005] A system of robotic devices can be used to perform tasks at a work site. For example, a robotic system can include a robotic manipulator to manipulate an instrument for performing a task. The robotic manipulator can include two or more connecting rods coupled together by one or more joints. The joints can be active joints that are actively controlled. The joints can also be passive joints that follow the movement of the active joints when the active joints are actively controlled. Such active and passive joints can be revolute joints or prismatic joints. The configuration of the robotic manipulator can then be determined by the position and orientation of the joints, the structure of the robotic manipulator, and the coupling of the connecting rods.

[0006] Robotic systems include industrial and entertainment robotic systems. Robotic systems also include medical robotic systems used in procedures for diagnosis, non-surgical treatment, surgical treatment, etc. As a specific example, robotic systems include minimally invasive robotic telesurgery systems, in which surgeons can operate on patients from a bedside or remote location. Telesurgery generally refers to a surgical procedure performed using a surgical system, in which the surgeon uses some form of remote control (e.g., a servo mechanism) to manipulate the movement of surgical instruments rather than directly holding and moving the instruments by hand. Robotic medical systems that can be used for telesurgery or other telemedicine procedures may include a remotely controllable robotic manipulator. An operator can remotely control the movement of a remotely controllable robotic manipulator. The operator can also manually move the workpiece of the robotic medical system to a position or orientation within its environment. Summary of the invention

[0007] In one aspect, a computer-assisted medical system includes a user device that can be worn by an operator. The user device includes a display device configured to present an image superimposed (overlaid) in the environment of a manipulator assembly, and a sensor configured to detect one or more landmarks (landmark) in the environment. The medical system includes a controller configured to execute instructions to perform an operation. The operation includes receiving position or orientation information of the one or more landmarks in the environment from the sensor, and guiding the manual movement of a part of the manipulator assembly by causing the display device to present the image superimposed in the environment based on the received position or orientation information.

[0008] In another aspect, a method of providing a computer-assisted medical system including a manipulator assembly is featured. The method includes receiving position or orientation information of one or more landmarks in an environment from a sensor of a user device of the computer-assisted medical system. The method also includes guiding manual movement of a portion of the manipulator assembly by causing a display device of the user device to present an image superimposed on the environment based on the received position or orientation information.

[0009] The foregoing advantages may include, but are not limited to, those described below and elsewhere herein. The systems and methods described herein can improve the accuracy and precision of manual movement of objects in an environment. An operator wearing a user device can easily see the recommended position, orientation, and configuration of an object relative to the actual position, orientation, and configuration of the object because the image is directly superimposed on the environment.

[0010] The systems and methods described herein can also improve workflow efficiency and safety. A user device can present an image to guide an operator in a task to be performed without diverting the operator's attention from the environment. The operator can view and interact with the environment while viewing the guidance provided by the image presented by the user device. In an embodiment in which there are multiple user devices for multiple operators, the operators can easily collaborate with each other to prepare the environment and the objects in the environment for the procedures to be performed on the workpiece. For example, an operator can interact with a user device to collaboratively update the information presented on the user device so that the information is effectively disseminated to each operator. In addition, an operator can track the progress of tasks being performed by other operators, thereby making the workflow more efficient.

[0011] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1Ais a top view of a medical environment including a robotic device depicting an image of the environment presented to an operator via a user device.

[0013] Figure 1B yes Figure 1A A perspective view of an isolated portion of an environment.

[0014] Figure 2 is a perspective view of an example of a manipulator assembly.

[0015] Figure 3 is a front view of an operator wearing a user device.

[0016] Figure 4 It is the block diagram of the robot system.

[0017] Figure 5A is a flow chart of an example of a process for directing manual movement of a portion of a manipulator assembly.

[0018] Figure 5B is a flow diagram of another example of a process for directing manual movement of a portion of a manipulator assembly.

[0019] Figure 6 is a diagram depicting inputs and outputs of a controller that performs a process of directing manual movement of a portion of a manipulator assembly.

[0020] Fig. 7A is a perspective view of another example of a surgical environment with a robotic device and an operator.

[0021] Figure 7B is a perspective view of the surgical environment, including a depiction of images in the environment presented to the operator via a user device. DETAILED DESCRIPTION

[0022] Although some examples described herein relate to surgical procedures or tools or medical procedures and medical tools, the disclosed technology is applicable to medical and non-medical procedures and medical and non-medical tools. For example, the tools, systems and methods described herein can be used for non-medical purposes, including industrial uses, general robot uses, and sensing or manipulating non-tissue artifacts. Other exemplary applications include cosmetic improvements, imaging of human or animal anatomical structures, collecting data from human or animal anatomical structures, setting up or disassembling systems, and training medical personnel or non-medical personnel. Additional exemplary applications include procedures for tissues removed from human or animal anatomical structures (not returning to human or animal anatomical structures), and performing procedures for human or animal corpses. In addition, these technologies can also be used for medical treatment or diagnostic procedures that include or do not include surgical aspects.

[0023] Exemplary Systems

[0024] from Figure 1A Beginning with the illustrated medical example, a computer-assisted medical system 100 in an environment 10 includes a robotic manipulator assembly 102 having a robotic manipulator 103. The medical system 100 may be operated to perform a procedure on a workpiece, for example, a medical procedure on a patient 104. One or more operators (e.g., one or more of a surgeon, a surgical assistant, a nurse, a technician, and other medical practitioners) may operate the medical system 100 or portions of the medical system 100 to perform a surgical procedure.

[0025] The configuration of the manipulator assembly 102 can be established to prepare for performing a medical procedure on the patient 104. The manipulator assembly 102 or a portion of the manipulator assembly 102 can be manually moved (e.g., repositioned or reoriented) relative to the patient 104 so that the manipulator 103 of the manipulator assembly 102 can be used to perform a medical procedure on the patient 104. In some examples, the manipulator assembly 102 has a recommended configuration, for example, recommended by a controller of the medical system 100 based on data (such as presets, input data from sensors or users, etc.). The operator 106 manually moves the manipulator assembly 102 into the recommended configuration. The recommended configuration of the manipulator assembly 102 can be defined by the position and orientation of the individual components of the manipulator assembly 102. In some examples, the entirety of the manipulator assembly 102 can be manually translated on the floor surface 20 to reposition the manipulator 103 relative to the patient 104. In another example, the manipulator assembly 102 can be manually reoriented to reorient the manipulator 103 relative to the patient 104. In further examples, manipulator 103 or a portion of manipulator 103 is translated or reoriented in environment 10. As described herein, user device 108 of operator 106 can direct manual movement of manipulator assembly 102, manipulator 103, a portion of manipulator assembly 102, or a portion of manipulator 103 by presenting an image superimposed in environment 10.

[0026] To control the user device 108, the controller 122 ( Figure 4The controller 122 receives information related to one or more landmarks in the environment 10 to locate the user device 108 worn by the operator 106 (e.g., using simultaneous localization and mapping (SLAM) techniques). The landmarks may correspond to any physical object in the environment 10, such as the operating table 105, the patient 104, another manipulator in the environment 10, other equipment in the environment 10, features on the surface of a wall in the environment 10, features on the surface of a floor in the environment 10, or other unique features that can be used (as described herein) to locate the user device 108 in the environment 10. The controller 122 then controls the user device 108 to present an image to the operator 106 equipped with the user device 108. The image may be presented in a manner such that, to the operator 106 equipped with the user device 108, the image appears to be superimposed in the environment 10 based on the received information. The superimposed information is used to guide manual movement facilitated by the operator 106 to move the manipulator 103 or a portion of the manipulator 103 to a suggested position or orientation.

[0027] Figure 1A and Figure 1B An example of an image superimposed in environment 10 is depicted. Controller 122 causes display device 124 ( Figure 3 ) presents an image superimposed with a floor surface 20 in the environment 10.

[0028] Display device 124 presents an image indicating path 150 along floor surface 20. For example, the image includes a representation of path 150 that is superimposed with environment 10, floor surface 20, or other portions of environment 10 for operator 106 wearing user device 108. Path 150 indicates a recommended path along which manipulator assembly 102 should be manually moved to reach its recommended position. The image may be based on various forms of input data (e.g., including information about Figure 6 104. In some examples, when manipulator assembly 102 is in the recommended position, manipulator 103 can easily approach patient 104. In the case of using an image to guide the repositioning of manipulator assembly 102 as a whole, path 150 indicates a current position 152 of manipulator assembly 102 and a recommended position 154 of manipulator assembly 102. Path 150 guides the repositioning from the current position 152 of manipulator assembly 102 toward the recommended position 154 of manipulator assembly 102. Operator 106 manually moves manipulator assembly 102 as a whole from current position 152 to recommended position 154 along path 150.

[0029] In some embodiments, in addition to indicating the current position 152 and the recommended position 154, the path 150 also indicates a plurality of waypoints 156 along the floor surface 20. These recommended waypoints 156 for the manipulator assembly 102 are selected so that the manipulator assembly 102 does not contact other objects in the environment 10 when the manipulator assembly 102 is manually moved along the path 150. In one example, an obstacle (e.g., a chair 158) is located in the environment 10 between the manipulator assembly 102 and the patient 104. The waypoints 156 are selected so that the manipulator assembly 102 is manipulated away from the chair 158. In some embodiments, the image presented by the user device 108 includes an indicator 160 that the chair 158 is close to the path 150. The indicator 160 notifies the operator 106 of potential obstacles that the manipulator assembly 102 may collide with when moving along the path 150. Figure 1B Four waypoints 156 are depicted, but there may be fewer or more waypoints in other embodiments.

[0030] like Figure 1B As shown, the image includes a map 162 indicating the desired and undesired positions of the manipulator assembly 102. The map 162 can be superimposed on the floor surface 20 and the environment 10 so that the operator 106 equipped with the user device 108 can easily see where the desired position of the manipulator assembly 102 is located in the environment 10. Area 164 of the map 162 indicates the desired position, while area 166 of the map 162 indicates the undesired position. Area 164 is located near the patient 104 and the operating table 105 and is selected by the controller 122 to include positions that are determined to be easily accessible by the manipulator assembly 102. Area 166 corresponds to undesired positions of the manipulator assembly 102. In some embodiments, area 166 indicates positions that are near the patient 104 and the operating table 105 but are undesired, rather than indicating all undesired positions of the manipulator assembly 102, because, for example, the manipulator 103 will be too far from the patient 104 and the operating table 105 or will be near obstacles that the manipulator 103 may collide with during the procedure.

[0031] In some embodiments, to generate an image including regions 164, 166, controller 122 determines a desirability value for each potential position of manipulator assembly 102. Controller 122 then designates positions having desirability values ​​above a predetermined threshold as desired positions (e.g., corresponding to positions in region 164), and designates positions having desirability values ​​less than or equal to the predetermined threshold as undesirable positions (e.g., corresponding to positions in region 166). In some examples, map 162 is a heat map indicating the desirability value for each potential position, rather than showing two discrete regions. Map 162 may include a color-coded representation of the desirability values. If map 162 is a heat map, map 162 may indicate more than two regions, each region indicating a different predetermined range of desirability values. The desired range of values ​​may be determined based on various forms of input data (e.g., including relative to the desired range of values). Figure 6 160 , the desired location, the undesired location, the waypoint 156 , the indicator 160 , the map 162 , the area 164 , and the area 166 are generated using the input data 600 described above.

[0032] Figure 2 An example of a manipulator assembly 102 movable on a floor surface 20 is depicted ( Figure 1A and Figure 1B In addition to manipulator 103, manipulator assembly 102 also includes support structure 110 that supports manipulator 103 above floor surface 20. Support structure 110 can be translated and oriented relative to floor surface 20. For example, support structure 110 includes wheels 112 (e.g., casters) that enable operator 106 ( Figure 1A and Figure 1B ) can be relative to the patient 104 ( Figure 1A and Figure 1B 103 and supports the manipulator 103 at a height above the floor surface 20. Figure 1A and Figure 1B In the exemplary process of manually moving a portion of manipulator assembly 102 described, the portion of manipulator assembly 102 for which the movement is directed may correspond to support structure 110. Specifically, operator 106 may manually reposition the entirety of manipulator assembly 102 by manually moving support structure 110. In other examples, the portion may correspond to manipulator 103 or another portion of manipulator assembly 102.

[0033] The position or orientation of manipulator 103 may be manually adjusted by other mechanisms. In one example, the height of manipulator 103 above floor surface 20 is adjustable. Manipulator 103 may move vertically relative to support structure 110. In another example, manipulator 103 may be reoriented relative to support structure 110. Support structure 110 may include one or more passive joints about which manipulator 103 may rotate. Figure 2 In the example shown, the support structure 110 can include a passive setting arm 107 that connects the manipulator 103 to the column 109 of the support structure 110. The passive setting arm 107 includes a series of passive links and joints that can be manually repositioned and reoriented. The passive setting arm 107 can translate vertically relative to the column 109, thereby vertically repositioning the manipulator 103 relative to the column 109 of the support structure 110.

[0034] The base 111 of the manipulator 103 is connected to the support structure 110, for example, to the passive setting arm 107 of the support structure 110. The manipulator 103 includes one or more joints and one or more links that are operable to move an instrument holder 114 configured to hold an instrument 116. The one or more links of the manipulator 103 extend distally from the base 111 of the manipulator 103. For example, the manipulator 103 includes joints 118 and links 120, and one or more of the joints 118 are controllable by a controller 122 ( Figure 4 1 ). In some embodiments, one or more of the joints 118 are passive joints. By actuating the joints 118, the instrument holder 114 with the instrument 116 can be repositioned relative to the patient 104 or the environment 10 (shown in FIG. 1 ). In preparation for a procedure, the base 111 of the manipulator 103 can be repositioned to a desired position so that a desired range of motion of the links 120 and joints 118 of the manipulator 103 can be achieved. In this regard, in some examples described herein, the controller 122 ( Figure 4 1 ) can cause user device 108 (shown in FIG. 1 ) to present an image to guide manual movement of manipulator 103 , linkage 120 , joint 118 , base 111 , or another portion of manipulator 103 .

[0035] Figure 3 An example of a user device 108 worn by an operator 106 is shown. The user device 108 includes a display device 124 and a sensor 126. The display device 124 presents an image to the operator 106. For example, in Figure 3In the example shown, the user device 108 is a wearable head mounted display device that is wearable over the eyes of the operator 106. The display device 124 includes a see-through display that presents images. When the user device 108 is worn over the eyes of the operator 106, at least some of the images may be superimposed in the environment 10. At least some of the images may be transparent so that when superimposed on a portion of the environment 10, both the portion of the image and the portion of the environment 10 are visible to the operator 106. In some embodiments, at least some of the images superimposed on a portion of the environment 10 may be opaque so that the portion of the environment 10 is not visible to the operator 106, while the portion of the image is visible to the operator 106. The view of the display device 124 is framed in front of the eyes of the operator 106 so that the images presented by the display device 124 appear to be superimposed on the portion of the environment 10 that is seen by the operator 106. Thus, the operator 106 sees the environment 10 and any superimposed images presented on the display device 124 simultaneously.

[0036] Sensor 126 is configured to detect one or more landmarks in environment 10 to generate information indicating the position and / or orientation of the one or more landmarks in environment 10. Sensor 126 may generate one or more signals in response to detecting one or more landmarks in the environment, and the one or more signals may be processed and analyzed, for example, by controller 122 to generate information indicating the position and / or orientation of the one or more landmarks. In some examples, sensor 126 includes an image capture device that captures an image of environment 10, including any landmarks in environment 10. Using SLAM technology, the position and orientation information may be used to position user device 108 and thereby display device 124 relative to environment 10 and other features within environment 10, so that the image presented by display device 124 may be superimposed in a manner that is meaningful to operator 106 wearing user device 108. Specifically, from the perspective of operator 106, the image appears to be superimposed on a portion of environment 10, so that operator 106 can easily use the image as a guide for interacting with environment 10.

[0037] Figure 4 An example diagram of a medical system 100 that can be used to direct manual movement of a manipulator assembly 102 is shown. The medical system 100 includes a user device 108, a manipulator assembly 102, and a controller 122. The controller 122 includes one or more computer processors. In some embodiments, the controller 122 corresponds to a combination of processors of the manipulator assembly 102, the user device 108, and other systems of the medical system 100. The controller 122 directs the operation of the various systems of the medical system 100.

[0038] In some embodiments, the medical system 100 also includes a user control system 132 (also in Figure 1A 102). The user control system 132 includes a user input system and a user output system. The user input system of the user control system 132 can be operated by one of the operators 128 to control the movement of the manipulator assembly 102. In some cases, the user device receives a user command from the operator 106 to move the remote-operated manipulator 103. In some cases, the user input system is manually operable, so that the manual operation of the user input system causes the corresponding movement of the manipulator assembly 102. The user input system may include one or more of a foot pedal with one or both of a toe control and a heel control, one or more joysticks or other manually operable user input devices. In some cases, the user input system includes an image capture device or other sensor that can detect user motion. The user input system generates a control signal based on the detected user motion to control the movement of the manipulator assembly 102. The user output system of the user control system 132 provides an image of the work site to the operator operating the user control system 132.

[0039] Where the medical system 100 includes a user control system 132, the instructions for manual movement of the manipulator assembly 102 provided by the controller 122 may include instructions for operating a user input system of the user control system 132 to move the manipulator assembly 102. For example, if the user input system includes a user input device, the operator 106 manually operates the user input device to manually move the manipulator assembly 102. Alternatively, if the user input system includes an image capture device or another sensor that detects operator motion, the controller 122 operates the user device 108 to present an image that instructs the operator to move in a certain manner to cause the manipulator assembly 102, or a portion thereof, to move.

[0040] The medical system 100 includes a sensor system 137 that can detect features of the environment 10, including the sensors 126 of the user device 108. Data generated by the sensor system 137 can be used with SLAM techniques to locate the user device 108 within the environment 10. Using the data provided by the sensor system 137, the posture of the user device 108 relative to the environment 10 (e.g., the position and orientation of the user device 108) can be determined.

[0041] Sensor system 137 generates position or orientation information for one or more landmarks extracted from signals generated by sensor system 137. A landmark is a unique signal or set of signals generated by sensor system 137 that can be distinguished from other signals that may be generated by sensor system 137 when sensor system 137 detects different features within environment 10. For example, where sensor system 137 includes sensor 126 of user device 108 and sensor 126 is an image capture device, a landmark may correspond to a unique visual feature in environment 10 that does not generally change position or orientation relative to environment 10, such as console 105, other equipment in environment 10, a corner of a room, or another unique visual feature. When sensor 126 observes such a visual feature, information received by controller 122 from sensor 126 indicates the orientation or position of the landmark relative to sensor 126, thereby enabling controller 122 to determine the position or orientation of sensor 126, and therefore user device 108, relative to environment 10 using SLAM techniques.

[0042] The sensor system 137 may include one or more sensors in addition to the sensors 126 of the user device 108. For example, the medical system 100 includes one or more of a kinematic sensor 138, a patient sensor 140, an instrument sensor 142, an obstacle sensor 144, or an image capture device 146. The output data generated by the sensors of the sensor system 137 can be used by the controller 122 to locate the user device 108 in the environment 10. In some embodiments, the output data can be used to provide other information to the operator 106 through the user device 108. In some examples described herein, the output data is used to determine a recommended configuration of the manipulator assembly 102, and therefore used to generate an image for guiding manual movement of the manipulator assembly 102 toward the recommended configuration. In other examples, the output data is used to provide information related to the state of a subsystem of the medical system 100, an operator in the environment 10, a patient 104, or another object in the environment 10.

[0043] Kinematic sensor 138 may be a kinematic sensor of manipulator assembly 102. For example, kinematic sensor 138 may detect the pose of joint 118 or link 120 of manipulator 103. In some cases, kinematic sensor 138 is configured to detect the pose of instrument holder 114 so that the position and orientation of instrument 116 can be determined. Kinematic sensor 138 may be an accelerometer, a gyroscope, an encoder, a torque sensor, a force sensor, or other type of sensor that may detect the motion of one of joint 118 or link 120 of manipulator 103. In some examples, manipulator assembly 102 includes a single kinematic sensor, however in other embodiments, manipulator assembly 102 includes two or more kinematic sensors.

[0044] The patient sensor 140 is configured to detect a characteristic of the patient 104 (shown in FIG. 1 ). For example, the patient sensor 140 may be a patient motion sensor that detects when the patient 104 moves, for example, relative to the environment 10 or relative to the operating table 105 (shown in FIG. 1 ) on which the patient 104 is located. In some cases, the patient sensor 140 includes an image capture device or an optical sensor (e.g., mounted in the environment 10 or mounted to the manipulator 103 ) that detects whether the patient 104 is located on the operating table 105. In some cases, the patient sensor 140 includes an accelerometer or other motion sensor attached to the patient 104 that detects the movement of the patient 104. In other cases, the patient sensor 140 detects another physical characteristic of the patient 104 (such as the weight or size of the patient 104, the blood pressure of the patient 104, the heart rate of the patient 104), rather than detecting the movement or position of the patient 104.

[0045] The instrument sensor 142 is configured to detect characteristics of the instrument 116 mounted to the instrument holder 114. For example, the instrument sensor 142 is a sensor on the instrument holder 114 that detects whether the instrument has been mounted to the instrument holder 114. In some embodiments, the instrument sensor 142 detects the type of instrument 116 mounted to the instrument holder 114. For example, the instrument sensor 142 can be a sensor of the manipulator assembly 102 that detects the identity indicated in the EEPROM of the instrument 116. In some embodiments, the instrument sensor 142 detects movement of the instrument 116. For example, the instrument sensor 142 includes an accelerometer, a gyroscope, a force sensor, a torque sensor, or other types of sensors mounted to the instrument 116 or the instrument holder 114 to detect movement of the instrument 116.

[0046] Obstacle sensor 144 is configured to detect obstacles in environment 10. In some cases, obstacle sensor 144 is an optical or acoustic proximity sensor that detects when an obstacle is located near manipulator assembly 102. For example, obstacle sensor 144 is mounted to manipulator assembly 102 and is capable of detecting when an obstacle is within a predetermined distance of manipulator assembly 102. In some embodiments, obstacle sensor 144 is an image capture device mounted in environment 10 and configured to detect when an obstacle moves within the vicinity of manipulator assembly 102. Obstacles may include objects, such as other devices of medical system 100, other devices within environment 10, people in environment 10, or other objects in environment 10. Obstacle sensor 144 may include contact sensors, proximity sensors, optical transit time sensors, and other sensors suitable for detecting contact with an obstacle or the distance of an obstacle.

[0047] Image capture device 146 may correspond to one of the image capture devices described with respect to one of the other sensors described herein (e.g., one of sensor 136 of user device 130, kinematic sensor 138, patient sensor 140, instrument sensor 142, obstacle sensor 144, or another sensor). Image capture device 146 is located within environment 10 to capture images of environment 10. In some cases, one of user devices 130 includes image capture device 146. In other cases, image capture device 146 is fixed to a portion of environment 10 (e.g., a wall, ceiling, or other fixture in environment 10). Image capture device 146 may be an optical sensor (e.g., a camera or an acoustic sensor).

[0048] Example Process

[0049] Figure 5A Depicted is a diagram for guiding the manipulator assembly 102 ( Figure 1A and Figure 1B 102). As described herein, the portion of manipulator assembly 102 may correspond to support structure 110 of manipulator assembly 102. Process 500A may be performed by controller 122 as described herein. At operation 502A, controller 122 receives position or orientation information of one or more landmarks in the environment. For example, controller 122 receives this information from one or more sensors, such as those described with respect to sensor system 137 or with respect to sensor 126 of user device 108.

[0050] At operation 504A, the controller 122 directs manual movement of portions of the manipulator assembly 102. The controller 122 causes the display device 124 (relative to the display device 124 of the user device 108 worn by the operator 106) to Figure 1A , Figure 1B and Figure 3 502A) presents an image superimposed on environment 10 based on the position or orientation information received at operation 502A. The image may be presented to guide operator 106 to manually move a portion of manipulator assembly 102 toward a recommended configuration, which is determined based on the position or orientation information received at operation 502A. The image may also be presented so that the image is positioned or oriented relative to environment 10 based on the position or orientation information received at operation 502A. In some examples, the image indicates a recommended movement path for a portion of manipulator assembly 102, such as relative to Figure 1A and Figure 1B As described herein, other examples are possible.

[0051] Figure 5BAnother process 500B is depicted for updating an image presented to an operator 106. In some cases, operation 502A of process 500A is performed so that an image has been presented to a user device 108 ( Figure 1A After the user device 108 is presented to the operator 106, process 500B is performed. In process 500B, at operation 502B, the controller 122 receives updated position or orientation information of one or more landmarks in the environment (e.g., one or more landmarks in the environment whose information was received at operation 502A). The updated position or orientation information may indicate movement of the operator 106, and therefore the user device 108, relative to the one or more landmarks.

[0052] At operation 504B, the controller 122 updates the presented image based on the updated position or orientation information. For example, if the user device 108 has moved relative to one or more landmarks, the presented image may be updated so that the position or orientation of the image relative to the one or more landmarks is maintained even though the user device 108 has moved. In this regard, if a portion of the image is superimposed on an object in the environment, the presented image is updated when the user device moves relative to the object so that the portion of the image remains superimposed on the object.

[0053] Alternative or additional embodiments

[0054] A number of implementations have been described. However, it should be understood that various modifications may be made to the processes, systems, and mechanisms described herein.

[0055] Some embodiments described herein are described with respect to medical examples. In other embodiments, the medical system 100 is a surgical system for performing surgical procedures on a patient 104. The techniques disclosed herein are also applicable to non-surgical uses. For example, they can be used with general or industrial robotic operations and improve these operations (such as those for manipulating workpieces). These techniques can also be used with medical robotic operations for diagnosis and non-surgical treatment and improve these operations.

[0056] The specific examples presented in this disclosure may be applicable to teleoperated robotic systems and remotely controllable arms. The techniques disclosed herein are also applicable to robotic systems that are partially or entirely moved directly and manually by an operator. For example, these techniques may be applied to a robotic system designed to help stabilize an instrument held by manipulator 103 when the instrument is manipulated by the operator's hand. As another example, any controllable manipulator discussed herein may be configured to allow direct manipulation and accept operator instructions through inputs applied directly to the linkages or joints of the manipulator. The techniques are also applicable to robotic systems that are partially or entirely moved automatically.

[0057] The operator can manually move the object (e.g., part or all of the manipulator assembly 102) by applying force directly on the object (e.g., using a hand, foot, or other body part to apply force directly on the object). The operator can manually push the object to reposition or reorient the object. In other examples, the operator can manually move the object by interacting with a user input device that causes the object to move. For example, if the object to be manually moved is the manipulator assembly 102, the wheel 112 of the support structure 110 can be, for example, a powered wheel that can be controlled by the user input device. In this regard, the manual movement of the manipulator assembly 102 directed by the user device 108 can correspond to a manual movement generated in response to manual manipulation of a user input device separate from the manipulator assembly 102.

[0058] Although the setup arm 107 is described as passive in the above examples, in other embodiments, the setup arm 107 is an actively controllable setup arm. For example, the setup arm can be moved in response to operation of the user input system described herein. In some embodiments, the setup arm can be backdriven by operation of the power joint of the manipulator 103. For example, the distal portion of the manipulator 103 can be fixed, and the power joint can be operated to backdrive the setup arm, thereby repositioning or reorienting the installation arm.

[0059] Support structure 110 is depicted as including wheels 112. In some embodiments, support structure 110 is mounted in environment 10 in a manner that enables support structure 110 to be easily moved in environment 10, rather than including wheels. For example, support structure 110 can be mounted directly to operating table 105, directly to a wall of environment 10, or directly to a ceiling of environment 10.

[0060] In some implementations, the user device 108 is worn by the operator 106 above the eyes of the operator 106. The user device 108 may be a head mounted user device, and the display device 124 of the user device 108 may be a see-through display, such as with respect to Figure 2 As described. In other embodiments, display device 124 may be an opaque display device that is substantially opaque to light, rather than a see-through display device. User device 108 is a virtual reality device. To allow operator 106 to view environment 10, sensor 126 of user device 108 may capture an image of environment 10, and display device 124 may present the image of environment 10. In some cases, controller 122 may generate an image of environment 10 with one or more indicators superimposed on the image to guide manual movement of manipulator assembly 102.

[0061] Although the user device 108 is described as a head-mounted device with a perspective display, the user device may vary in other embodiments. In other embodiments, the user device 108 may be carried by the operator 106 in other ways. The user device may be a mobile computing device, such as a tablet computer, a smart watch, or a smart phone. For example, the user device 108 may be a smart watch worn on the wrist of the operator 106. The mobile computing device is a handheld computing device that the operator 106 can easily carry by hand around the environment 10. Using an augmented reality process, an image presented on a display device of the mobile computing device may be superimposed on an image of the environment 10. In this regard, at operations 504A, 504B, rather than superimposing an image directly above the environment 10, an image for guiding the manual movement of the manipulator assembly 102 is superimposed on the image of the environment 10. The image of the environment 10 may be captured by an image capture device of the mobile computing device or an image capture device of another part of the medical system 100.

[0062] Although relative to Figure 1A and Figure 1B A single user device 108 is described, but in other embodiments, multiple operators with multiple user devices may be present in environment 10. Environment 10 may include any number of operators. Figure 4 , each operator has a corresponding user device 130-1, 130-2, ..., 130-n (collectively referred to as user device 130). In some embodiments, each user device 130 includes a corresponding display device. Multiple user devices 130 may each include a corresponding display device 134-1, 134-2, ..., 134-n (collectively referred to as display device 134). In some embodiments, user device 130 also includes a corresponding sensor 136-1, 136-2, ..., 136-n (collectively referred to as sensor 136). Each sensor 136 may be similar to the corresponding Figure 3 The controller 122 may include sensors 126 described herein. For example, one or more sensors 136 may include an image capture device. In other embodiments, the sensor 136 may include any of an accelerometer, a motion sensor, a gyroscope, or another type of sensor. In embodiments where there are multiple user devices 130, the sensor system 137 may include any of the sensors 136 of the multiple user devices 130. In this regard, the controller 122 may receive data generated by the sensor 136 and use the data for the processes described herein. For example, if one user device 130 includes a corresponding sensor 136, the output from the sensor 136 may be used to provide information that can be used to locate another user device 130, determine a recommended configuration for the manipulator 103, or perform other operations described herein.

[0063] exist Figure 1A In the example shown, the environment 10 includes four operators 128-1 (i.e., operator 106), 128-2, 128-3, 128-4 (collectively referred to as operators 128). One or more of the operators 128 may carry a user device. For example, in some embodiments, only a portion of the operators 128 in the environment 10 carry a user device. In the example of FIG. 1 , operators 128-1, 128-2, 128-3 each carry a corresponding user device 130-1, 130-2, 130-3. Operator 128-4 operates a user control system 132. One or more of the user devices 130-1, 130-2, 130-3 may include a corresponding sensor. In some cases, each of the user devices 130-1, 130-2, 130-3 includes a corresponding sensor, while in other cases, one or two of the user devices 130-1, 130-2, 130-3 include a corresponding sensor. Similarly, in some cases, each of the user devices 130-1, 130-2, 130-3 includes a corresponding display device, while in other cases, one or both of the user devices 130-1, 130-2, 130-3 include a corresponding display device.

[0064] In some embodiments, the image presented on one of the user devices 130 may correspond to an image captured by one of the other user devices 130. For example, in an embodiment where the medical system 100 includes a user device 130-1 carried by the operator 128-1 and a second user device 130-2 carried by the operator 128-2, in operations 504A, 504B ( Figure 5A and Figure 5B ), the image presented on user device 130-1 may correspond to the image captured by sensor 136-2 (in Figure 4 If user device 130-2 is a head mounted device, sensor 136-2 of user device 130-2 may capture an image of a portion of environment 10 as seen by operator 128-2. For example, sensor 136-2 may capture an image of device table 168 in front of operator 128-2. In this regard, display device 134-1 of user device 130-1 (shown schematically in FIG. 1 ) may be configured to capture an image of a portion of environment 10 as seen by operator 128-2. Figure 4 1) may present an image of operator 128-2 preparing an apparatus on apparatus station 168. In some implementations, operator 128-1 may operate user device 130-1 to cause display device 134-1 to present an image captured by user device 130-2 of operator 128-2. This allows operator 128-1 to easily determine the status of operator 128-2 or the status of a task performed by operator 128-2.

[0065] In some cases, display device 134-1 may present an indicator that indicates the status of operator 128-2 or the status of a task being performed by operator 128-2. For example, the indicator may indicate that operator 128-2 is moving a particular workpiece of the apparatus, is preparing an instrument for use, or is performing some other operation to prepare for a procedure to be performed on a workpiece. Although user device 130-1 is described as presenting an indicator displayed by sensor 136-2 (in Figure 4 ), but in other embodiments, the display device 134-2 of the user device 130-2 (in Figure 4 1) may also or alternatively present images captured by sensor 136-1 of user device 130-1. Additionally, in some implementations, display device 134-2 may present an indicator indicating the status of operator 128-1 or the status of a task performed by operator 128-1.

[0066] In one example, as shown in FIG1 , operator 128-1 is moving manipulator assembly 102, and operator 128-2 is preparing an instrument for use during a procedure on patient 104. In such a case, if both user device 130-1 and user device 130-2 include corresponding sensors and display devices. Display device 134-1 of user device 130-1 can be operated to present an indicator indicating the status of the device preparation being performed by operator 128-2, and display device 134-2 of user device 130-2 can be operated to present an indicator indicating the status of the manual movement operation being performed by operator 128-1.

[0067] In some embodiments, not all user devices 130 include corresponding sensors or corresponding display devices. For example, user device 130-1 includes sensor 136-1 and display device 134-1, and user device 130-2 includes sensor 136-2 but does not include a display device. In such cases, the image presented on display device 134-1 is generated based on data generated by sensor 136-1 and data generated by sensor 136-2. In other examples, user device 130-1 includes sensor 136-1 and display device 134-1, and user device 130-2 includes display device 134-2 but does not include a sensor. In such cases, the image presented by display device 134-2 can be generated based on data collected by sensors of sensor system 137 rather than sensors on user device 130-2.

[0068] In some embodiments, if both user device 130-1 and user device 130-2 include display devices, the image presented by user device 130-1 and the image of user device 130-2 may be similar to each other. Both the image presented by user device 130-1 and the image of user device 130-2 may be superimposed on environment 10. In some cases, because the positioning and orientation of user devices 130-1, 130-2 are different from each other, the perspective of the image presented by user device 130-1 may be different from the perspective of the image presented by user device 130-2. The image presented by user device 130-1 and the image presented by user device 130-2 may be positioned and oriented relative to a single global reference system. Both user device 130-1 and user device 130-2 may present an image indicating a path 150 along which manipulator assembly 102 should move. In both the image presented by user device 130 - 1 and the image presented by user device 130 - 2 , the representation of path 150 is superimposed on the same portion of environment 10 , such as the same portion of floor surface 20 .

[0069] In some embodiments, user device 130-1 is operated to direct manual movement of manipulator assembly 102, and another user device is operated to direct manual movement of another portion of medical system 100. For example, user device 130-3 carried by operator 128-3 may direct auxiliary system 170 ( Figure 1A In some embodiments, assistance system 170 is a system that provides visual or image processing capabilities to present images of internal anatomical structures of patient 104, for example. User device 130-3 can present images that guide manual movement of assistance system 170 in environment 10. Other examples of portions thereof that can guide manual movement are described herein. The images presented on user device 130-3 can be similar to the images described with respect to user device 108 for guiding manual movement of manipulator assembly 102.

[0070] In an embodiment where both user device 130-1 and user device 130-2 include sensors, sensors 136-1, 136-2 of user devices 130-1, 130-2 may detect landmarks in environment 10 for locating user devices 130-1, 130-2. For example, a first landmark may be extracted from data generated by sensor 136-1 as user device 130-1 moves around within environment 10. A second landmark may be extracted from data generated by sensor 136-2 as user device 130-2 moves around within environment 10. The extracted first landmark may correspond to a physical feature in environment 10 that is different from a physical feature corresponding to the second landmark. For example, if sensors 136-1, 136-2 are image capture devices, the first landmark may correspond to a portion of an image captured by sensor 136-1 representing user control system 132, and the second landmark may correspond to a portion of an image captured by sensor 136-2 representing console 105. In other embodiments, the landmarks extracted from the data of sensors 136-1, 136-2 may correspond to the same physical feature in environment 10. Although a first landmark and a second landmark are described, in other embodiments, each of sensors 136-1, 136-2 of user devices 130-1, 130-2 may collect data from which multiple landmarks are extracted.

[0071] In operations 502A and 502B, position or orientation information is received so that the controller 122 can generate or update an image to provide guidance to the operator when manually moving the manipulator 103. The reference frame of the position or orientation information can vary in different embodiments. In embodiments where the controller 122 determines the posture of the user device 108, the reference frame of the position or orientation information of the user device 108 can be the same as the reference frame of the position or orientation information of one or more landmarks. In other embodiments, the reference frame of the position or orientation information of the landmarks is different from the reference frame of the position or orientation information of the user device 108.

[0072] In some implementations, other information besides position or orientation information may also be used to generate or update an image. For example, controller 122 receives other contextual information related to objects in environment 10 besides one or more landmarks. Figure 6An example of contextual information that can be used by the controller 122 to generate or update an image presented on the user device 108 is shown. The contextual information can include device information, operator information, obstacle information, and patient information. For example, the controller 122 uses input data 600 to generate output 602, such as a signal for causing the user device 108 to present an image to guide manual movement of the manipulator assembly 102 or a portion thereof. The input data 600 includes data loaded into a memory associated with the controller 122, user-specified data, data generated by the sensor system 137, and the like. For example, the input data 600 includes procedure data 600a, device data 600b, posture data 600c, operator data 600d, obstacle data 600e, workpiece data 600f, and port data 600g.

[0073] The data 600a, 600b, 600c, 600e, 600f, 600g represent some examples of data that may be used by the controller 122 to control the medical system 100 and to generate images for presentation on the user device 108. The controller 122 may use other types and content of data to control the medical system 100 or to control images for presentation on the user device 108 as appropriate. Furthermore, although described as input data 600, in some implementations, some data of the input data 600 is generated from other data of the input data 600. Furthermore, although the operations 502A, 502B are described with respect to generating images for guiding manual movement of a portion of the manipulator assembly 102, the input data 600 may be used to generate images to provide other information to the operator 106, as described herein.

[0074] The procedure data 600a includes data indicating a particular procedure to be performed. For example, in the medical example, the procedure data 600a includes data indicating a medical procedure to be performed on a patient. In some embodiments, the operator selects the type of procedure before the controller 122 directs the manual movement of the manipulator assembly 102. The controller 122 then generates an image presented on the user device 108 based on the type of medical procedure to be performed by the manipulator 103 of the manipulator assembly 102.

[0075] The procedure data 600a may refer to specific requirements of the workspace, such as an area around the workspace that the instrument 116 should be able to access. In one example, the type of procedure may indicate a specific workspace requirement. The specific type of procedure may require a predetermined range of the workspace. In the example of a medical or surgical procedure, the workspace may correspond to an area around the patient 104 that the instrument 116 should be able to access during the surgical procedure due to the specific medical or surgical procedure to be performed on the patient 104.

[0076] In another example, when the operator selects the extent of the workspace before performing a manual move, procedure data 600a is generated. The operator may select the extent of the workspace in any number of ways. For example, the operator may input data by highlighting or tracing an area representing the desired extent of the workspace on a representation of the workspace presented on a user interface device. The operator may indicate the boundaries of the desired extent of the workspace.

[0077] As another example, the desired range of the workspace can be indicated by existing operator-directed movement of the manipulator 103 (e.g., the distal portion of the manipulator 103). The operator can move the manipulator 103 (with or without holding the instrument) to indicate the desired workspace, or by moving a surrogate of the instrument 116 to indicate the desired workspace. Exemplary surrogate instruments 116 include devices that represent common instruments that can be used during a procedure, devices that replicate the proximal portion of the instrument 116 but not the entire shaft and end effector, devices that project visual indications of positions associated with the distal end of an instrument that can be used during a procedure, and the like. An image capture device or other sensor can detect movement of the manipulator 103. If the user interface device corresponds to the user device 108, the sensor 126 of the user device 108 can detect movement of the manipulator 103. In some embodiments, when the operator moves the manipulator 103, the controller 122 can determine the range of motion limits based on signals generated by sensors associated with the joint 118 of the manipulator 103.

[0078] Information about the desired setup configuration of the manipulator assembly 102 or the medical system 100 can be derived, at least in part, from such a demonstration. For example, the desired range of motion of the joints of the manipulator 103 or the instrument 116, and therefore the desired position or orientation of the manipulator 103 can be derived, at least in part, from such a demonstration. For example, a posture sensor of the sensor system 137 can provide data indicating the configuration of the manipulator 103, the configuration of the instrument 116, or other system characteristics detectable during a manual demonstration of the desired workspace. The sensor system 137 can therefore provide information about the desired range of motion of the joints of the manipulator 103 or the instrument 116, or information about the desired motion envelope. The controller 122 or other computing system can then process the sensor information to determine the extent of the workspace demonstrated by the operator.

[0079] In some embodiments, the procedure data 600a may include a plan indicating a desired or recommended configuration of the manipulator assembly 102 or the medical system 100. For example, the plan may indicate the position or orientation of an object in the environment 10. The plan may alternatively or additionally indicate the desired or recommended position of one or more objects (such as the manipulator 103) in the environment 10 that the controller 122 will instruct it to move manually. Alternatively or additionally, the plan may indicate the known position of one or more objects (such as the device used for the procedure) in the environment 10. The one or more objects may include devices, landmarks, obstacles, or other objects in the environment 10. For example, the plan corresponds to a map generated before the controller 122 guides the manual movement of the manipulator 103. The operator can generate the map by interacting with the graphical representation of the environment 10. In the medical example, the operator 106 can indicate the position of an operating table, a patient, a medical device, or another object that will be in the environment 10 during the medical procedure. The operator can use a user input device (e.g., a mouse, a keyboard, or other appropriate user input device) to indicate the position. The graphical representation can be presented by the display device 124 of the user device 108. In other embodiments, the computing device used by the operator to generate the plan is separate from the user device 108 worn during the procedure.

[0080] In some embodiments, the procedure data 600a includes a stored plan for setting up a previous procedure (e.g., a procedure similar to the present procedure). For example, during a previous procedure, a plan for setting up a medical system or manipulator assembly is stored for use in setting up another medical system or manipulator assembly for another procedure.

[0081] In one example, the plan is established before any operator is in the environment 10. When the operator 106 begins the preparation process for the medical procedure, such as placing the medical system 100 or the manipulator assembly 102 in a setup configuration, the user device 108 can present a representation of the plan. The operator 106 can interact with the user device 108 to input changes to the plan, such as updating the position of an object in the environment 10. In examples where the user device 108 can detect gestures, as described herein, the operator 106 can perform gestures to update the plan.

[0082] The device data 600b includes data indicating the specifications of the device to be used during the procedure. The device data 600b may include data specifying a range of motion for each joint 118 of the manipulator assembly 102. The range of motion may be a structural or mechanical limitation. In some examples, an initial motion envelope of the instrument 116 is estimated based on the initial position (and / or orientation) of the joint 118 and the range of motion of the joint 118 of the manipulator 103. The controller 122 determines a recommended position or orientation of the joint 118 within the range of motion that will enable the instrument 116 to achieve the recommended motion envelope of the instrument 116. The controller 122 may drive one or more powered joints in the joint 118 so that the joint 118 moves toward the recommended position (or orientation) and the instrument 116 is able to move through the recommended range of motion.

[0083] The device data 600b may also include information related to the type of instrument 116 mounted to the manipulator 103. This information may be obtained, for example, from the instrument sensor 142 (relative to Figure 4 Description) is generated. For example, the type of instrument 116 can affect the scope of the workspace and the amount of torque required to perform the operation. The type of instrument 116 can be manually input by an operator. In some examples, the instrument 116 can include a detectable label indicating the type of instrument 116.

[0084] The posture data 600c includes data indicating a posture of a portion of the medical system 100. The posture data 600c includes the position and orientation information received at operations 502A and 502B. In some embodiments, the posture data 600c also includes position or orientation information of the manipulator assembly 102 or a portion of the manipulator assembly 102, such as the joint 118, the link 120, the instrument holder 114, or other components of the manipulator assembly 102.

[0085] The posture data 600c may indicate a target position or orientation or an actual position or orientation of a portion of the medical system. In one example, the posture data 600c may include information indicating an actual configuration of the manipulator 103. In such cases, the posture data 600c may include an initial posture of each of the joints and / or links of the manipulator 103. Alternatively, the posture data 600c may include information indicating an actual configuration of the manipulator 103 during a guided manual movement of the manipulator 103. The postures of the joints and links or the configuration of the manipulator 103 may be detected by sensors of the sensor system 137.

[0086] In another example, gesture data 600c may include information indicating a target configuration for manipulator 103. The target configuration for manipulator 103 may be determined based on other data and information described herein, or in some cases, may be selected by an operator for manipulator 103. In some cases, the target configuration for manipulator 103 may be indicated on a predetermined plan described relative to procedure data 600a.

[0087] Operator data 600d includes data related to the operator. In the medical example, operator data 600d includes data related to the medical team (e.g., operator 128) performing the procedure. For example, operator data 600d includes information related to capabilities, preferences for device layout, experience level, skill level, and other operator-specific attributes. In some embodiments, operator data 600d includes information related to the specific role of the operator on the medical team. For example, operator data 600d may include information indicating which operators are equipped to perform tasks requiring aseptic processing (e.g., processing medical devices to be placed in patient 104). In some examples, an operator profile is created for each operator before the procedure. Alternatively or additionally, a team profile is created for a specific team.

[0088] Obstacle data 600e includes data indicating the posture (e.g., one or more parameters of position or orientation) of patients and obstacles in environment 10 relative to manipulator assembly 102. In some examples, obstacle data 600e may include a map of environment 10 input by an operator. The map may include the locations of potential obstacles within environment 10, such as other artifacts of the device (e.g., medical system 100). The map may be similar to the map generated as part of procedure data 600a.

[0089] Obstacle data 600e alternatively or additionally includes data from obstacle sensors of sensor system 137. Obstacle sensors 144 may generate signals indicative of the location, orientation, or posture of obstacles within environment 10 either prior to a procedure or as manipulator 103 moves around within environment 10 during a procedure.

[0090] Although chair 158 is described as an example of a potential obstacle that manipulator assembly 102 may collide with, other examples are possible. In some embodiments, potential obstacles include operating table 105, auxiliary system 170, another manipulator assembly, a human operator, user control system 132, or other fixtures in environment 10.

[0091] In some medical environments, the workpiece data 600f includes patient data and includes data indicating patient-specific characteristics. Such patient data may include data indicating patient body habits and patient geometry. In some examples, the operator inputs data indicating patient habits and patient geometry. In some cases, the imaging device may generate an image that can be analyzed by the controller 122 (or by a computing system before or during a procedure) to determine the patient habits and patient geometry. The imaging device may include a portion of the instrument 116. In some examples, the workpiece data 600f may also include data indicating the posture of the patient relative to the manipulator 103 and / or the posture of the operating table 105 relative to the manipulator 103. The workpiece data 600f may include preoperative images, such as x-ray images, x-ray computed tomography images, magnetic resonance imaging scans, etc. In some cases, the workpiece data 600f includes intraoperative images or surface scans. A portion of the workpiece data 600f may be generated by the patient sensor 140.

[0092] The port data 600g includes data indicating the characteristics of the access port. In medical embodiments, the access port corresponds to a device inserted through the body wall of the patient 104, through which a portion of the medical system 100 can be inserted to access the anatomical structure of the patient 104. The access port provides, for example, a catheter through which the instrument 116 is inserted to access the anatomical structure of the patient 104. In one example, the port data 600g can indicate a recommended position or orientation of the access port. The recommended position or orientation of the access port can be indicated on a predetermined plan described relative to the procedure data 600a. In some embodiments, the recommended position or orientation of the access port is based on the posture of the manipulator 103 when the cannula coupled to the manipulator 103 is docked, when the operator indicates that the base 111 is ready to be repositioned, when the instrument 116 is installed, etc. In some medical embodiments, a component (such as an instrument 116) is inserted through the patient's access port, and the controller 122 can determine the position and orientation of the access port based on signals from sensors on the manipulator 103.

[0093] In another example, port data 600g may indicate an actual position or orientation of the access port. The actual position or orientation of the access port may be detected by a sensor of sensor system 137. For example, sensor 126 of user device 108 may detect the position or orientation of the access port. Controller 122 may guide manual movement of manipulator 103 based on the recommended or actual position or orientation of the access port so that an instrument held by instrument holder 114 of manipulator 103 may be easily inserted into the access port.

[0094] As described with respect to procedure data 600a, operator 106 may specify procedure data 600a or portions of procedure data 600a before controller 122 directs manual movement of manipulator assembly 102. Also, as described with respect to procedure data 600a and obstacle data 600e, operator 106 may specify locations of objects in environment 10. In some embodiments, operator 106 may specify other types of input data 600. For example, in some embodiments, operator 106 may specify device data 600b by specifying the type of device to be used during the procedure. Operator 106 may also specify specifications for the device. In another example, operator 106 may specify characteristics of an operator who will participate in the procedure and, therefore, specify operator data 600d.

[0095] For portions of input data 600 specified by operator 106, operator 106 may interact with a computing device (e.g., a personal computer, a tablet computing device, a mobile phone, a user device 108, or other computing device) to specify the data. In some embodiments, operator 106 provides user input by performing gestures detectable by sensor 126 of user device 108. For example, sensor 126 is positioned to detect movement of operator 106, and controller 122 may direct manual movement of manipulator 103 in response to the detected movement. Controller 122 may be configured to detect predefined operator gestures that cause controller 122 to perform corresponding operations.

[0096] In some examples, the image presented on the user device 108 includes one or more indicators, e.g., indicating a potential location of an object, indicating guidance for moving the manipulator 103, indicating a warning or alarm, or indicating other information relevant to the operator 106. In response to the gesture performed by the operator 106, the indicator in the image can be updated, e.g., repositioned in the image, cleared from the image, adjusted to provide additional detail, or adjusted in some other manner. The controller 122 can update the image in response to the gesture approaching or pointing to the indicator.

[0097] In one example, when the operator 106 performs a wide swipe motion from one side of the viewing frame of the user device 108 to the other, the controller 122 causes the user device 108 to stop presenting the image.

[0098] In another example, user device 108 may present an image of waypoints 156 indicating a recommended path 150 for movement of manipulator assembly 102. Operator 106 may perform a gesture pointing to one of waypoints 156. For example, operator 106 may extend a hand toward the waypoint and move the hand in a direction for manually moving the waypoint. Specifically, in response to the gesture, user device 108 updates the image so that the waypoint is repositioned in the direction indicated by the gesture of operator 106. In some examples, such as relative to Figure 1A and Figure 1B As described, the image may include an indicator of the target position of the manipulator assembly 102. Rather than causing the waypoint to be repositioned, the operator 106 performs a gesture to cause the target position of the manipulator assembly 102 to be repositioned at a new target position.

[0099] In another example, the operator 106 may perform gestures to rearrange the graphical representations of the objects in the environment 10, as described with respect to the procedure data 600a. If the graphical representation of the environment 10 is presented by a perspective display on the user device 108, the perspective display may present an image including a representation of the objects that will be located in the environment 10 during the medical procedure. The gestures performed by the operator 106 may indicate the location of the object during the procedure. The user device 108 updates the image to indicate the location of the object. In some cases, the location of the object is indicated in the image, and the gestures performed by the operator 106 may cause the user device 108 to update the image to indicate the new location of the object. In some examples, the object corresponds to a device to be used during the procedure. The image includes graphical representations of various artifacts of the device, such as a Mayo stent, an accessory cart, a medical imaging system, or other devices that may be used during the procedure. For the purpose of preoperative planning, the operator 106 performs gestures to create a plan indicating the location of various artifacts of the device, and as described herein, the controller 122 updates the image based on these gestures. The controller 122 may further guide the manual movement of the manipulator 103 based on the plan.

[0100] In some embodiments, the user device 108 presents an image to guide manual movement of the manipulator 103, and the operator 106 performs gestures to indicate the location of the workpiece of the device to be placed in the environment 10. Based on the indicated location, the controller 122 can update the image so that the guidance provided by the image takes into account the workpiece of the device to be placed in the environment 10.

[0101] Although the operator 106 may provide user input that the controller 122 uses to control the images presented by the user device 108, in some embodiments described herein, the sensors of the sensor system 137 generate portions of the input data 600 used by the controller 122 to present the images. For example, as described with respect to the posture data 600c, the sensors of the sensor system 137 may detect the position and orientation of a portion of the medical system 100. In some embodiments, the controller 122 may receive information from both the sensor 126 of the user device 108 and another sensor of the sensor system 137. The information provided by the other sensors may also indicate the position or orientation of one or more landmarks detected by the sensor 126 of the user device 108. Alternatively or additionally, the information may be kinematic information of the manipulator 103 or kinematic information of another portion of the medical system 100. The controller 122 may cause the user device 108 to present an image superimposed on the environment based on the received kinematic information.

[0102] In one example, the other sensor is a sensor of another user device worn by another operator in the environment 10. For example, the other sensor is a sensor of a user device 130 worn by an operator 128. In another example, the other sensor is a sensor mounted to the environment 10, such as an image capture device mounted to a wall surface, a floor surface, or some other part of the environment 10. The image capture device can be fixed in the environment 10 or can be mounted to a movable object within the environment 10. In some cases, the image capture device is part of an apparatus of the medical system 100. In another example, the other sensor is an integrated part of the medical system 100. For example, the other sensor can correspond to a joint sensor of the manipulator assembly 102.

[0103] In embodiments where there are multiple operators 128 in the environment 10, different operators 128 may collaborate to provide or update input data 600. For example, if a first operator 128-1 and a second operator 128-2 both carry user devices 130-1, 130-2, the first user device 130-1 and the second user device 130-2 may both present an image (described relative to the procedure data 600a) indicating a plan for different objects in the environment 10. For example, the image may include a graphical representation of an object superimposed in the environment 10, so that the operator 128-1, 128-2 may easily manually move the real-world object to be consistent with the graphical representation of the object. The first operator 128-1 and the second operator 128-2 may reposition and reorient objects in the environment based on a predefined plan, for example by individually moving different objects in the environment 10 to their planned locations.

[0104] As described herein, the first user device 130-1 and the second user device 130-2 can provide guidance for the manual movement of the object. In another embodiment, the first operator 128-1 and the second operator 128-2 can collaboratively update the plan. For example, the first operator 128-1 and the second operator 128-2 can operate the user input device as described herein (e.g., operate a touch screen or perform a gesture) to select a new desired position or orientation of an object in the environment 10. The graphical representation of the object is repositioned or reoriented accordingly to reflect the newly selected position or orientation in both the image presented by the first user device 130-1 and the image presented by the second user device 130-2. If the second operator 128-2 selects a new desired position or orientation for the device stage 168, the graphical representation of the device stage 168 in both the image presented by the first user device 130-1 and the image presented by the second user device 130-2 is updated.

[0105] In operations 504A and 504B, an image is presented via display device 124 of user device 108. As described herein, an image may be presented to guide operator 106 to reposition or reorient manipulator assembly 102 or a portion of manipulator assembly 102. Although the image is described as representing path 150 along which manipulator assembly 102 should move, in other examples, the image represents other types of indicators for guiding manual movement of manipulator assembly 102. For example, the image may include an indicator indicating a direction in which operator 106 should manually translate manipulator assembly 102.

[0106] In an example where the operator 106 is guided to reposition the manipulator assembly 102 or a portion of the manipulator assembly 102, the user device 108 may present an image including an alignment indicator. For example, if the controller 122 guides the reorientation of the manipulator 103, the alignment indicator is a line, mark, or other indicator indicating the target orientation of a link or joint of the manipulator 103.

[0107] The image presented by display device 124 is described as being superimposed on environment 10. If display device 124 is part of a head-mounted user device worn over the eyes of operator 106, the image presented on display device 124 is positioned between environment 10 and display device 124 so that the image is superimposed on environment 10 as seen by operator 106. In some embodiments, user device 108 presents an image with a portion of the environment superimposed. For example, if user device 108 is a tablet computer, smartphone, or other device that may have an opaque display, the opaque display may present a captured image of environment 10 with an additional image or indicator that is separate from the captured image of environment 10. The additional image may include content similar to that described with respect to the image presented by user device 108. In some examples, the additional image is presented so that it appears to be superimposed on the captured image of environment 10.

[0108] In some embodiments, a portion of the captured image of environment 10 appears to overlay the additional rendered image. Figure 1A , the operator console 105 is positioned between the operator 106 and the auxiliary system 170. If the user device 108 presents an indicator indicating the location of the auxiliary system 170, the user device 108 may present the indicator so that the operator console 105 appears superimposed on the indicator. The indicator may be partially superimposed by the operator console 105.

[0109] The image may indicate information other than instructions for manual movement of the manipulator assembly 102 or a portion thereof. Fig. 7A and Figure 7B Another example of an image that may be presented by a head mounted user device is depicted. Fig. 7A , an operator 700 wearing a user device 702 (eg, similar to the user device 108) is located in the environment 30, near an operating table 704 supporting a patient 706. Manipulators 708a and 708b are mounted on the operating table 704.

[0110] Figure 7B An example of an image superimposed on the environment 30 presented by the user device 702 to the operator 700 is depicted. The image includes status indicators 710a, 710b of the manipulators 708a, 708b. Figure 7B In the example shown, the status indicators 710a, 710b indicate whether the manipulator 708a, 708b requires manual repositioning. In other embodiments, the status indicators 710a, 710b indicate whether the manipulator 708a, 708b requires manual reorientation or requires both manual repositioning and manual reorientation. Figure 7B In the example shown, the image includes a direction indicator 716 indicating the direction manipulator 708b should be moved.

[0111] In some examples, manual movement of manipulator 708b precedes any portion of a medical procedure performed on patient 706. For example, manual movement may be directed before an access port is inserted into patient 706, and before instruments 712a, 712b are inserted into patient 706. In other examples, manual movement of manipulator 708b is directed after a portion of a medical procedure has been performed. For example, manual movement of manipulator 708b is directed after instrument 712b has been inserted into patient 706. Instrument 712b is retracted from patient 706, and user device 702 presents an image for directing manual movement of manipulator 708b to a new position or orientation.

[0112] After a medical procedure has been initiated but before the medical procedure is completed, such repositioning or reorientation may be required to complete the medical procedure. For example, a first phase of a medical procedure may require the use of instruments 712a, 712b on a first region of a target anatomy, and a second phase of the medical procedure may require the use of instruments 712a, 712b on a second region of the target anatomy. When the first phase of the medical procedure is completed, manual repositioning of manipulator 708b and manual repositioning of manipulator 708a may be initiated. Manipulators 708a, 708b may be repositioned or reoriented so that they may access the second region of the target anatomy.

[0113] In some embodiments, the indicators 710a, 710b indicate other conditions of the manipulators 708a, 708b, rather than indicating whether the manipulators 708a, 708b need to be moved. For example, the indicators 710a, 710b may indicate whether the manipulators 708a, 708b are operable, or whether the manipulators 708a, 708b have been properly mounted to the operating table 704. In some examples, the indicators 710a, 710b indicate the condition or status of the instruments 712a, 712b mounted to the manipulators 708a, 708b. For example, the indicators 710a, 710b may indicate the type of instruments 712a, 712b currently mounted to the manipulators 708a, 708b, the duration that the instruments 712a, 712b have been used, or whether the instruments 712a, 712b should be replaced with new instruments.

[0114] Figure 7BThe exemplary image of also includes an anatomical structure indicator 714 indicating an area corresponding to the target anatomical structure to be approached by the instrument 712a, 712b. The anatomical structure indicator 712 may include a representation of the internal anatomical structure of the patient 706 so that the operator 700 can easily move the manipulator 708a, 708b relative to the internal anatomical structure, which the operator 700 would not normally be able to see without a separate medical imaging system. The anatomical structure indicator 712 may be generated based on an image captured by the medical imaging system, particularly in an embodiment where the user device 702 is worn above the eyes of the operator 700. For example, the anatomical structure indicator 712 may be a graphical representation of the internal anatomical structure of the patient 706 present in the image captured by the medical imaging system.

[0115] In other embodiments, the image may include anatomical indicators that indicate areas corresponding to anatomical features that the instruments 712a, 712b will not approach. These anatomical indicators may indicate areas that the operator 700 should avoid when moving the manipulators 708a, 708b.

[0116] In further embodiments, the image may include a patient indicator 718 indicating the status of the patient 706. For example, the patient indicator 718 may indicate the heart rate and blood pressure of the patient 706. Figure 6 The described input data 600 (eg, the workpiece data 600 f ) is used to generate the patient indicator 718 .

[0117] Figure 1A and Figure 1B The example of depicts an example of instructions for manual movement of the manipulator assembly 102. For example, in Figure 1A and Figure 1B In the embodiment, the image indicates a path 150 for guiding the manual movement of the manipulator assembly 102. In other embodiments, the image indicates a plurality of optional paths along which the manipulator assembly 102 should be moved. Each optional path guides the manipulator assembly 102 from its current position to a desired or recommended position.

[0118] exist Figure 1A and Figure 1BIn the example shown, path 150 is shown as including a plurality of waypoints 156. In some embodiments, path 150 corresponds to an area through which manipulator assembly 102 should move to move manipulator assembly 102 from its current position to its desired or recommended position. An area can be larger than the footprint of manipulator assembly 102 or the footprint of support structure 110 of manipulator assembly 102. An area defines a boundary within which support structure 110 of manipulator assembly 102 should remain to avoid obstacles in environment 10. Manipulator assembly 102 can move through the area along any of a plurality of paths.

[0119] In some embodiments, in addition to or in lieu of presenting a representation of the desired or recommended path of movement of manipulator assembly 102, user device 108 presents an indicator indicating the desired or recommended direction of movement of manipulator assembly 102. As operator 106 moves manipulator assembly 102 in the desired or recommended direction, the desired or recommended direction of movement indicated by the indicator is updated so that operator 106 can move manipulator assembly 102 to the desired or recommended position. Additionally or alternatively, the indicator can indicate the desired or recommended magnitude of movement of manipulator assembly 102. For example, the indicator can include text specifying the distance manipulator assembly 102 should move. In examples where the indicator is a directional arrow, the size of the directional arrow can indicate the amount of movement manipulator assembly 102 should move in the indicated direction.

[0120] Although indicator 160 is Figure 1B Although obstacles are described as being presented near path 150 in the example embodiment, in some embodiments, the indicator presented by user device 108 indicates the location of the obstacle, but not necessarily near the obstacle. For example, as operator 106 moves manipulator assembly 102 along path 150, user device 108 may present an annotation indicating the direction of the obstacle. For example, the annotation includes an indicator in the form of a directional arrow that indicates that the obstacle is in a particular direction.

[0121] In relation to Figure 1A and Figure 1BIn the described examples, the manipulator assembly 102 and its manipulator 103 are described as being manually moved according to instructions presented by the user device 108. In some embodiments, the user device 108 may present images for guiding the manual movement of other portions of the medical system 100 or other portions of the manipulator assembly 102. For example, in some embodiments, the user device 108 presents images for guiding the manual movement of the base 111 of the manipulator 103, the support structure 110 of the manipulator assembly 102, or the instrument holder 114 of the manipulator 103. In other embodiments, manual movement of the intermediate portion of the manipulator 103 is guided. For example, the user device 108 may present images for guiding the manual movement of a specific joint in the joint 118 or a specific link in the link 120. In other embodiments, manual movement of the distal portion of the manipulator assembly 102 or the proximal portion of the manipulator assembly 102 is guided. For example, the user device 108 can present an image for guiding manual movement of the instrument holder 114 to guide manual movement of the distal portion of the manipulator assembly 102, and the user device 108 can present an image for guiding manual movement of the column 109 or support structure 110 to guide manual movement of the proximal portion of the manipulator assembly 102.

[0122] In some examples, the presented image includes a representation of a configuration of manipulator 103 recommended by controller 122 based on input data 600. The representation of the recommended configuration of manipulator 103 may indicate a recommended orientation, a recommended position, or both a recommended orientation and a recommended position of one or more of joints 118, one or more of links 120, base 111, or a combination thereof. The representation of the recommended configuration may correspond to an image superimposed on environment 10. In some embodiments, the representation is an opaque or partially transparent graphical representation of manipulator 103 in the recommended configuration.

[0123] In some examples, the image provides an animation of a recommended movement of the manipulator 103 to a recommended orientation, a recommended position, or a recommended posture. The animation may indicate a sequence of positions or orientations between the current position and orientation of the manipulator 103 and the recommended position and orientation of the manipulator 103. In some cases, the animation may provide a representation of the operator's movement pattern to manually move the manipulator 103 to the recommended position or orientation. For example, the animation may represent the operator's hands and the suggested hand movements to manually move the manipulator 103. The animation may include one or more transparent graphical representations of the manipulator 103 or the operator superimposed on the environment.

[0124] In some embodiments, the user device 108 presents an image for guiding manual movement of another portion of the medical system 100, rather than guiding manual movement of a portion of the manipulator assembly 102. In some embodiments, an image is presented to guide manual movement of an instrument 116 mounted to the manipulator 103 or a cannula through which the instrument 116 is inserted. In embodiments where an access port is inserted through the body wall of the patient 104, the image presented by the user device 108 may indicate a desired or undesired location for the access port. The image may indicate a recommended port location where the operator 106 should manually place the access port. The recommended port location corresponds to a recommendation determined by the controller 122 based on the input data 600. In other embodiments, the image may indicate a no-entry zone indicating a location where the access port should not be placed.

[0125] The user device 108 can guide the manual movement of other parts of the medical system 100. In some embodiments, the user device 108 presents an image for guiding the manual movement of another manipulator assembly according to the embodiments described herein with respect to guiding the manual movement of the manipulator assembly 102 and its parts. The other manipulator assembly is different from the manipulator assembly 102. In some embodiments, the other manipulator assembly is separated from the manipulator assembly 102. The manipulator assembly 102 and the other manipulator assemblies are independent assemblies each including its own separate support structure. In other embodiments, the manipulator assembly 102 and the other manipulator assemblies share a common support structure that can be moved around in the environment 10.

[0126] In other embodiments, additionally or alternatively, the user device 108 presents images for guiding manual movement of auxiliary systems of the medical system 100. The auxiliary systems include resources such as illumination sources, energy sources, insufflation sources, instrument storage components, central processing units, or combinations thereof. The auxiliary systems may be independent systems that can be moved around within the environment 10. For example, the auxiliary systems may correspond to the medical imaging system, the user control system 132, the device table 168, the insufflation system ( Figure 1A 10), console 105, or other systems in environment 10. Other auxiliary systems whose manual movement can be guided include trays, tables, imaging systems, user consoles, surgical instruments, medical instruments, diagnostic instruments, or other devices. Features of the embodiments described herein for guiding manual movement of manipulator assembly 102 can also be features of embodiments in which the images presented are used to guide manual movement of auxiliary systems. By presenting images for guiding manual movement of auxiliary systems, user device 108 enables an operator to easily set the device relative to manipulator assembly 102 or relative to other devices in environment 10.

[0127] In some embodiments, the user device 108 presents an image for guiding the movement of the patient 104 in the environment 10. For example, the operating table 105 on which the patient 104 is supported can be moved to allow the patient 104 to move around within the environment 10. In other embodiments, the user device 108 presents an image for guiding the movement of the operator in the environment 10. For example, in an embodiment in which the environment 10 includes a first operator 128-1 and a second operator 128-2, the user device 130-1 worn by the first operator 128-1 can guide the movement of the first operator 128-1, the second operator 128-2, or both. Similarly, the user device 130-2 worn by the second operator 128-2 can guide the movement of the first operator 128-1, the second operator 128-2, or both. The desired positions of the first operator 128-1 and the second operator 128-2 can be determined based on the operator data 600d and the different types of position and orientation information described herein (e.g., for obstacles, manipulator assemblies 102, and other objects in the environment 10). In an embodiment where the operator data 600d includes information related to whether the operator is equipped to operate sterile devices, this information can be used to determine the tasks that the operator should perform in the environment and the location where the tasks are performed. For example, the information can indicate that the first operator 128-1 is not equipped to operate sterile devices, and therefore guides the first operator 128-1 to manually move the manipulator assembly 102. In contrast, the operator 128-2 is equipped to operate sterile devices and therefore guides the operator to prepare the instrument for insertion into the patient 104. The operator 128-2 can also be guided to be at the bedside of the patient 104 to perform tasks close to the patient 104, such as pointing out the position for access ports or for inserting instruments 116 ( Figure 2 shown).

[0128] As described herein, the techniques, systems, and methods described herein can be used to reposition or reorient a portion of a manipulator assembly or a portion of a medical system prior to performing a medical procedure (e.g., preoperative repositioning or reorientation) or during a medical procedure (e.g., intraoperative repositioning or reorientation). For example, during a medical procedure, these techniques, systems, and methods can be used to reposition or reorient a portion of a manipulator assembly or a portion of a medical system in response to an emergency condition (e.g., a patient condition or device condition that requires the medical system or manipulator assembly to be placed in a new configuration). In other embodiments, these techniques, systems, and methods can be used after a medical procedure to facilitate disassembly of a manipulator assembly or a medical system.

[0129] The systems (e.g., medical system 100) and robotic components of medical systems (e.g., manipulator 103) described herein can be computer-assisted systems. For example, each of these systems and robotic components can be at least partially controlled using one or more computer program products, such as one or more computer programs tangibly embodied in one or more information carriers (such as one or more non-transitory machine-readable media) for execution by one or more data processing devices (e.g., programmable processors, computers, multiple computers and / or programmable logic components) or for controlling the operation of the one or more data processing devices. A single computer program product or multiple different computer program products can be used for different systems and robotic components to control the system and robotic components. For example, the computer program for one or more computer program products may be written in any form of programming language (including compiled or interpreted languages), and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment.

[0130] Operations associated with controlling the medical system described herein can be performed by a controller (e.g., controller 122). The controller may include one or more programmable processors that execute one or more computer programs to perform the functions described herein. The one or more programmable processors may include multiple processors that are part of different subsystems in the medical system 100, for example, a processor of the manipulator assembly 102, a processor of the user control system 132, a processor of the auxiliary system 170 and / or another processor of the medical system 100. In other embodiments, the one or more programmable processors may include one or more processors remote from the environment 10, for example, a processor of a remote server. Control of all or part of the medical system described herein may be implemented using dedicated logic circuits (e.g., FPGA (field programmable gate array) and / or ASIC (application-specific integrated circuit)).

[0131] For example, processors suitable for executing computer programs include general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only storage area or a random access storage area or both. The elements of a computer include one or more processors for executing instructions and one or more storage area devices for storing instructions and data. Typically, a computer will also include one or more machine-readable storage media (such as a large-capacity PCB for storing data, for example, a magnetic disk, a magneto-optical disk, or an optical disk) or be operably coupled to receive data from the machine-readable storage medium or transfer data to the machine-readable storage medium or both. Machine-readable storage media suitable for embodying computer program instructions and data include all forms of non-volatile storage areas, such as semiconductor storage area devices (e.g., EPROM, EEPROM, and flash storage area devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0132] The elements of the different embodiments described herein can be combined to form other embodiments not specifically described above. Elements can be excluded from the structure described herein without adversely affecting its operation. In addition, various independent elements can be combined into one or more individual elements to perform the functions described herein.

Claims

1. A computer-assisted medical system, comprising: A user device wearable by an operator, the user device comprising: a display device configured to present an image superimposed on an environment of the manipulator assembly, and a sensor configured to detect one or more landmarks in the environment; and A controller configured to execute instructions to perform operations, the operations comprising: receiving position or orientation information of the one or more landmarks in the environment from the sensor, and Manual movement of a portion of the manipulator assembly is directed by causing the display device to present the image superimposed on the environment based on the received position or orientation information.

2. The medical system of claim 1, wherein the operations further comprise: Manual movement of a portion of the medical system relative to the manipulator assembly is directed by causing the display device to present the image superimposed on the environment based on the received position or orientation information, the portion of the medical system being distinct from the manipulator assembly.

3. The medical system of claim 2, wherein the image comprises: a first indicator indicating a target position of the portion of the manipulator assembly, and A second indicator indicates a target location of the portion of the medical system.

4. The medical system of claim 2, wherein: The portion of the medical system is selected from the group consisting of: an operating table, a manipulator system supporting the manipulator assembly, a separate manipulator system physically separate and movable relative to the manipulator system supporting the manipulator assembly, an operator console for remotely operating the manipulator assembly, a cannula, a human operator, and an auxiliary system including resources for the medical system, and The resource is selected from the group consisting of: an illumination source, an energy source, an insufflation source, an instrument storage component, and a central processing unit.

5. The medical system of claim 1, wherein: The image is a first image, and The operations also include guiding the patient in manual movement of the manipulator assembly by causing the display device to present a second image superimposed on the environment based on the received position or orientation information.

6. The medical system of claim 1, wherein: The operations also include receiving user input information indicating a recommended configuration for the medical system, and Directing the manual movement includes directing the manual movement further based on the user input information. 7 . The medical system of claim 1 , wherein directing the manual movement of the portion of the manipulator assembly comprises directing a user to translate or reorient the portion of the manipulator assembly.

8. The medical system of claim 1, wherein: The image includes an alignment indicator, and Directing the manual movement of the portion of the manipulator assembly includes directing a user to reorient the portion of the manipulator assembly relative to the alignment indicator.

9. The medical system of claim 1 or any one of claims 2 to 8, wherein the image superimposed on the environment indicates one or more waypoints of the manual movement of the portion of the manipulator assembly.

10. The medical system of claim 1 or any one of claims 2 to 8, wherein the image superimposed on the environment includes a map having a first area indicating an undesirable position of the portion of the manipulator assembly.

Images