Associated processes and related systems for manipulators

Abstract

This application, titled "Association Process and Related System for a Manipulator," discloses a computer-aided medical system including a manipulator, a user input system, a user output system including a display device, and a controller configured to execute instructions to perform operations. The operations include, in a pairing mode and in response to a first set of signals generated by the user input system, moving a virtual selector displayed on the display device relative to an image displayed on the display device. The operations further include, in the pairing mode, associating a first manipulator with a portion of the user input system based on movement of the virtual selector relative to a represented position of a first instrument, and in a follow mode, controlling the movement of the first manipulator according to a second set of signals generated by the user input system in response to user actions on a portion of the user input system.

Description

[0001] This application is a divisional application of Chinese patent application 2018800228283 entitled “Association process and related system for manipulator”, filed on July 18, 2018.

[0002] Cross-reference to related applications

[0003] This application is a non-provisional application of U.S. Provisional Patent Application No. 62 / 537,795, filed July 27, 2017, and U.S. Provisional Patent Application No. 62 / 551,702, filed August 29, 2017, and claims priority to those applications. The entire contents of each of the above applications are incorporated herein by reference. Technical Field

[0004] This specification relates to associated processes and systems for manipulators (e.g., for remote operation of manipulators). Background Technology

[0005] Robotic manipulators can be operated to control the movement of instruments within a workspace. For example, such manipulators can be used to perform both non-medical and medical procedures. As a specific example, a remotely operated surgical manipulator can be used to perform minimally invasive surgical procedures. An operator can control the manipulator using a user control system, which may be wirelessly connected or connected to the remotely operated manipulator via a wired connection. The user control system may include multiple user input devices, such that each remotely operated manipulator can be controlled by a different user input device within the user control system. Therefore, the operator can independently control each remotely operated manipulator using the user input devices. Summary of the Invention

[0006] In one aspect, a computer-assisted medical system includes a remotely operated manipulator, a user input system, a user output system including a display device, and a controller configured to execute instructions to perform operations. The operations include, in a pairing mode and in response to a first set of signals generated by the user input system, moving a virtual selector displayed on the display device relative to an image displayed on the display device. The image represents the position of a first instrument supported by a first manipulator of a plurality of manipulators and the position of a second instrument supported by a second manipulator of a plurality of manipulators. The operations further include, in the pairing mode, associating the first manipulator with a portion of the user input system based on the movement of the virtual selector relative to the represented position of the first instrument, and in a follow mode, controlling the movement of the first manipulator according to a second set of signals generated by the user input system in response to user actions on a portion of the user input system.

[0007] In another aspect, a method for operating a computer-assisted medical system comprising a plurality of remotely operated manipulators is provided. The method includes causing a display device to present: an image representing the position of a first instrument supported by a first manipulator of the plurality of manipulators and the position of a second instrument supported by a second manipulator of the plurality of manipulators; and a virtual selector movable relative to the image in response to a first set of signals generated by a user input system. The method further includes: in a pairing mode, associating the first manipulator with a portion of the user input system based on movement of the virtual selector relative to the represented position of the first instrument; and in a follow mode, controlling the movement of the first instrument according to a second set of signals generated by the user input system in response to user actions on a portion of the user input system.

[0008] In another aspect, it is characterized by one or more non-transitory computer-readable media. The one or more non-transitory computer-readable media store instructions executable by a processing device and, upon such execution, cause the processing device to perform operations. The operations include causing a display device to present: an image representing the position of a first instrument supported by a first manipulator of a plurality of remotely operated manipulators and the position of a second instrument supported by a second manipulator of a plurality of manipulators; and a virtual selector that moves relative to the image in response to a first set of signals generated by a user input system. The operations further include, in a pairing mode, associating the first manipulator with a portion of the user input system based on the movement of the virtual selector relative to the represented position of the first instrument, and in a follow mode, controlling the movement of the first manipulator according to a second set of signals generated by the user input system in response to user actions on a portion of the user input system.

[0009] The aforementioned advantages may include, but are not limited to, those described below and elsewhere herein. For example, the association between the user-operable portion of the user input system and the remotely operated manipulator can be formed in an intuitive manner for the operator. The operator can control a virtual selector overlaid on or otherwise superimposed on an image of the workspace to initiate the association between the user-operable portion and a specific remotely operated manipulator without having to interact with lists and information presented on a display screen, which do not provide the operator with a sense of the configuration or relative orientation of the remotely operated manipulator. Specifically, by controlling the position of the virtual selector relative to an image of the workspace representing the instrument supported by the manipulator, the operator can intuitively manipulate the user input system to associate the user-operable portion with the remotely operated manipulator.

[0010] Human-detectable feedback can be provided during pairing mode, allowing the operator to be informed of the status and process of devices (e.g., user-operable parts of a user input system and manipulators to be associated). For example, the controller can generate feedback indicating the association status of the user-operable part, the manipulator, or both. Based on this feedback, the operator can initiate the association process for devices that are not yet associated. Alternatively, the controller can generate feedback indicating a suggested association before finalizing the association between the user-operable part and the manipulator. This allows the operator to adjust the suggested association, providing greater control during the association process. In some implementations, human-detectable feedback can be continued or newly provided after association has been established, indicating the part of the user input system associated with a specific manipulator, and vice versa. Furthermore, the controller can unassociate a user input device or manipulator in response to user input or system events.

[0011] While some of the examples described herein generally relate to medical procedures and medical devices, the disclosed techniques are also applicable to non-medical procedures and non-medical devices. For example, the instruments, systems, and methods described herein can be used for non-medical purposes, including industrial applications, general robotic applications, manipulation of non-tissue artifacts, and / or cosmetic improvements. Other non-surgical applications include for the removal of tissue from human or animal anatomy (without returning the human or animal anatomy) or for use with human or animal cadavers.

[0012] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other potential features, aspects, and advantages will become apparent from the description, drawings, and claims. Attached Figure Description

[0013] Figure 1A It is a top view of the system, including the controller and console.

[0014] Figure 1B yes Figure 1A The front view of the console.

[0015] Figure 1C It can be replaced Figure 1A and Figure 1B The diagram shows a top view of a console-based user input and output system.

[0016] Figure 2 It is a frontal perspective view of the operating table and the patient.

[0017] Figure 3A and Figure 3B It is a view of the display device during the process of associating the user-operable part of the system with the controller.

[0018] Figure 4 This is a block diagram of the system used to perform the manipulator association process.

[0019] Figure 5 The relationship between the manipulator and the user-operable part of the user input system is shown.

[0020] Figure 6 This is a flowchart illustrating the process of operating the user input system to control the manipulator.

[0021] Figure 7A This is a flowchart illustrating the process of associating a user-operable part with a manipulator.

[0022] Figures 7B-7E It is a view of the display device during the process of associating the user-operable part of the system with the controller.

[0023] Figure 8A This is a flowchart illustrating the process of optimizing the association formed between the manipulator and the user-operable part of the user input system.

[0024] Figure 8B The left side shows the user input system and the display device of the display instrument, while the right side shows a top view of the manipulator supporting the instrument.

[0025] Figure 9 This is a flowchart illustrating the process of reorienting the user-operable portion of a user input system.

[0026] Figure 10 This is a schematic diagram of a computer system.

[0027] Figure 11 This is a front view of the control system.

[0028] Figure 12 It is a top view of the system, including the manipulator and motion detection system.

[0029] Figure 13 This is a front view of the console, including the eye-tracking system.

[0030] The same reference numerals and names in the various figures denote the same elements. Detailed Implementation

[0031] Example System

[0032] refer to Figure 1ASystem 100 in environment 10 includes manipulator system 101, which includes remotely operated manipulators 102a, 102b, 102c, and 102d (collectively referred to as manipulator 102 or remotely operated manipulator 102). Manipulators 102 are referred to as "remotely operated manipulators" because they can be remotely operated by operator 104 through a physically separate user input system 106. In some implementations, manipulator 102 can also be directly controlled through manual interaction with itself. Therefore, "remotely operated manipulator" as used in this application includes manipulators that can be controlled only remotely, as well as manipulators that can be controlled both remotely and through direct manual control. Manipulator 102 includes a movable portion that can support an instrument (not shown) (e.g., surgical and medical devices). The movable portion corresponds, for example, to the distal ends 112a, 112b, 112c, and 112d of manipulator 102.

[0033] Further reference Figure 1B and Figure 1C , Figure 1B yes Figure 1A The front view of the console. Figure 1C It can be replaced Figure 1A and Figure 1B The diagram shows a top view of a console-based user input and output system. Operator 104 can use (e.g., including a display device) a user input and output system to remotely operate the manipulator 102 and monitor the equipment supported by the manipulator 102. In some examples, such as Figure 1 and... Figure 1B As shown, the standalone console 103b includes a user input system 106b and a user output system. In the example shown, console 103 includes a user input system portion (such as input devices 108a, 108b) and a user output system including a stereoscopic display device 107b.

[0034] In some examples, such as Figure 1C As shown, a non-console-based user input and output system 113 can be used. Figure 1CIn the example shown, user input and output system 113 includes user input system 106c, which includes handheld user input devices 108c and 108d. The handheld user input devices 108c and 108d are part of user input system 106c, and their movement is not physically constrained to a base or console by links and joints. User input system 106c also includes sensor system 109, which communicates with user input devices 108c and 108d to detect user input. User input and output system 113 also includes user output system, which includes a monitor-type display device 107c that provides a single field of view or 3D image in various implementations. For ease of description below, 106 is used to refer to a general user input system, and 106b and 106c are used to refer to... Figure 1B , Figure 1C The specific example shown. Similarly, 107 is generally used to indicate a display device that includes a user output system, while 107b and 107c are used to indicate... Figure 1B , Figure 1C The specific example shown.

[0035] When the system 100 operates in follow mode, the operator 104 can operate the user input system to generate a set of user input signals to control the movement of the manipulator 102. Figure 1B and Figure 1C A controller 110 is shown physically located along with the user input and output system. However, the controller 110 may be physically separated from the user input and output system and communicate with the user input and output system via electronic signals transmitted through wired or wireless technologies.

[0036] During operation of system 100, while manipulator 102 is being controlled by operator 104, operator 104 can view display device 107 to view images, such as two-dimensional or three-dimensional images, representing the apparatus mounted on manipulator 102. For example, an apparatus including an image capture device such as a camera is mounted on one of the manipulators 102. The image capture device generates images of the distal portions of other apparatuses mounted on other manipulators 102. During operation of system 100, operator 104 can use the images presented on display device 107 to monitor the attitude of the distal portions of the apparatus.

[0037] User input system 106 is connected to controller 102, for example, wirelessly or via a wired connection. User input system 106 includes several different parts operable by operator 104 to control the operation of controller 102. In some cases, these user-operable parts correspond to different user input devices. Figure 1B and Figure 1CIn the example depicted, user input system 106 includes manually operable user input devices (e.g., 108a, 108b are shown for 106b, and 108c and 108d are shown for 106c) (collectively referred to as user input devices 108) corresponding to the user-operable portion of user input system 106 and movable relative to manipulator 102 to control the movement of manipulator 102. In addition to the user-operable portion, user input system 106 may also include other user input devices, such as keyboards, touchscreens, buttons, foot pedals, etc., which are operated in follow mode to control the movement of manipulator 102 or other operations of manipulator 102. These other user-operable portions can be used to allow user control of display device 107 and additionally allow user control of other operations of system 100.

[0038] As described herein, in response to operations of the user-operable portion in pairing mode, the controller 110 of system 100 ( Figure 1B and Figure 1C As shown, the user-operable portion of the user input system 106 can be associated with a corresponding manipulator in the manipulator 102. When associated, the user-operable portion can be operated to control the corresponding manipulator in follow mode to perform operations, such as medical procedures, surgical procedures, diagnostic procedures, etc.

[0039] Figure 2 An example of a manipulator system 101 is shown. For simplicity, only the manipulators 102a and 102b of the manipulator system 101 are shown. In some implementations, the manipulator system 101 may include a single manipulator or include three or more manipulators, for example, as shown in... Figure 1A The four manipulators depicted are 102a, 102b, 102c, and 102d.

[0040] Although the description of manipulators 102a and 102b Figure 2 ,but Figure 1A The manipulators 102c and 102d may include features similar to those present with respect to manipulators 102a and 102b. Manipulators 102a, 102b, 102c, and 102d may differ from one another, as different instruments may be mounted on manipulators 102a, 102b, 102c, and 102d. Furthermore, manipulators 102a, 102b, 102c, and 102 may be supported by the operating table 105 at different locations along the operating table 105.

[0041] Manipulators 102a and 102b include portions movable around workspace 114. For example, these portions may correspond to distal ends 112a and 112b of manipulators 102a and 102b movable around workspace 114. Distal ends 112a and 112b support instruments 116a and 116b such that instruments 116a and 116b are movable around workspace 114 when distal ends 112a and 112b move around workspace 114. In some implementations, actuation modules 117a and 117b may be supported at the distal ends 112a and 112b of manipulators 102a and 102b. Actuation modules 117a and 117b are removably mounted to the distal ends 112a and 112b of manipulators 102a and 102b, and include one or more actuators operable to generate insertion and rolling movements of instrument 116a and one or more actuators 116b. Instruments 116a and 116b can be inserted through actuation modules 117a and 117b such that instruments 116a and 116b are attached to actuation modules 117a and 117b, which in turn are attached to the distal ends 112a and 112b of manipulators 102a and 102b.

[0042] Manipulators 102a and 102b include power joints 118a and 118b that can be driven to move the distal ends 112a and 112b of manipulators 102a and 102b about workspace 114. Each manipulator 102a and 102b includes multiple power joints 118a and 118b capable of enabling movement of the distal ends 112a and 112b in multiple degrees of freedom, such as pitch, yaw, and roll movements of the distal ends 112a and 112b of manipulators 102a and 102b. The instruments and manipulators described herein may have one or more degrees of freedom that vary in implementation. For example, one or more degrees of freedom may include one or more of the following: yaw movement of the distal portion of the manipulator, pitch movement of the distal portion of the manipulator, insertion movement of the instrument supported by the manipulator, roll movement of the instrument, yaw movement of the end effector of the instrument, wrist movement of the end effector of the instrument, or gripping or holding movement of the end effector of the instrument.

[0043] System 100 is a computer-aided system. For example, controller 110 can control the operation of system 100 and coordinate the operation of various subsystems of system 100, including but not limited to manipulator 102, user input system 106, and user output system. Although schematically depicted as a controller 103, in some implementations, controller 110 may include one or more processors external to console 103 and be operable to control any subsystem of system 100, such as manipulator 102 and console 103.

[0044] In some examples, controller 110 can control the operation of actuators of power joints 118a, 118b and actuators of actuation modules 117a, 117b. When the user-operable portion of the user input system 106 associated with manipulators 102a, 102b is operated by operator 104, the distal ends 112a, 112b of manipulators 102a, 102b, and therefore instruments 116a, 116b, can move around workspace 114.

[0045] In follow mode, the follower of the manipulator moves in response to movement of the leader. The movement of the follower can mimic the movement of the leader. For example, for a specific manipulator, the leader can be one or more of the user input devices 108, and the follower can be one or more components of the manipulator. The follower can be an end effector of the manipulator, a remote center of the manipulator, or some other component of the manipulator. In some examples, in follow mode, the distal ends 112a, 112b are the followers. For example, actuators of the power joints 118a, 118b can be controlled to generate movement of the links of the manipulators 102a, 102b about the power joints 118a, 118b, thereby repositioning the distal ends 112a, 112b of the manipulators 102a, 102b. The movement of the distal ends 112a, 112b mimics the movement of the user input device 108. In other examples, movement of the user input device 108 in follow mode can cause an instrument mounted to the distal end 112a or 112b to eject from the distal end 112a or 112b. In other examples, in follow mode, actuation modules 117a, 117b can be controlled to generate insertion movement of instruments 116a, 116b or to actuate the end effectors of instruments 116a, 116b.

[0046] See Figure 1A , Figure 1B , Figure 1C and Figure 2 As shown, in some implementations, system 100 is a medical system for performing medical procedures on patient 120. For example, system 100 is a diagnostic system that can be used to perform diagnoses on patient 120. Alternatively or additionally, system 100 is a surgical system that can be used to perform surgery on patient 120.

[0047] Various alternative computer-aided telemanipulation instruments 116a, 116b can be used. For example, telemanipulation instruments 116a, 116b can be different types of surgical instruments with different end effectors. In some cases, instruments 116a, 116b include multiple degrees of motion (DOFs), such as, but not limited to, rolling, pitching, deflecting, insertion depth, gripper opening / closing, staple delivery actuation, electrocautery activation, and the like. Movements in at least some of these DOFs can be generated by actuation modules 117a, 117b of manipulators 102a, 102b to which instruments 116a, 116b are selectively coupled.

[0048] If instruments 116a and 116b are medical or surgical instruments, possible end effectors include, for example, DeBakey forceps, microforceps, and Potts scissors, which include first and second end effector elements that pivot relative to each other to define a pair of end effector grippers. Other end effectors (including scalpels and electrocautery probes) have a single end effector element. For instruments with end effector grippers, the grippers are typically actuated by squeezing the gripping member of the input device. The instrument may include a flexible shaft that can be deflected to reposition the distal end of the shaft. In some cases, one or more of instruments 116a and 116b include an image capture device. Examples of instruments with image capture devices include endoscopes, ultrasound probes, fluorescence microscope probes, etc. The image capture device can capture images of other instruments in the workspace 114 (such as...). Figure 2 (As shown), and this image can be presented to operator 104 to allow operator 104 to visually monitor the position of other instruments in workspace 114. Pairs of user-operable portions of user input system 106 can also be used to control the actuation of end effector.

[0049] Figure 3A and Figure 3B Examples of display devices 107 are shown. These examples depict display devices 107 in a pairing mode, where display devices 107 present representations 122a, 122b of instruments 116a, 116b and present a virtual selector 121. In pairing mode, operator 104 operates user input system 106 to reposition (e.g., translate or rotate) the virtual selector 121 to associate the manipulators 102a, 102b supporting instruments 116a, 116b with corresponding user-operable portions of user input system 106 (e.g., user input device 108). Although Figure 3A , Figure 3B as well as Figure 4 The association of user input device 108 is described herein, but in other implementations described herein, other examples of user-operable portions of the user input system may be associated with manipulators 102a, 102b.

[0050] Go to Figure 3A and Figure 3B For example, the image presented on display device 107 is captured by one or more image capture devices. In some examples, as described herein, the image capture device is coupled to an instrument mounted to one of the manipulators 102 to capture images of instruments (e.g., instruments 116a, 116b) coupled to other manipulators 102 (e.g., manipulators 102a, 102b). In some examples, the image capture device is a still image capture device in environment 10 that captures images of instruments 116a, 116b.

[0051] exist Figure 3A In the example shown, display device 107 presents images of diagrams 122a and 122b, which include instruments 116a and 116b. Diagrams 122a and 122b show the positions of instrument 116a supported by manipulator 102a and instrument 116b supported by manipulator 102b, respectively.

[0052] Figures 122a and 122b may represent a portion of an image captured by an imaging system (e.g., an endoscope). The captured image presented on display device 107 is unchanged compared to the image captured by the imaging system. In other examples, Figures 122a and 122b may represent a portion of an image captured by an imaging system, but then altered in some way. For example, the image may be altered to include highlighting, edge finding, overlaying text, overlaying graphics, or other indicators. In a further example, Figures 122a and 122b may represent part or all of sensor information (e.g., from the information described herein and in…) Figure 4 The sensor information (collected by the sensor system 200 shown in the figure) is part of a composite image. For example, the sensor information may include information collected from shape sensing sensors or other kinematic information for the joints and links of the manipulators 102a and 102b.

[0053] In a two-dimensional image implementation, the location representing one of the instruments 116a and 116b in the image can be a single point, a set of points, or a two-dimensional region in the image. In a three-dimensional image implementation, the location representing one of the instruments 116a and 116b can be a single point, a set of points, a two-dimensional region, or a three-dimensional volume in the image. For a three-dimensional image, two or more image capture devices or image capture devices with depth sensing or stereoscopic configuration can be used to capture the image to form the representation shown on the display device 107.

[0054] The diagrams 122a and 122b show the instruments 116a and 116b in the image, indicating that instruments 116a and 116b are in the workspace 114 of instruments 116a and 116b. Figure 2 The relative orientation, such as orientation, or both, is shown in the figure. In some examples, the image is a digital image captured by an image capture device coupled to another instrument in the workspace, and Figures 122a and 122b are therefore represented as portions of the captured digital image. Alternatively or additionally, the image is a rendering generated based on the image captured by the image capture device. In this case, Figures 122a and 122b are represented as graphic indicators or portions of the rendering showing the relative orientation of instruments 116a and 116b.

[0055] Virtual selector 121 is a graphical indicator. Virtual selector 121 in Figure 3A The virtual selector 121 has a roughly arrow-triangle shape; in other implementations, the virtual selector 121 can have any suitable shape, including symmetrical shapes such as circles. In some implementations, the virtual selector 121 is a two-dimensional virtual selector overlaid on a two-dimensional or three-dimensional image presented by the display device 107. In some implementations, the virtual selector 121 is a three-dimensional virtual selector overlaid on a three-dimensional image presented by the display device 107. In some implementations, the virtual selector 121 represents a two-dimensional or three-dimensional rigid body. In some implementations, the virtual selector 121 represents a compliant body or assembly coupled to one or more joints that allow internal degrees of freedom. In some implementations, in addition to being defined by its geometry, or instead of being defined by its geometry, the virtual selector 121 is also defined by shading, patterns, flashing lights, or other graphical attributes. Similar to the geometry of the virtual selector, this graphical attribute can be symmetrical or asymmetrical. In other implementations, the virtual selector is an augmented reality element. For example, the virtual selector can be a graphical indicator overlaid on a part of the environment.

[0056] The position (including positional changes through movement of the virtual selector 121), orientation, or a combination of position and orientation, of the virtual selector 121 can be controlled by the operator 104 through the operation of the user input system 106. The virtual selector 121 has multiple degrees of freedom of motion relative to the image presented by the display device 107. These degrees of freedom allow the virtual selector 121 to be movable in the space represented by the image (e.g., two-dimensional or three-dimensional space). In some implementations, the virtual selector 121 includes fewer than six degrees of freedom, such as five degrees of freedom without scrolling in three-dimensional space, three degrees of freedom (translation and rotation) in two-dimensional space, two degrees of freedom without rotation in two-dimensional space, etc.

[0057] When the images and representations 122a and 122b are three-dimensional representations, the virtual selector 121 can be translated and rotated in the three-dimensional space represented in the images. In some examples, the virtual selector 121 has six degrees of freedom, including three translational degrees of freedom (e.g., horizontal movement along a first axis, horizontal movement along a second axis, and vertical movement) and three rotational degrees of freedom (e.g., yaw, pitch, and roll).

[0058] In some implementations, the image presented on display device 107 may correspond to a two-dimensional projection of the workspace (including instruments 116a, 116b), the projection being formed based on an image captured by a two-dimensional digital image capture device. The image presented on display device 107 represents instruments 116a, 116b in two-dimensional space (e.g., figures 122a, 122b represent two-dimensional representations of instruments 116a, 116b). The virtual selector 121 has movable degrees of freedom including two translational degrees of freedom and two rotational degrees of freedom.

[0059] Controller 110 is configured to operate display device 107 in pairing mode to present virtual selector 121 so that a user can select a manipulator (e.g., one of manipulators 102) associated with a specific user input device. Controller 110 can operate display device 107 in any appropriate manner. For example, the controller can directly drive display device 107 and provide pixel-by-pixel instructions for rendering images. As another example, controller 110 can provide one or more images to the display controller of display device 107 for rendering, blending, or blending and rendering. As yet another example, controller 110 can provide instructions to a coprocessor or display processing system to determine an image to be displayed, and the coprocessor or display processing system will operate display device 107 to display such an image.

[0060] In response to an operation by operator 104 on user input system 106, virtual selector 121 may be repositioned and / or reoriented to form an association between a specific manipulator and a specific user input device. Specifically, operator 104 controls the position and / or orientation of virtual selector 121 relative to the representation diagrams 122a, 122b of instruments 116a, 116b to select one of manipulators 102 for association.

[0061] When operator 104 operates user input system 106, in response to user input signals generated by user input system 106, controller 110 generates control signals to reposition virtual selector 121 on display device 107. The user input signals may be generated based on the operator's intention to move virtual selector 121 in one or more degrees of freedom. In some implementations, controller 110 receives user input signals indicating movement in multiple degrees of freedom and generates control signals to move virtual selector 121 along a subset of the multiple degrees of freedom. For example, the repositioning of virtual selector 121 in one or more of the multiple degrees of freedom may not be visible to operator 104. If the image is a two-dimensional image presented on display device 107 and display device 107 only presents two-dimensional images, the subset of multiple degrees of freedom may include horizontal translation degrees of freedom and vertical translation degrees of freedom. This subset of multiple degrees of freedom excludes another horizontal translation degree of freedom, where the movement of virtual selector 121 in another horizontal translational freedom will not be represented on display device 107.

[0062] In some cases, the virtual selector 121 is axisymmetric, for example, about a geometry, a graphic property, or both. For example, the virtual selector 121 may be a cone, arrowhead, prism, or other axisymmetric shape. In some implementations, the virtual selector 121 is axisymmetric about one, two, or more axes. Through axisymmetry, the virtual selector 121 does not appear to be repositioned due to rotation about a particular axis. In this respect, a subset of the multiple degrees of freedom may include one or two of the three available rotational degrees of freedom. This subset of the multiple degrees of freedom excludes rotational degrees of freedom about an axis in which the virtual selector 121 is axisymmetric.

[0063] Figure 3A and Figure 3B The instruments 116a, 116b or their corresponding manipulators 102a, 102b are shown. Figure 2 (shown in the image) and user input device 108 ( Figure 1B and Figure 1C The process is associated with (as shown in the diagram). When pairing mode is activated, operator 104 operates user input system 106 to select a user input device associated with the operator. For example, operator 104 selects one of user input devices 108a and 108b. After activating pairing mode, refer to... Figure 3A As shown, the display device 107 presents the virtual selector 121 at an initial position where the association condition for associating the user input device with the manipulator is not met. For example, in a scenario where proximity (e.g., graphic proximity) of the virtual selector 121 is an implementation of the association condition, the initial position is not close to the representations 122a and 122b of the instruments 116a and 116b presented on the display device 107.

[0064] Then, operator 104 operates user input system 106 to control the position of virtual selector 121 and reposition virtual selector 121 relative to the image. This repositioning of virtual selector 121 can be controlled by operator 104 in a way that selects one of the representation figures 122a, 122b of instruments 116a, 116b and thus selects one of manipulators 102a, 102b to associate with the selected user input device. In response to operation of user input system 106, user input system 106 generates a set of signals for controlling the orientation of virtual selector 121. Specifically, this set of signals causes display device 107 to reposition virtual selector 121. For example, controller 110 processes this set of signals to determine that virtual selector 121 should move relative to the image; then controller 110 causes display device 107 to reposition virtual selector 121.

[0065] To select the manipulator 102a associated with the chosen user input device, the display device 107 is controlled such that the presented virtual selector 121 is repositioned relative to the image to a position closer to the position shown in Figure 122a. Based on the repositioning of the virtual selector 121 relative to the represented position of the instrument 116a, the manipulator 102a supporting the instrument 116a is associated with the user input device. In response to the repositioning of the virtual selector 121 relative to the represented position of the instrument 116a satisfying the association condition, the controller 110 establishes an association between the manipulator 102a and the user input device.

[0066] In some implementations, input to the user input device can reposition the virtual selector 121 but not reorient it. In other implementations, input to the user input device can both reposition and reorient the virtual selector 121; the reorientation of the virtual selector 121 can be achieved using techniques similar to those described above for repositioning the virtual selector 121.

[0067] In some implementations, the association conditions used to associate the manipulator with the user input device do not include the orientation of the virtual selector 121. In other implementations, the association conditions used to associate the manipulator with the user input device include the orientation of the virtual selector 121.

[0068] As an example of how orientation-related association conditions are implemented, the association conditions include a first condition and a second condition. The first condition corresponds to the virtual selector 121 being close to the position representing Figure 122a, and the second condition corresponds to the virtual selector 121 being oriented toward the orientation representing Figure 122a (e.g., the virtual selector 121 has such an orientation as shown in Figure 122a). Figure 3A The vertex or point shown makes the longitudinal axis passing through the vertex pass through (represented in Figure 122a).

[0069] Many of the examples discussed below focus on changing the orientation (e.g., position) of the virtual selector 121 and its orientation or translational motion compared to position- or motion-based association conditions. However, similar to Figure 3A Examples, and other examples discussed in this disclosure, can also be implemented with or without orientation considerations in the associated conditions.

[0070] refer to Figure 3B As shown, the association condition corresponds to the overlap between the virtual selector 121 and the region representing the location of the instrument 116a in the presented image. The user input system 106 is operated such that the virtual selector 121 is repositioned closer to the represented location of the instrument 116a to satisfy the association condition. In some examples, reference... Figure 3B As shown, the area defined by the representation position of device 116a corresponds to the range occupied by representation figure 122a in the presented image. Alternatively or additionally, this area is defined by the representation position of the distal portion of device 116a in the presented image or the representation position of the end effector of device 116a in the presented image. In the example association process, the virtual selector 121 selects from its position in... Figure 3A The position shown is repositioned to its location in Figure 3B The location shown is in Figure 3B In this diagram, the virtual selector 121 overlaps with the area represented by Figure 122a. This overlap corresponds to an association condition that links the device 116a to the user input device. In this respect, the association condition is satisfied when the virtual selector 121 is repositioned to achieve this overlap. When the association condition is satisfied, the controller 110 operates the display device 107 to present a success indicator 133.

[0071] In some implementations, controller 110 presents visual feedback during the pairing process. For example, in some implementations, controller 110 presents colors, numbers, text, graphics, or other visual indicators that overlap with or are close to the representation of Figure 122a to indicate the pairing status or suggest pairing. As a specific example, when pairing mode is initiated, a green light or a flashing “O” indicating the vicinity of Figure 122a may indicate that device 116a (and its manipulator 102a) is in a non-paired state, and a red light or a steadily presented (non-flashing) “O” may indicate that device 116a (and its manipulator 102a) is in a paired state. In some cases, a yellow light or a flashing or steadily presented “X” may provide a visual warning.

[0072] In some implementations, controller 110 causes visual feedback indicating additional information about the status of user input device 108 and manipulator 102. In some implementations, the visual feedback indicates which user input device is recommended to be associated with which manipulator, or indicates which user input device will be associated with which manipulator after confirmation. As an example, when suggesting that user input device 108a and instrument 116a (or its manipulator 102a) be associated with each other, or which manipulator they will be associated with after confirmation, virtual selector 121 and display figure 122a may flash matching, similar, or identical visual feedback: colors, numbers, text, graphics, flashing sequences, or other visual feedback.

[0073] In some implementations, the controller elicits visual feedback indicating which user input device has been or is currently associated with which instrument or manipulator. As an example, after user input device 108a is associated with instrument 116a (or its manipulator 102a), controller 110 may consistently present color, numbers, text, graphic patterns, or other visual feedback indicating association in the vicinity of an appropriate representation of the instrument. In various implementations, this consistent presentation of color, numbers, text, graphics, or other textual feedback may persist throughout the entire duration of the association between the user input device and the instrument.

[0074] exist Figure 3B In the example shown, controller 110 has made the association indicator 128b (displayed as the letter "L") appear to overlap with representation figure 122a to indicate the association status of instrument 116a with the input device identified by "L". The association indicator 128b may be displayed for part or all of the time during which user input device 108a is associated with instrument 116a.

[0075] about Figures 7A to 7E Other examples describing association conditions.

[0076] refer to Figure 4 And as described herein, examples of system 100 for performing the associated process include manipulator system 101, controller 110, user output system 202, and user input system 106. Although referring to Figure 1, Figure 2 , Figure 3A and Figure 3B It is described as containing four manipulators, but in some implementations, such as Figure 4 As shown, the manipulator system 101 may include any number of manipulators. For example, the manipulator system 101 includes N manipulators (e.g., manipulators 1 to manipulator N, collectively referred to as manipulator 102). Similarly, although referring to Figure 1, Figure 2 , Figure 3A and Figure 3BIt is described as including two user input devices, but in some implementations, such as Figure 4 As shown, the user input system 106 includes any number of user input devices or user-operable parts. For example, the user input system 106 includes M user input devices (e.g., user input devices 1 to M, collectively referred to as user input devices 108). Examples of user input devices 108 include: joysticks, touchscreens, gloves, foot pedals, or handheld remote controls.

[0077] In some implementations, system 100 includes sensor system 200. Sensor system 200 includes sensors operable to detect movement of user input device 108. Sensor system 200 can detect the attitude of user input device 108 and manipulator 102 within environment 10, such as orientation, orientation, or both. Sensors of sensor system 200 include, for example, infrared sensors, ultrasonic sensors, image capture devices, accelerometers, orientation encoders, optical sensors, or other suitable sensors for detecting movement and attitude of manipulator 102 and user input device 108.

[0078] User output system 202 provides human-perceptible feedback to operator 104 and includes display device 107. The feedback provided by user output system 202 may include feedback provided during association processes or during follow mode to provide operator 104 with guidance for controlling virtual selector 121 or for controlling manipulator 102, respectively. Furthermore, user output system 202 is operable to present virtual selector 121 during pairing mode (e.g., on display device 107) to enable operator 104 to select user input devices and manipulators associated with each other. In some implementations, user output system 202 and user input system 106 correspond to console 103.

[0079] System 100 may further include a memory storage element 204. The memory storage element 204 may store data indicating an association formed between the operator 102 and the user input device 108. The controller 110 may retrieve this stored data to determine whether the user input device or the operator is in an associated or unassociated state. (Reference) Figure 5 The manipulator 102 is associated with the user input device 108, such that each user input device 108 is associated with a different manipulator 102. As a result, the user input device 108 can be controlled by the operator 104, allowing the associated manipulator to be controlled independently. In some cases, each manipulator 102 is associated with a corresponding one of the user input devices 108. Consequently, each manipulator 102 can be controlled using the user input device 108.

[0080] Example process

[0081] refer to Figure 6 The system 100 described herein presents a process 600 including an association process and a follow-up process. Process 600 is performed by the user input system 106 and user output system 202, manipulator system 101, controller 110, other parts of system 100 (e.g., console 103, system 113), or a combination thereof. At operation 601, a pairing mode is initiated to associate one or more user-operable portions of the user input system 106 with one or more manipulators of the manipulator system 101. At operation 602, an association process is performed to associate a specific manipulator with a specific user-operable portion of the user input system 106. Both the specific manipulator and the specific user-operable portion can be selected by operator 104. Further operations and examples of sub-operations of operations 601 and 602 are provided in the reference section. Figures 7A to 7E , Figure 8A , Figure 8B and Figure 9 Describe it.

[0082] At operation 603, a follow mode is initiated, allowing the manipulator to be controlled in response to operations on the user-operable portion during the follow process. In some implementations, in follow mode, the manipulator associated with the user-operable portion at operation 602 can move in response to operations on the user-operable portion by operator 104. In response to operations on the user-operable portion, user input system 106 generates a set of user input signals to control the orientation of the manipulator. Controller 110 then generates a corresponding set of control signals based on this set of user input signals. This set of control signals is sent to the manipulator to move (e.g., during pairing mode) the manipulator associated with the user-operable portion. This results in movement of the manipulator and the instrument mounted on it. In this respect, the user-operable portion and the manipulator form a master-slave system (or guide-follower system), where the user-operable portion is the guide device and the manipulator is the follower device, thereby enabling the manipulator to be remotely operated via operations on the user-operable portion. If system 100 is a surgical system, instruments supported by the manipulator can be controlled to perform surgical procedures on a patient.

[0083] Figure 7A An example of an association process 700 is shown that associates a specific user-operable portion of the user input system 106 with a specific manipulator of the manipulator system 101. For example, process 700 is executed during operations 601 and 602 as described in reference process 600.

[0084] Figure 7AOperations 701-703 illustrate example setup operations for initiating the pairing mode. At operation 701 of process 700, operator 104 operates user input system 106 to initiate the pairing mode. For example, user input system 106 includes a user-operable section dedicated to initializing the pairing mode, and operator 104 operates this dedicated user-operable section to initialize the pairing mode. This dedicated user-operable section may correspond to a button that initiates the pairing mode when manually operated by operator 104. At operation 702, user input system 106 sends a signal to controller 110 to initiate the pairing mode. At operation 703, controller 110 initiates the pairing mode.

[0085] Once in pairing mode, a specific user-operable part is selected for association. For example, operator 104 operates user input system 106 to select a user-operable part. Alternatively, controller 110 automatically selects one of the user-operable parts for association. In pairing mode, operator 104 further provides association intent to associate a specific user-operable part with a specific manipulator. Additionally, feedback is provided to operator 104 so that operator 104 can be informed of the status of the manipulator of manipulator system 101 and the user-operable part of user input system 106. Operations 704-713 illustrate examples of operations that occur during pairing mode.

[0086] In some implementations, after the pairing mode is initiated at operation 703, at operation 704, controller 110 sends a signal to provide an association indicator to operator 104. The signal can be sent to user output system 202. User output system 202 presents the association indicator to indicate the association status of each manipulator in manipulator system 101.

[0087] Figure 7B An example of visual feedback that can be provided at operation 703 is shown. The visual feedback includes an association indicator for providing information indicating the association status of manipulators 102a, 102b (not shown). The association status of manipulators 102a, 102b can be unassociated or associated, wherein an unassociated status indicates that the manipulator is not associated with a user-operable part, while an associated status indicates that the manipulator is associated with user-operability.

[0088] refer to Figure 7BThe display device 107 can present visual feedback, including a status indicator 132a for the manipulator 102a of the supporting device 116a and a status indicator 132b for the manipulator 102b of the supporting device 116b. Status indicators 132a and 132b are positioned near the distal portions of the representation diagrams 122a and 122b of the devices 116a and 116b. Status indicator 132a indicates that the manipulator 102a is in a non-associative state, while status indicator 132b indicates that the manipulator 102b is in an associated state. Status indicators 132a and 132b can visually inform the operator 104 of the associated state of the manipulators 102a and 102b, allowing the operator 104 to provide an association intention based on the associated state of the manipulators 102a and 102b.

[0089] Back Figure 7A At operation 705, operator 104 operates user input system 106 to provide associated intent. For example, user input system 106 generates a set of user input signals for controlling the orientation (e.g., position) and / or orientation of virtual selector 121. This set of user input signals is generated in response to operation of user input system 106. The following discussion focuses on controlling the orientation of virtual selector 121. In implementations that also control the orientation of virtual selector 121, a similar process can be used to orient and reorient virtual selector 121.

[0090] In some implementations, the operator 104 manipulates a user-operable portion of the user input system 106 to generate the set of signals. The manipulated user-operable portion may correspond to a specific user-operable portion to be paired with the operator. In other implementations, the user input system 106 includes a user-operable portion dedicated to use by the operator 104 to induce repositioning of the virtual selector 121. In this respect, the user-operable portion manipulated to control the orientation and orientation of the virtual selector 121 may be different from the user-operable portion that can be associated with the operator.

[0091] At operation 706, in response to a set of user input signals generated by user input system 106, the user output system 202 of console 103 repositions the virtual selector 121 relative to the image presented by user output system 202. Figure 3A and Figure 3B (Description). For example, controller 110 generates a set of control signals in response to the set of user input signals and sends the set of control signals to user output system 202 to control the orientation and orientation of virtual selector 121.

[0092] The repositioning of the virtual selector 121 can occur in several ways. In some implementations, the virtual selector 121 may move relative to the image in response to a set of signals generated by the user input system 106. The virtual selector 121 moves along a continuous path from a first orientation to a second orientation in response to this set of signals. For example, the user-operable portion includes a user input device such as a joystick, and the virtual selector 121 moves relative to the image in response to manual manipulation of the joystick. The user output system 202 presents the virtual selector 121 on the display device 107 such that, when viewed by the operator 104, the virtual selector 121 appears to translate across the display device 107. Similarly, the virtual selector 121 may move through an orientation between a first and a second orientation in response to this set of signals. In this respect, the virtual selector 121 appears to rotate continuously.

[0093] In some implementations, instead of moving across the display device 107 relative to the image, the virtual selector 121 is repositioned on the display device 107 from a first orientation to a second orientation without moving along a path from the first orientation to the second orientation. Alternatively or additionally, the virtual selector 121 is repositioned on the display device 107 from a first orientation to a second orientation without continuously rotating from the first orientation to the second orientation. The user input system 106 is operated to select the position or orientation of the virtual selector 121 after, for example, repositioning the virtual selector 121 to a second orientation. For example, the user input system 106 may include a touchscreen, and the operator 104 selects the position by touching a portion of the touchscreen. In response to this selection, the display device 107 presents the virtual selector 121 in the second orientation or second orientation without any movement of the virtual selector 121 between the first and second orientations or between the first and second orientations.

[0094] Back Figure 7A At operation 707, controller 110 determines whether the repositioning of virtual selector 121 satisfies association conditions. Association conditions can vary between implementations. Association conditions may include conditions regarding the orientation of virtual selector 121, conditions regarding the orientation of virtual selector 121, or the amount of time virtual selector 121 spends in a specific orientation or region. Furthermore, as combined with... Figure 3A As discussed and applicable to the various examples disclosed herein, associated conditions may include conditions based on the orientation of the virtual selector 121.

[0095] In some implementations, refer to Figure 7CThe association condition that links device 116a to the user-operable portion corresponds to the position where virtual selector 121 is repositioned on or near representation diagram 122a of device 116a. In some cases, display device 107 presents selectable indicators 121a and 121b close to representation diagrams 122a and 122b. In some cases, selectable indicators 121a and 121b do not overlap with representation diagrams 122a and 122b. The association condition is satisfied when virtual selector 121 is repositioned on or near one of selectable indicators 121a and 121b. For example, virtual selector 121 overlaps with selectable indicator 121a to satisfy the association condition for associating the user-operable portion with manipulator 102a, or virtual selector 121 overlaps with selectable indicator 121b to satisfy the association condition for associating the user-operable portion with manipulator 102b.

[0096] In some implementations, refer to Figure 7D The association condition that links the manipulator 102a to the user-operable portion corresponds to the virtual selector 121 being located within the region 123a surrounding the representation figure 122a. For example, region 123a includes the combination of (i) a portion 124 of the representation figure 122a covering the end effector of the device 116a in the image and (ii) a range surrounding that portion 124 in the image. Therefore, the virtual selector 121 can trigger an association with the manipulator 102a without overlapping the range covered by the representation figure 122a in the image.

[0097] In some implementations, region 123a is defined by a predetermined distance to a specific point on the representation of Figure 122a. For example, the specific point could be the centroid of the area covered by the representation of Figure 122a, the centroid of the area covered by a portion 124 of the representation of Figure 122a, or another point along the representation of Figure 122a. Alternatively or additionally, region 123a is a shape having a predefined size and defining the representation of Figure 122a or defining a portion 124 of the representation of Figure 122a. The shape of region 123a can be, for example, rectangular, circular, oval, or other suitable shapes.

[0098] When the virtual selector 121 is repositioned into region 123a, the association condition can be immediately satisfied. In some implementations, the controller 110 further requires the virtual selector 121 to remain positioned within region 123a for a predefined time period, such as 0.5 seconds to 2 seconds, before considering the association condition satisfied. In some implementations, the controller 110 further requires the virtual selector 121 to remain substantially stationary within region 123a for a predetermined time period.

[0099] In some implementations, controller 110 provides feedback to operator 104 while virtual selector 121 is being repositioned during pairing mode. Controller 110 provides feedback in response to the repositioning of virtual selector 121. See, for example, [link to relevant documentation]. Figure 7D The virtual selector 121 is repositioned to be close to region 125a, which represents Figure 122a. For example, region 125a surrounds both Figure 122a and region 123a. Therefore, region 125a includes at least a portion 124 of Figure 122a.

[0100] In some implementations, the repositioning of the virtual selector 121 that satisfies the association condition corresponds to the movement of the virtual selector 121 toward the representation diagram 122a of the instrument 116a. For example, the association condition is satisfied when the velocity or acceleration of the virtual selector 121 is defined by a vector that intersects with the representation position of the instrument 116a or region 123a.

[0101] Figure 7E An example of visual feedback provided to operator 104 via display device 107 after virtual selector 121 is repositioned within area 125a is shown. In response to the repositioning of virtual selector 121 into area 125a, display device 107 presents an information box 126, such as a prompt box, which includes information about instrument 116a and manipulator 102a, including the type 127 of instrument 116a and the association status 128 of manipulator 102a. Association status 128 may include any appropriate amount of information about the association status. In some implementations, association status 128 simply indicates "associated" or "not associated." In some implementations, association status 128 indicates which input device instrument 116a is associated with. Figure 7E In the example shown, associated state 128 indicates that the state is "associated" with the input device "L", and is supplemented by an associated indicator 128b ("L") that covers or is close to the one in Figure 122a.

[0102] The association status 128 and association indicator 128b can be indicated by any one or a combination of color, numbers, text, graphic patterns, or other visual feedback. In some embodiments where the input device 108 includes a visual feedback device such as a light or display, the input device may also present matching, similar, or identical colors, numbers, text, graphics, or other visual feedback, such as those used for a representation of an associated instrument. In various implementations, the association indicator 128b is presented for a portion or the entire duration of the association between the user input device 108a and the instrument 116a.

[0103] like Figure 7EIn the example, instrument 116a is a cutter, and the associated state of manipulator 102a is a non-associated state. As virtual selector 121 approaches representation 122a, display device 107 is also operated to provide a magnified representation 129 of instrument 116a. The magnified representation 129 can provide operator 104 with visual confirmation that instrument 116a is the desired instrument for association with a user-operable portion. Operator 104 can more easily identify instrument 116a through the magnified representation 129.

[0104] A magnified view 130 of the virtual selector 121 can also be presented, allowing the operator 104 to monitor the movement of the virtual selector 121 relative to view 122a by monitoring the movement of the magnified view 130 relative to the magnified view 129. These magnified views 129, 130 provide the operator 104 with a larger target for selection by using the virtual selector 121, thus making the selection device 116a easier.

[0105] Back Figure 7D As shown, in some implementations, the virtual selector 121 may be repositioned into region 123b to be associated with instrument 116b. Region 123b may have features similar to those of region 123a. Similarly, while repositioning the virtual selector 121 into region 125a for instrument 116a is described as triggering feedback to be provided to operator 104, in some implementations, the virtual selector 121 is repositioned into region 125b to trigger feedback to be provided. Movement of the virtual selector 121 may trigger the provision of feedback associated with instrument 116b.

[0106] In some implementations, the controller 110 associates the actuator with the user-operable part only when the user input system's user-operable part is in a non-associated state. Return to... Figure 7A After controller 110 determines that the repositioning of virtual selector 121 satisfies the association conditions of a specific manipulator, at operation 708, controller 110 determines the association state of the manipulator. Controller 110 determines whether the manipulator is in a non-association state. For example, controller 110 can access memory storage element 204 ( Figure 4 (As shown in the diagram) to determine whether the association of the operator has been stored on the memory storage element 204. If the operator is not in a non-associative state, for example, if it is in an associated state, then at operation 709, the operator 104 confirms that a new association will be provided to the operator or instructs the operator to maintain the stored association. If the operator 104 instructs the operator to maintain the stored association, then at operation 705, the operator 104 operates the user input system to provide another association intention to select another operator.

[0107] If operator 104 confirms that a new association will be provided, controller 110 may remove the stored manipulator association. If it is confirmed at operation 709 that a new association will be created for the manipulator, or if it is determined at operation 708 that the manipulator is in a non-associated state, controller 110 requests user confirmation at operation 710 regarding the association between the user-operable portion and the manipulator. For example, controller 110 sends data indicating a confirmation request to user output system 202.

[0108] At operation 711, operator 104 provides confirmation of the association. In some implementations, operator 104 may provide this confirmation by operating user input system 106. For example, operator 104 may move virtual selector 121 to a predefined area presented on display device 107 to confirm the association. The predefined area may correspond to a selectable button presented on display device 107.

[0109] At operation 712, after receiving confirmation of the association, controller 110 stores the association in, for example, memory storage element 204. Then, at operation 713, controller 110 provides a success signal. For example, user output system 202 is operated to provide a human-perceptible signal indicating successful association between the manipulator and the user input element. The human-perceptible success signal may correspond to... Figure 3B The success indicator 133 is described.

[0110] While the description focuses on associating a single user-operable portion with a single manipulator, in some implementations, operations 704-713 can be repeated to associate other user-operable portions of the user input system 106 with other manipulators of the manipulator system 101. System 100 can remain in a paired mode until operator 104 operates the user input system 106 to provide input indicating the initiation of a follow mode (e.g., initiation operation 603). In follow mode, the user-operable portion already associated with a manipulator can be operated by operator 104 to control the movement of the manipulator.

[0111] In some implementations, controller 110 can provide recommendations to optimize the association formed between the manipulator and the user-operable portion. Figure 8A Process 800 illustrates an example process for providing this recommendation. Process 800 is initiated after pairing mode is activated. After pairing mode is activated, at operation 801, user input system 106 sends an indication of the gesture of user-operable parts of user input system 106 (e.g., the gesture of user-operable parts in environment 10). Figure 1AThe signals are shown in the diagram. At operation 802, the manipulator system 101 sends a signal indicating the attitude of the manipulator of the manipulator system 101 to the controller 110. At operation 803, the controller 110 receives these signals from the user input system 106 and the manipulator system 101. In some implementations, the sensor system 200 detects the attitude of the user-operable part, the manipulator, or both, and sends these signals to the controller 110. Furthermore, the controller 110 further receives signals indicating the orientation and orientation of an image capturing device (e.g., on an instrument supported on the manipulator 102c).

[0112] The controller 110 receives signals and uses kinematic modeling to determine the orientation and orientation of manipulators 102a, 102b, instruments 116a, 116b, and the image capture device. In some cases, one or more signals are generated by sensors of the manipulators (e.g., manipulators 102a, 102b and manipulators equipped with image capture devices) or sensors of the instruments (e.g., instruments 116a, 116b and the image capture device). Sensors of the manipulators include, for example, accelerometers, gyroscopes, encoders, or other sensors associated with joints of manipulators 102a, 102b. Sensors of the instruments include, for example, shape sensors through the axis of the instrument. Alternatively, the orientation and orientation of the manipulators and / or the instruments are determined based on one or more signals from optical sensors (e.g., the image capture device). The manipulators or instruments are equipped with optical references detectable by the optical sensors.

[0113] At operation 804, based on the received signals, controller 110 determines the optimal association between the actuators of actuator system 101 and the user-operable portion of user input system 106. Figure 8B The relative orientation of the display device 107 and the user input device 108 is schematically depicted. In this example, the user input device 108 corresponds to the orientation of the display device 107 and the user input device 108. Figure 8A The user-operable portion is described. A representation of instrument 116a (Figure 122a) appears on the left side of the image presented on display device 107, while a representation of instrument 116b (Figure 122b) appears on the right side of the image. When instruments 116a and 116b are displayed on display device 107, in order to provide operator 104 with intuitive control of instruments 116a and 116b, controller 110 provides a recommendation to associate user input device 108a (in operator 104's left hand) with instrument 116a represented on the left side of the image. Furthermore, controller 110 provides a recommendation to associate user input device 108b (in operator 104's right hand) with instrument 116b represented on the right side of the image.

[0114] Back Figure 8AAs shown, controller 110 can determine the relative orientation and orientation of the user-operable portion and manipulators 102a, 102b based on signals indicating the attitude of these devices. Alternatively, in some embodiments, controller 110 determines the orientation and orientation of the user-operable portion relative to instruments 116a, 116b supported by manipulators 102a, 102b. Controller 110 can determine the relative attitude of instruments 116a, 116b, as they appear on display device 107 to operator 104. Controller 110 can determine recommended associations between the user-operable portion and manipulators 102a, 102b based on these relative attitudes. In various implementations, recommendations may include recommended associations for a subset or all of user input devices (e.g., 108a, 108b) and a subset or all of manipulators (102a, 102b). Furthermore, in various implementations, the recommendation can indicate the degree of recommendation for a particular association, such as: a more recommended association between the user input device and the manipulator (e.g., between the user input device 108a and the manipulator of the holding device 116a), a less recommended association between the user input device and the manipulator (e.g., between the user input device 108a and the manipulator of the device not shown in the holding image), or a less recommended association between the user input device and the manipulator (e.g., between the user input device 108a and the manipulator of the holding device 116b).

[0115] In some implementations, controller 110 does not receive orientation and orientation information for determining recommended user-operable parts. User-operable parts can be configured such that they have fixed orientations and orientations relative to each other. In this respect, controller 110 can provide recommendations based on the orientation and orientation of manipulators 102a, 102b relative to each other or based on the orientation and orientation of instruments 116a, 116b relative to each other.

[0116] After controller 110 determines the optimal association, it provides operator 104 with a signal at operation 805 indicating the optimal association. For example, controller 110 controls user output system 202 to provide appropriate signals to guide operator 104 in forming an optimal association between manipulators 102a, 102b and user-operable parts. For example, display device 107 can be operated to present the operator with recommendations for forming the optimal association determined at operation 804. When a specific user-operable part is selected for association, controller 110 can cause display device 107 to present recommendations that associate the specific user-operable part with a recommended manipulator. Return Figure 8BFor example, when user input device 108a is selected for association, virtual selector 121 is presented on display device 107, and display device 107 further provides a recommendation to reposition virtual selector 121 to select manipulator 102a to associate with user input device 108a. Similarly, when user input device 108b is selected for association, virtual selector 121 is presented on display device 107, and display device 107 further provides a recommendation to reposition virtual selector 121 to select manipulator 102b to associate with user input device 108b.

[0117] In some implementations, before initiating follow mode and after establishing the association between the user-operable portion and the manipulator, the orientation of the user-operable portion and the manipulator can be adjusted for each user-operable portion to allow for easier operation of the user-operable portion in follow mode. To ensure that the operator 104 can control the manipulator 102 using the user-operable portion throughout its entire range of motion, a portion of the user-operable portion can be reoriented or repositioned. Figure 9 The process 900 is executed to achieve this reorientation or repositioning of a portion of the user-operable parts.

[0118] At operation 901, the user input system 106 sends a signal indicating the attitude of a portion of the user-operable portion of the user input system 106. This signal can indicate the orientation and orientation of the user-operable portion relative to its full range of motion. For example, the user-operable portion can correspond to a joystick and be coupled to a sensor system 200 of the joystick. Figure 4 The orientation sensor shown in the image can generate this signal.

[0119] At operation 902, the manipulator system 101 sends a signal indicating the attitude of the manipulator of the manipulator system 101. An orientation sensor (e.g., an encoder, accelerometer, etc.) of the sensor system 200 can generate and transmit this signal. At operation 903, the controller 110 receives these signals from the user input system 106 and the manipulator system 101.

[0120] Based on these signals, at operation 904, controller 110 determines whether the orientation of the user-operable portion relative to its entire range of motion matches the orientation of the manipulator relative to its entire range of motion. For example, the user-operable portion may have a degree of freedom of motion for controlling the deflection motion of the distal end of the manipulator. Controller 110 determines whether the orientation of the user-operable portion within its entire range of motion for that degree of freedom of motion matches the orientation of the distal end of the manipulator within its entire range of motion for its deflection degree of freedom. Controller 110 similarly compares the orientation of a portion of the user-operable portion with respect to each of its other degrees of freedom with the orientation of the manipulator with respect to its other degrees of freedom.

[0121] If a portion of the user-operable part does not match the orientation of the manipulator, controller 110 sends a signal at operation 905 to reorient the portion of the user-operable part. At operation 906, user input system 106 receives the signal. In some cases, the signal causes automatic movement of the portion of the user-operable part. For example, the signal drives one or more actuators to move the portion of the user-operable part. Alternatively, user input system 106 provides feedback to operator 104 to reorient or reposition the portion of the user-operable part. User input system 106 then sends another signal at operation 901 indicating the orientation of the portion of the user-operable part, and controller 110 again determines whether the portion of the user-operable part matches the orientation of the manipulator.

[0122] When controller 110 determines a match at operation 904, follow mode can be initiated. For example, a success signal can be provided at operation 713 of process 700, and then follow mode can be initiated.

[0123] Further implementation methods

[0124] Many implementations have been described, and various combinations, additions, or modifications are anticipated. For example, the above implementations can be combined in any suitable manner. As another example, an implementation may exclude the methods described below, or may include any one or more of the methods described below.

[0125] For example, the controllers, processors, and any related components described herein can be part of a computational system that helps control the system according to the processes and methods described herein. Figure 10 This is a schematic diagram of an example computer system 1000, which can be used to implement methods similar to any computer implementation described herein (e.g., including those related to...). Figures 6 to 9The controller described in association with one or more processes or methods of operation, such as controller 110 of system 100 or other controllers. System 1000 includes components such as processor 1010, memory 1020, storage device 1030, and input / output device 1040. Components 1010, 1020, 1030, and 1040 are interconnected using system bus 1050. Processor 1010 is capable of processing instructions that execute within system 1000. In some examples, processor 1010 is a single-threaded processor, while in some cases, processor 1010 is a multi-threaded processor. Processor 1010 is capable of processing instructions stored in memory 1020 or storage device 1030 to display graphical information for a user interface on input / output device 1040.

[0126] The memory storage device of system 1000 may include memory 1020 and storage device 1030. Memory 1020 stores information within system 1000. This information may be used by the processor when processor 1010 executes the processes and methods described herein. In some examples, memory 1020 is a computer-readable storage medium. Memory 1020 may include volatile memory and / or non-volatile memory. Storage device 1030 is capable of providing mass storage for system 1000. Typically, storage device 1030 may include any non-transitory tangible medium configured to store computer-readable instructions. Optionally, storage device 1030 is a computer-readable medium. Alternatively, storage device 1030 may be a floppy disk device, hard disk device, optical disk device, or magnetic tape device.

[0127] In some cases, processor 1010 communicates with remote computing system 1035. Remote computing system 1035 includes, for example, remote servers, cloud computing devices, or other computing devices remote from processor 1010 and its systems. Remote computing system 1035 includes computing resources in environments remote from processor 1010, such as computing resources remote from surgical environments. In some cases, remote computing system 1035 includes one or more servers that have established a wireless link with processor 1010. For example, remote computing system 1035 includes part of a network-accessible computing platform, which is implemented as computing infrastructure including processors, memory, software, data access, etc., accessible by processor 1010.

[0128] System 1000 includes an input / output device 1040. The input / output device 1040 provides input / output operations for system 1000. In some examples, the input / output device 1040 includes a keyboard, computer mouse, pointing device, voice-activated device, microphone, touchscreen, etc. In some cases, the input / output device 1040 includes a display unit for displaying a graphical user interface.

[0129] The features of the methods and systems described in this application can be implemented in digital electronic circuits, or computer hardware, firmware, or a combination thereof. These features can be implemented in a computer program product tangibly stored in an information carrier. The information carrier can be, for example, a machine-readable storage device for execution by a programmable processor. For example, the operations of processes 600, 700, 800, and 900 can be executed by a programmable processor that executes a program of instructions to perform the functions described herein by manipulating input data and generating output. The described features can be implemented in one or more computer programs executable on a programmable system including at least one programmable processor coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to transfer data and instructions to the data storage system, at least one input device, and at least one output device. The computer program includes a set of instructions that can be used directly or indirectly in a computer to perform an activity or produce a result. The computer program can be written in any form of programming language, including compiled or interpreted languages. The computer program can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. This computer program implements, for example, a Fast Genetic Algorithm (FGA).

[0130] Typically, a computer will also include one or more mass storage devices for storing data files, or operatively coupled to communicate with one or more mass storage devices. Such devices may include disks, such as internal hard disks and removable disks, magneto-optical disks, and optical disks. Storage devices suitable for storing computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM discs. The processor and memory may be supplemented or incorporated into ASICs (Application-Specific Integrated Circuits).

[0131] To provide user interaction, these features can be implemented on a computer with a display device and one or more input devices, such as a CRT (cathode ray tube), LCD (liquid crystal display), or OLED (organic light-emitting diode) monitor for displaying information to the user, and the user can provide input to the computer through the one or more input devices, such as a keyboard, buttons, switches, pedals, computer mouse, touchpad, touchscreen, joystick, or trackball. Alternatively, the computer may not have an attached keyboard, mouse, or monitor and may be remotely controlled by another computer. In some implementations, the display device includes a head-mounted display or an augmented reality display (e.g., augmented reality glasses).

[0132] These features can be implemented in computer systems that include back-end components (such as data servers), middleware components (such as application servers or internet servers), front-end components (such as client computers with graphical user interfaces or internet browsers), or any combination thereof. The components of the system can be connected via digital data communication of any form or medium (such as communication networks). Examples of communication networks include, for example, LANs, WANs, and the computers and networks that form the internet.

[0133] Computer systems can include clients and servers. Clients and servers are typically geographically separated and usually interact over a network. The client-server relationship is established by computer programs running on various computers and having a client-server relationship with each other.

[0134] Processor 1010 executes instructions related to a computer program. Processor 1010 may include hardware such as logic gates, adders, multipliers, and counters. Processor 1010 may further include a separate arithmetic logic unit (ALU) for performing arithmetic and logical operations.

[0135] The association process 700 is described as being performed to associate a specific user-operable portion of the user input system 106 with a specific actuator. In some implementations, the user-operable portion corresponds to a user input device of the user input system 106, such as a joystick. During process 700, each user input device of the user input system 106 is associated with a corresponding actuator of the actuator system 102. In some implementations, instead of associating a specific user input device with a specific actuator, a specific user-operable portion of the user input system 106 is associated with a specific actuator during the association process 700. For example, the user input system 106 may include a user input device having multiple different user-operable portions. If the user input device is a touchscreen device, the different user-operable portions correspond to user interface elements located on different parts of the touchscreen device. In this respect, each user interface element may be associated with a corresponding actuator during the association process 700.

[0136] Although manipulator 102 is described and shown as being mounted separately to different manipulators, such as an operating table, in mounting positions that are movable relative to each other, the association process described herein also applies to manipulators mounted to a common base. For example, refer to Figure 11The manipulator system 1101 includes manipulators 1102a, 1102b, 1102c, and 1102d (collectively referred to as manipulators 1102), each of which is mounted to a common base 1104. A joint 1106 can be driven to reorient all manipulators 1102. The base 1104 can be mounted on a movable trolley portion 1108. The movable trolley portion 1108 is supported above a floor surface, for example, by wheels. In this respect, the manipulator system 1101 can be easily moved around the environment.

[0137] Although a single user-operable part is described as being associated with a single manipulator during process 700, in some implementations, a group of user-operable parts are associated with a single manipulator during the association process. For example, the left or right joystick and the corresponding left or right foot pedal can be associated with the same manipulator. This allows the operator to use multiple user-operable parts to control different features of the manipulator, such as the manipulator's orientation, speed, etc. In some implementations, it is not necessary to associate each user-operable part in the group of user-operable parts individually; the operator can associate each user-operable part in the group of user-operable parts with the manipulator simultaneously. For example, if the group of user-operable parts includes a pedal and a joystick, the operator provides an association intention at operation 705 to move the virtual selector 121 and simultaneously initiate the association of both the pedal and the joystick with the manipulator.

[0138] While the display device 107 shown in the figure is described as providing visual feedback to operator 104, in some implementations, other indicating devices are operated to provide human-perceptible indications relating to equipment or the progress of an associated process. For example, the indicating device may provide human-perceptible tactile feedback, auditory feedback, or a combination thereof. If the indicating device provides tactile feedback, the tactile feedback may include vibration-tactile feedback, force feedback, or other forms of feedback associated with the user's touch. The indicating device may include, for example, a vibration generator. For example, in an implementation where the user input system 106 is manually operable, the indicating device may be coupled to the user input system 106 and may generate vibrations as tactile feedback to the operator 104 while the operator 104 is manually operating the user input system. If the indicating device provides auditory feedback, the indicating device includes, for example, an audio output device, such as a speaker. In this case, the indicating device may narrate the auditory feedback to the operator.

[0139] Although Figure 7BRegion 123a shown is described as a portion 124 of representation 122a surrounding device 116a (corresponding to the end effector of device 116a), but in some implementations, region 123a surrounds another portion of the representation. In some examples, region 123a surrounds a portion of representation 122a corresponding to a specific component of device 116a, such as a pivot of the end effector, a joint of the end effector, an axis of device 116a, or other parts of device 116a.

[0140] A pairing mode can be initiated in response to a specific event. For example, operations 701-703 illustrate a specific example of initiating a pairing mode in response to operation of user input system 106. In some implementations, the user input device of user input system 106 operated to initiate pairing mode corresponds to a user input device operable to initiate clutch mode, in which the actuator can be manually repositioned. In clutch mode, the actuator's braking system is disabled or the actuator's joint is released, allowing the actuator to be manually repositioned by the operator. In some examples, a pairing mode is also initiated when clutch mode is initiated.

[0141] In some implementations, the pairing mode can be initiated in response to an event unrelated to the operation of a manually operable user input device. For example, controller 110 can be configured to initiate pairing mode when system 100 is initialized. In some cases, system 100 includes an audio input system that detects voice commands issued by operator 104. Operator 104 can issue voice commands, and controller 110 thus initiates pairing mode. Alternatively or additionally, pairing mode can be initiated when a new operator accesses and operates user input system 106.

[0142] In some implementations, in a pairing mode, as described herein, the user-operable portion to be associated in process 700 is selected by operator 104. In other implementations, the user-operable portion is selected by controller 110. For example, controller 110 selects a specific user-operable portion that is in a non-associated state. If user input system 106 includes multiple user-operable portions in a non-associated state, controller 110 selects user-operable portions based on their relative association priority. For example, by default, controller 110 may start by selecting the leftmost user-operable portion and sequentially select user-operable portions to the right of the leftmost user-operable portion. This selection scheme can be intuitive for operator 104 and can reduce the number of operator steps required during process 700.

[0143] Although Figure 7BThe visual feedback shown and provided at operation 703 is described as indicating the association status of the manipulator, but in some implementations, the feedback provided at operation 703 indicates the association status of a user-operable portion. For example, display device 107 presents a graphical indicator indicating the available user-operable portion for association, and further presents an association status indicator to indicate the association status of the available user-operable portion.

[0144] Although the process has been described as the association between a user-operable portion and the manipulator 102, in some implementations, system 100 includes one or more sensors that detect motion and form an association based on the detected motion. For example, the process described herein is used for the association of the operator 104's hand. Reference Figure 12 In some implementations, system 100 includes an optical motion detection system 1200, which includes optical sensors 1202a and 1202b. The optical motion detection system 1200 is part of a user input system 106 and is operable to move the virtual selector 121 described herein. System 100 does not include a console 103 with a user output system, but its user output system includes a separate display device 1206 for presenting images of the instrument and the virtual selector 121. In this respect, the display device 1206 is similar to the display device 107 described herein.

[0145] Optical sensors 1202a and 1202b can provide a stereoscopic image of the operator 104 and can be used to detect the movement of the operator 104, particularly the movement of the operator's hands 1204a and 1204b. The movement of hands 1204a and 1204b can be used to control the movement of the manipulator 102 in follow mode. For example, hands 1204a and 1204b move in a pattern or sequence according to predefined gestures for controlling the system 100. Predefined gestures may include gestures for initiating a pairing mode, gestures for proposing an association between the hand and the manipulator, gestures for initiating follow mode, or other appropriate gestures for controlling the system 100. In some implementations, hands 1204a and 1204b are equipped with gloves detectable by the optical motion detection system 1200.

[0146] Furthermore, according to the association process described herein, operator 104 moves hands 1204a and 1204b to control the position of virtual selector 121. Optical motion detection system 1200 detects the movement of one of hands 1204a and 1204b, and controller 110 controls the position of virtual selector 121 based on the hand movement. When hands 1204a and 1204b move in a manner that satisfies association conditions, controller 110 establishes an association between hands 1204a and 1204b and the corresponding manipulator. For example, at operation 705, operator 104 moves hand 1204a or hand 1204b to control virtual selector 121 to satisfy association conditions. Hands 1204a and 1204b can then be used in follow mode to control the movement of the manipulator.

[0147] In some implementations, the user input system 106 is described as including a user-operable portion for controlling the orientation of the virtual selector 121, which is different from a user-operable portion that can be associated with a manipulator. In some examples, the user-operable portion for controlling the orientation of the virtual selector 121 includes a joystick, a touchscreen, or another manually operable user input device.

[0148] In some examples, reference Figure 13 The user-operable portion of the user input system 106 includes an eye-tracking system 1300 that detects the gaze movement of the operator 104 as they view the display device 107. Figure 13 In the example shown, eye-tracking system 1300 is part of console 103 and detects eye movement when the operator 104's eye is placed on eyepiece 1302 of console 103. In other examples, eye-tracking system 1300 is part of a non-console input system (such as system 113). During operation, operator 104 can operate eye-tracking system 1300 by shifting operator 104's gaze. When the eye is positioned on eyepiece to view display device 107, eye-tracking system 1300 detects movement of the eye gaze and generates one or more signals indicating gaze movement. One or more signals may correspond to a set of signals for controlling the orientation of virtual selector 121. Controller 110 can then operate display device 107 based on this set of signals to move virtual selector 121 or reposition it relative to an image presented on display device 107.

[0149] As described herein, the manipulator is associated with a user-operable portion, enabling the manipulator to move in response to certain operations of the user-operable portion. Therefore, in some implementations, the associated user-operable portion can be used to control the movement of the manipulator. Furthermore, in some implementations, the associated user-operable portion is operable to control other functions of the manipulator or the apparatus mounted to the manipulator, alternatively or additionally, to control the movement of the manipulator. In this regard, at operation 603, when the follow mode is activated, the manipulator does not necessarily move in response to operations of the associated user-operable portion, but rather receives signals to perform a specific function or to cause the apparatus to perform a specific function. For example, in some implementations where the apparatus is an image capturing device, the associated user-operable portion is operable to control the image capturing functions of the image capturing device, such as zoom settings, illumination settings, shutter speed settings, or other image capturing settings. As another example, in some implementations where the apparatus is a suction or rinsing device, the associated user-operable portion is operable to control the application of suction or rinsing. In some implementations where the apparatus is an image capturing device, the associated user input device is operable to control the image capturing device to capture images. In some implementations where the device is a cauterization device or other energy-applying device, the associated user input device is operable to control the energy-applying device to apply energy to the tissue.

[0150] In some implementations, in a paired mode, multiple manipulators are associated with a single user-operable portion of the user input system 106. For example, two or more manipulators may be associated with a single user-operable portion of the user-operable portion 108. When a single user-operable portion is associated with multiple manipulators, the user-operable portion is operable to generate movement for each manipulator. For example, in some implementations, if operator 104 wishes to move a combination of multiple manipulators or their associated instruments to a different workspace, operator 104 may operate the user-operable portion to move each manipulator to the vicinity of that different workspace. In some implementations, operator 104 may associate all manipulators to be moved with a single user-operable portion and operate the single user-operable portion to move multiple manipulators as a group to the vicinity of different workspaces, rather than moving each manipulator one by one to reach different workspaces.

[0151] As another example, in some implementations, multiple manipulators may be associated with a single user-operable section, and the single user-operable section controls only one manipulator at a time. In some implementations, the operator selects which manipulator will be controlled by operating the single user-operable section via appropriate methods (e.g., pressing a button, turning a dial, clicking a pedal, voice command, etc.). In some implementations, the operator operates buttons or pedals to cycle through the various manipulators until the manipulator to be controlled becomes active.

[0152] Alternatively or additionally, two or more user-operable parts may be associated with a single manipulator. For example, in some implementations, one user-operable part associated with the manipulator is operable to move the manipulator, while another user-operable part associated with the manipulator is operable to control the non-moving functions of the manipulator or the functions of an apparatus mounted on the manipulator. In some implementations, each of the two or more associated user-operable parts is operable to control different degrees of freedom or different sets of degrees of freedom of the manipulator. For example, in some implementations, one user-operable part is manually operable to control the pitch, yaw, and roll movements of the manipulator, while another user-operable part is manually operable to control the movement of the apparatus along the insertion axis relative to the manipulator or to control the actuation of the end effector of the apparatus. As yet another example, in some implementations, multiple user-operable parts are used to implement multi-handed input. For example, the orientation, separation distance, direction of movement, and speed of movement of the user-operable parts relative to each other or relative to a reference can be used to control the manipulator or an apparatus supported by the manipulator.

[0153] As a specific example, in one implementation, two user input devices are associated with a single manipulator holding an imaging system (such as a camera). An operator holding the user input devices in each hand can control the imaging system using combined hand inputs that simulate manipulation of a workpiece relative to the imaging system. For example, in a camera implementation, combined movement of the two input devices away from the operator moves the camera away or zooms it out, as if the workpiece had been moved away. As another example, in a camera implementation, combined movement of the two input devices around a common center rotates the camera's field of view, as if the workpiece had been rotated. As yet another example, in a camera implementation, an increase in the separation distance between the user input devices causes the camera to zoom out, and a decrease in the separation distance causes the camera to zoom in.

[0154] In some implementations, controller 110 is configured to unassociate one or more user input devices 108 with one or more manipulators 102 in response to user input or a system event. As an example, controller 110 may be configured to unassociate a pair of manipulators and user input devices in response to receiving a signal instructing a user to request unassociation of a first manipulator and a user input device. As another example, controller may be configured to unassociate all manipulators associated with a user input device or all user input devices associated with a manipulator in response to receiving a signal instructing a user to request unassociation of such user input devices or such manipulators. In some implementations, user input system 106 includes unassociating user-operable portions to initiate the unassociation of user-operable portions with manipulators 102. For example, each of the user input devices 108 may include an unassociation control or feature. As another example, for each user-operable portion, unassociating a corresponding one of the user-operable portions may be operated to unassociate the user-operable portion with the manipulator. Additionally, in some cases, unassociating a user-operable portion may also initiate a pairing mode.

[0155] Therefore, other implementations are within the scope of the claims.

Claims

1. A computer-assisted medical system comprising: a plurality of manipulators; a user input system; a user output system including a display device; and a controller configured to execute instructions to perform operations comprising: in a pairing mode and in response to a first set of signals generated by the user input system, causing a virtual selector displayed on the display device to move relative to an image displayed on the display device, wherein the image represents a position of a first instrument supported by a first manipulator of the plurality of manipulators, in the pairing mode, based on the represented position of the first instrument and movement of the virtual selector relative to the represented position of the first instrument, associating the first manipulator with a portion of the user input system, and in a following mode, in response to user operation of the portion of the user input system, controlling motion of the first instrument in accordance with a second set of signals generated by the user input system.

2. The computer-assisted medical system of claim 1, wherein the operations further comprise: initiating the pairing mode prior to associating the first manipulator with the portion of the user input system.

3. The computer-assisted medical system of claim 2, wherein initiating the pairing mode comprises: initiating the pairing mode upon initialization of the computer-assisted medical system.

4. The computer-assisted medical system of claim 2, wherein initiating the pairing mode comprises: initiating the pairing mode in response to receiving a signal from the user input system indicating a request to initiate the pairing mode.

5. The computer-assisted medical system of any of claims 1-4, wherein the operations further comprise: based on the represented position of the first instrument and movement of the virtual selector, providing feedback indicating a proposed association between the portion of the user input system and the first manipulator, and then in response to receiving a signal indicating user confirmation of the proposed association, associating the portion of the user input system with the first manipulator. the operations further comprise causing the user output system to generate a human- perceptible indication of an association status of the first manipulator, the human-perceptible indication including at least one of: visual, audible, or tactile feedback.

6. The computer-assisted medical system of any of claims 1 to 4, wherein, 7. The computer-assisted medical system of claim 6, wherein the human-perceptible indication includes a first indicator proximate to the represented position of the first instrument.

8. The computer-assisted medical system of claim 6, wherein the association status corresponds to an associated state of the first manipulator.

9. The computer-assisted medical system of claim 6, wherein the association status corresponds to an unassociated state of the first manipulator.

10. The computer-assisted medical system of any of claims 1-4, wherein: the operations further comprise causing the display device to present a first selectable indicator proximate to the represented position of the first instrument, and the operations further comprise causing the display device to present a second selectable indicator proximate to the represented position of the first instrument. Associating the first manipulator with the portion of the user input system includes associating the first manipulator with the portion of the user input system in response to the virtual selector moving to the first selectable indicator.

11. The computer-assisted medical system of any of claims 1 to 4, wherein associating the first manipulator with the portion of the user input system includes: associating the first manipulator with the portion of the user input system in response to the virtual selector overlapping an area in the image defined by the represented position of the first instrument.

12. The computer-assisted medical system of any of claims 1 to 4, wherein associating the first manipulator with the portion of the user input system includes: associating the first manipulator with the portion of the user input system in response to the virtual selector being within a predetermined distance from the represented position of the first instrument.

13. The computer-assisted medical system of any one of Claims 1 to 4, wherein the operations further comprise: causing the display device to present an enlarged representation of the first instrument when the virtual selector is positioned proximate to the represented position of the first instrument.

14. The computer-assisted medical system of any of claims 1 to 4, wherein associating the first manipulator with the portion of the user input system includes: associating the first manipulator with the portion of the user input system in response to the virtual selector moving toward the represented position of the first instrument.

15. The computer-assisted medical system of any of claims 1 to 4, further comprising a console including the user input system and the user output system, the console including a plurality of manually operable user input devices, wherein the portion of the user input system includes a first user input device of the plurality of manually operable user input devices.

16. The computer-assisted medical system of any of claims 1 to 4, wherein the user input system includes an eye tracking system configured to detect gaze motion of the user relative to the image, and generate the first set of signals in response to the detected gaze motion.

17. The computer-assisted medical system of any of claims 1 to 4, wherein the operations further include: initiating the follow mode after an orientation of the portion of the user input system aligns with an orientation of a representation of the first instrument in the image.

18. The computer-assisted medical system of claim 17, wherein the operations further include: generating motion of the portion of the user input system to align the orientation of the portion of the user input system with the orientation of the representation of the first instrument in the image.

19. The computer-assisted medical system of claim 17, wherein the operations further include: manually positioning the portion of the user input system to align the orientation of the portion of the user input system relative to the orientation of the representation of the first instrument in the image.

20. The computer-assisted medical system of any of Claims 1-4, wherein: the user input system comprises a plurality of user input devices, a first user input device of the plurality of user input devices is operable to generate the first set of signals, and the operations further comprise: associating the first manipulator with the portion of the user input system based on a position or orientation of a representation of the first instrument in the image.

21. The computer-assisted medical system of any of Claims 1-4, wherein associating the first manipulator with the portion of the user input system comprises: associating the first manipulator with the portion of the user input system in response to a user confirmation signal.

22. The computer-assisted medical system of any of Claims 1-4, wherein associating the first manipulator with the portion of the user input system comprises associating the first manipulator with the portion of the user input system only if another manipulator of the plurality of manipulators is not associated with the portion of the user input system.

23. The computer-assisted medical system of any of Claims 1-4, wherein associating the first manipulator with the portion of the user input system comprises: associating the first manipulator with the portion of the user input system only if the portion of the user input system is in an unassociated state.

24. The computer-assisted medical system of any of Claims 1-4, wherein associating the first manipulator with the portion of the user input system comprises: associating the first manipulator with the portion of the user input system only if the first manipulator is in an unassociated state.

25. The computer-assisted medical system of any of Claims 1-4, wherein the operations further comprise: disassociating the first manipulator from the portion of the user input system in response to receiving a signal indicating a user request to disassociate the first manipulator or the portion of the user input system.

26. The computer-assisted medical system of any of Claims 1-4, wherein the operations further comprise: visually indicating that the first manipulator is associated with the portion of the user input system after associating the first manipulator with the portion of the user input system.

27. The computer-assisted medical system of Claim 26, wherein visually indicating that the first manipulator is associated with the portion of the user input system comprises: presenting a textual element indicating association with the portion of the user input system.

28. The computer-assisted medical system of Claim 27, wherein the textual element is a number.

29. One or more non-transitory computer-readable media storing instructions that are executable by a processing device and that, upon such execution, cause the processing device to perform operations comprising: causing a display device to present an image representing a position of a first instrument supported by a first manipulator of a plurality of manipulators, and a virtual selector movable relative to the image in response to a first set of signals generated by a user input system; and in a pairing mode, associating the first manipulator with a portion of the user input system based on a represented position of the first instrument and movement of the virtual selector relative to the represented position of the first instrument, and in a following mode, controlling motion of the first instrument in accordance with a second set of signals generated by the user input system in response to user operation of the portion of the user input system.

30. The one or more non-transitory computer-readable media of claim 29, wherein the operations further comprise: initiating the pairing mode prior to associating the first manipulator with the portion of the user input system.

31. The one or more non-transitory computer-readable media of claim 30, wherein initiating the pairing mode comprises: initiating the pairing mode upon initialization of a computer-assisted medical system.

32. The one or more non-transitory computer-readable media of claim 30, wherein initiating the pairing mode comprises: initiating the pairing mode in response to receiving a signal from the user input system indicating a request to initiate the pairing mode.

33. The one or more non-transitory computer-readable media of any of claims 29-32, wherein the operations further comprise: based on the represented position of the first instrument and movement of the virtual selector, providing feedback indicating a proposed association between the portion of the user input system and the first manipulator, and then in response to receiving a signal indicating user confirmation of the proposed association, associating the portion of the user input system with the first manipulator.

34. The one or more non-transitory computer-readable media of any of claims 29-32, wherein the operations further comprise generating a human-perceptible indication of an association status of the first manipulator, the human-perceptible indication comprising at least one of visual, audible, or tactile feedback.

35. The one or more non-transitory computer-readable media of claim 34, wherein the operations further comprise generating the human-perceptible indication by causing the display device to present a first indicator proximate to the represented position of the first instrument.

36. The one or more non-transitory computer-readable media of claim 34, wherein the association status corresponds to an association status of the first manipulator.

37. The one or more non-transitory computer-readable media of any of claims 29-32, wherein the operations further comprise: causing the display device to present a first selectable indicator proximate the represented position of the first instrument, wherein associating the first manipulator with the portion of the user input system comprises associating the first manipulator with the portion of the user input system in response to the virtual selector moving to the first selectable indicator.

38. The one or more non-transitory computer-readable media of any of claims 29-32, wherein associating the first manipulator with the portion of the user input system comprises: associating the first manipulator with the portion of the user input system in response to the virtual selector overlapping an area in the presented image defined by the represented position of the first instrument.

39. The one or more non-transitory computer-readable media of any of claims 29-32, wherein associating the first manipulator with the portion of the user input system comprises: associating the first manipulator with the portion of the user input system in response to the virtual selector being within a predetermined distance from the represented position of the first instrument.

40. The one or more non-transitory computer-readable media of any one of claims 29-32, wherein the operations further comprise: causing the display device to present an enlarged representation of the first instrument when the virtual selector is positioned proximate the represented position of the first instrument.

41. The one or more non-transitory computer-readable media of any one of claims 29-32, wherein the operations further comprise: associating the first manipulator with the portion of the user input system in response to the virtual selector moving toward the represented position of the first instrument.

42. The one or more non-transitory computer-readable media of any of claims 29-32, wherein the operations further comprise: detecting gaze movement of the user relative to the presented image, and generating the first set of signals in response to the detected gaze movement.

43. The one or more non-transitory computer-readable media of any of claims 29-32, wherein the operations further comprise: initiating the follow mode after an orientation of the portion of the user input system aligns with an orientation of a representation of the first instrument in the presented image.

44. The one or more non-transitory computer-readable media of claim 43, wherein the operations further comprise: generating movement of the portion of the user input system to align the orientation of the portion of the user input system with the orientation of the representation of the first instrument in the presented image.

45. The one or more non-transitory computer-readable media of claim 43, wherein the operations further comprise: directing manual positioning of the portion of the user input system to align the orientation of the portion of the user input system relative to the orientation of the representation of the first instrument in the presented image.

46. The one or more non-transitory computer-readable media of any of claims 29-32, wherein the operations further comprise: based on an orientation or an orientation of a representation of the first instrument in the presented image, directing association of the first manipulator with the portion of the user input system.

47. The one or more non-transitory computer-readable media of any of claims 29-32, wherein associating the first manipulator with the portion of the user input system comprises: associating the first manipulator with the portion of the user input system in response to a user confirmation signal.

48. The one or more non-transitory computer-readable media of any of claims 29-32, wherein associating the first manipulator with the portion of the user input system comprises associating the first manipulator with the portion of the user input system only if another manipulator of the plurality of manipulators is not associated with the portion of the user input system.

49. The one or more non-transitory computer-readable media of any of claims 29-32, wherein associating the first manipulator with the portion of the user input system comprises: associating the first manipulator with the portion of the user input system only if the portion of the user input system is in an unassociated state.

50. The one or more non-transitory computer-readable media of any of claims 29-32, wherein associating the first manipulator with the portion of the user input system comprises: associating the first manipulator with the portion of the user input system only if the first manipulator is in an unassociated state.

51. The one or more non-transitory computer-readable media of any of claims 29-32, wherein the operations further comprise: disassociating the first manipulator from the portion of the user input system in response to receiving a signal indicating a user request to disassociate the first manipulator or the portion of the user input system.

52. The one or more non-transitory computer-readable media of any of claims 29-32, wherein the operations further comprise: visually indicating that the first manipulator is associated with the portion of the user input system after associating the first manipulator with the portion of the user input system.

53. The one or more non-transitory computer-readable media of claim 34, wherein the associated state corresponds to an unassociated state of the first manipulator.

Images