Translation locking of out-of-sight control points in computer-aided systems

Abstract

A computer-assisted device includes: a repositionable structure including one or more joints coupled to an end effector; and a control unit, where the control unit is configured to: determine that a control point associated with the end effector is not within a field of view of an imaging device that captures an image of the workplace; receiving an actuation command for translating the control point if the control point is not within the field of view of the imaging device; generating, based on the actuation command, a modification command that does not translate the control point; and actuating one or more joints of the end effector or repositionable structure based on the modification command.

Description

[0001] Related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 478,565, filed on January 5, 2023, entitled “TRANSLATIONAL LOCKING OF AN OUT-OF-VIEW CONTROLPOINT IN A COMPUTER-ASSISTED SYSTEM,” which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to the operation of computer-assisted systems having repositionable structures (eg, articulated arms), and more particularly to limiting translational motion of portions of such repositionable structures that are not within a field of view of an imaging device. Background Art

[0004] Computer-assisted electronic systems are being used more and more frequently. This is particularly true in industrial, entertainment, educational and other environments. As a medical example, today's medical facilities find a large number of electronic systems in operating rooms, interventional rooms, intensive care units, emergency rooms, etc. Many of these electronic systems may be capable of autonomous or semi-autonomous movement. It is also known that personnel use one or more input devices located at the user control system to control the movement and / or operation of the electronic system. As a specific example, minimally invasive robotic telesurgery systems allow surgeons to perform surgery on patients from the bedside or from a remote location. Telesurgery generally refers to surgery performed using a surgical system in which the surgeon uses some form of remote control, such as a servo mechanism, to manipulate the movement of the surgical instruments rather than directly holding and moving the instruments by hand.

[0005] When a computer-assisted system is used to perform a task at a workspace (e.g., the internal anatomy of a patient in a medical example), one or more instruments of the computer-assisted system are positioned within a workspace that is created, for example, by blowing gas into an area of the patient's anatomy surrounding the workspace. An imaging device, such as an endoscope, is typically inserted into the workspace. The imaging device is positioned and oriented so that the relevant portion of the one or more instruments is within the imaging device's field of view. This allows the operator of the computer-assisted system to observe and monitor the one or more instruments while performing the procedure in the workspace. Therefore, coordinated use of the imaging device and the one or more instruments is important.

[0006] Therefore, improved techniques for controlling the motion of instruments of computer-assisted systems using imaging devices for viewing are desired. Summary of the Invention

[0007] In accordance with some embodiments, a computer-assisted system includes: a repositionable structure comprising one or more joints coupled to an end effector; and a control unit. In some embodiments, the control unit, when coupled to the repositionable structure, is configured to: determine that a control point associated with the end effector is not within a field of view of an imaging device capturing an image of a worksite; if the control point is not within the field of view of the imaging device, receive an actuation command to translate the control point a specified distance; based on the actuation command, generate a modification command to prevent the control point from translating the specified distance; and actuate the end effector or the one or more joints of the repositionable structure based on the modification command.

[0008] Consistent with some embodiments, a method for operating a computer-assisted device including a repositionable structure comprising one or more joints coupled to an end actuator includes: determining, by a control unit, that a control point associated with the end actuator is not within a field of view of an imaging device that captures an image of a workplace; receiving, by the control unit, an actuation command for translating the control point a specified distance when the control point is not within the field of view of the imaging device; generating, by the control unit, a modification command based on the actuation command to prevent the control point from translating the specified distance; and actuating, by the control unit, the end actuator or one or more joints of the repositionable structure using one or more motors, solenoids, servo mechanisms, or actuators based on the modification command.

[0009] In some embodiments, a non-transitory machine-readable medium includes a plurality of machine-readable instructions that, when executed by one or more processors associated with a computer-assisted device, are adapted to cause the one or more processors to perform the methods disclosed herein.

[0010] In some cases, the ability of an imaging device to assist an operator in controlling one or more instruments in a workspace is limited. For example, longer instruments often extend beyond the field of view of the imaging device, particularly when an operator (e.g., a surgeon in a medical example) is magnifying the field of view of the imaging device to confirm that material has been correctly captured and / or that a particular instrument is precisely positioned in the proper location and orientation. When an operator commands a viewed portion of an instrument to move in such circumstances, translation or other movement of the unviewed portion of the instrument often occurs to effectuate the movement commanded by the operator. Because the unviewed portions of the instruments are moving beyond the field of view of the imaging device, it is not possible for the operator to fully monitor their movement. In such cases, it is helpful to alter the actuation commands for one or more instruments to limit the movement of the portions of the one or more instruments that are not within the field of view of the imaging device.

[0011] The foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this regard, additional aspects, features and advantages of the present disclosure will be apparent to those skilled in the art based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a simplified diagram of a computer-assisted system according to some embodiments.

[0013] Figure 2 is a simplified diagram showing a side view of an end effector and imaging device of a computer-assisted system in a workspace according to some embodiments.

[0014] Figure 3 is a simplified diagram illustrating an instrument, end effector, and associated articulating wrist joint according to some embodiments.

[0015] Figure 4 According to some embodiments Figure 2 Simplified perspective view of the distal end of the end effector and the articulated wrist joint.

[0016] Figure 5 is a simplified diagram showing the portion of an end effector extending outside the field of view of an imaging device according to some embodiments.

[0017] Figure 6 is a simplified diagram showing an end effector disposed within a field of view and coupled to a control point outside of the field of view, according to some embodiments.

[0018] Figure 7 is a simplified diagram of an exemplary method for translation locking of out-of-line-of-sight control points in accordance with some implementations.

[0019] In the figures, elements with the same reference numerals have the same or similar functions. DETAILED DESCRIPTION

[0020] The present specification and drawings, which illustrate aspects, embodiments, or modules of the invention, should not be construed as limiting—the claims define the protected invention. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of the present specification and claims. In some cases, well-known circuits, structures, or techniques are not shown or described in detail to avoid obscuring the present invention. The same numerals in two or more figures represent the same or similar elements.

[0021] In the following description, the specific details of some embodiments consistent with the present disclosure are set forth. However, it will be apparent to those skilled in the art that some embodiments may be put into practice without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative and not restrictive. Those skilled in the art may implement other elements within the scope and spirit of the present disclosure, although other elements are not specifically described herein. In addition, to avoid unnecessary repetition, one or more features shown and described in association with an embodiment may be incorporated into other embodiments, unless specifically described otherwise or if one or more features will render the embodiment inoperative. The term "comprising" means including but not limited to, and each of the one or more individual items included should be considered optional, unless otherwise stated. Similarly, the term "may" indicates that an item is optional.

[0022] In addition, the terms in this specification are not intended to limit the present invention. For example, spatial relative terms, such as "under ... ", "below ... ", "bottom", "above ... ", "top", "near side", "far side" etc., can be used to describe the relationship between an element or feature and another element or feature as shown in the figure. In addition to the position and orientation shown in the figure, these spatial relative terms are intended to cover the different positioning (i.e., position) and orientation (i.e., rotation placement) of elements or their operations. For example, if the content of one of the figures is flipped, the element described as "below" or "below" other elements or features will then be "above" or "on" other elements or features. Therefore, the exemplary term "under ... " can include both the position and orientation of above and below. Device can be oriented in other ways (rotated 90 degrees or in other orientations), and the spatial relative descriptors used in this article are interpreted accordingly. Similarly, the description of the motion along and around various axes includes various special element positions and orientations. In addition, unless the context otherwise indicates, the singular "one", "an" and "the" are also intended to include plural forms. Furthermore, the terms "comprising," "including," "having," etc. specify the presence of the described features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be directly electrically or mechanically coupled, or they may be indirectly coupled via one or more intermediate components.

[0023] Elements described in detail with reference to one embodiment, implementation, or module may be included in other embodiments, implementations, or modules that are not specifically shown or described, as long as feasible. For example, if an element is described in detail with reference to one embodiment without describing the element with reference to a second embodiment, the element may still be claimed to be included in the second embodiment. Therefore, in the following description, to avoid unnecessary repetition, one or more elements shown and described in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects, unless specifically described otherwise, unless one or more elements would render the embodiment or implementation inoperative or unless two or more of the elements provide conflicting functions.

[0024] In some instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0025] This disclosure describes various elements (e.g., systems and devices, and parts of systems and devices) by way of examples in three-dimensional space. In such examples, the term "positioning" refers to the location of an element or a part of an element in three-dimensional space ( For example , three translational degrees of freedom along Cartesian x-, y-, and z-coordinates). Furthermore, in such examples, the term "orientation" refers to the rotational placement of an element or a portion of an element (three rotational degrees of freedom— For example , roll, pitch, and yaw). Other examples may include other dimensional spaces, such as two-dimensional space. As used herein, the term "pose" refers to the position, orientation, or combination of position and orientation of an element or a portion of an element. As used herein, and for ( For example , a computer-assisted system or a repositionable structure, etc., an element or part of an element of a structure or assembly, the term "proximal" in a kinematic series refers to a direction toward the base of the kinematic series, and the term "distal" refers to a direction away from the base along the kinematic series.

[0026] Aspects of the present disclosure are described with reference to electronic systems, computer-assisted devices, and robotic devices, which may include systems and devices that are teleoperated, telecontrolled, autonomous, semi-autonomous, manually operated, and the like. Example computer-assisted systems include systems that include robots or robotic devices. Additionally, aspects of the present disclosure are described with reference to embodiments using medical systems, such as the da However, those skilled in the art will appreciate that the inventive aspects disclosed herein may be implemented and carried out in a variety of ways, including robotic and, if applicable, non-robotic embodiments. The embodiments described in the surgical system are merely exemplary and should not be considered to limit the scope of the invention disclosed herein. For example, the techniques described with reference to surgical instruments and surgical methods can be used in other situations. Therefore, the instruments, systems and methods described herein can be used for humans, animals, parts of human or animal anatomical structures, industrial systems, general robots or remote operating systems. As another example, the instruments, systems and methods described herein can be used for non-medical purposes, including industrial uses, general robotic uses, sensing or manipulating non-tissue artifacts, cosmetic improvements, imaging of human or animal anatomical structures, collecting data from human or animal anatomical structures, setting up or shutting down systems, training medical or non-medical personnel, etc. Additional example applications include processes for tissue removed from human or animal anatomical structures (returned or not returned to human or animal anatomical structures) and processes for human or animal corpses. In addition, these technologies can also be used for medical treatment or diagnostic processes that include or do not include surgical aspects.

[0027] Figure 1 is a simplified diagram of a computer-assisted system 100 according to some embodiments. Figure 1 As shown, the computer-assisted system 100 includes, but is not limited to, an apparatus 110 having one or more movable or articulating arms 120. Each of the one or more articulating arms 120 is a repositionable structure that supports one or more instruments or end effectors 122. In some examples, the apparatus 110 is consistent with a computer-assisted surgical apparatus. The one or more articulating arms 120 provide support for one or more instruments, surgical instruments, imaging devices, etc. mounted to the distal end of at least one of the articulating arms 120. The apparatus 110 can be further coupled to an operator workstation 190, which can include one or more master controls for operating the apparatus 110, the one or more articulating arms 120, and / or the end effectors. In some embodiments, the apparatus 110 and the operator workstation correspond to a computer-assisted surgical apparatus commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Surgical System. In some embodiments, computer-assisted surgical devices having other configurations, fewer or more articulated arms, etc., may optionally be used with the computer-assisted system 100.

[0028] Device 110 is coupled to control unit 130 via an interface. The interface may include one or more wireless links, cables, connectors, and / or buses, and may also include one or more networks having one or more network switches and / or routing devices. Control unit 130 includes, but is not limited to, a processor 140 coupled to memory 150. Operation of control unit 130 is controlled by processor 140. Although control unit 130 is shown as having only one processor 140, it should be understood that processor 140 may represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc. within control unit 130. Control unit 130 may be implemented as a separate subsystem and / or board added to a computing device, or as a virtual machine. In some embodiments, the control unit is included as part of operator workstation 190 and / or operates separately from, but in coordination with, operator workstation 190. Some examples of a control unit (e.g., control unit 130) include non-transitory, tangible, machine-readable media that includes executable code that, when executed by one or more processors (e.g., processor 140), causes the one or more processors to perform the processing of method 700.

[0029] The memory 150 is used to store software executed by the control unit 130 and / or one or more data structures used during operation of the control unit 130. The memory 150 may include one or more types of machine-readable media. Some common forms of machine-readable media may include a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, and / or any other medium from which a processor or computer is suitable for reading.

[0030] As shown, the memory 150 includes, but is not limited to, a motion control application 160 that supports autonomous and / or semi-autonomous control of the device 110. The motion control application 160 may include one or more application programming interfaces (APIs) for receiving position, motion, and / or other sensor information from the device 110, exchanging position, motion, and / or collision avoidance information with other control units regarding other devices (e.g., an operating table and / or imaging devices), and / or planning and / or assisting in planning the motion of the device 110, the articulated arm 120, and / or the end effector 122 of the device 110. Although the motion control application 160 is depicted as a software application, the motion control application 160 may be implemented using hardware, software, and / or a combination of hardware and software.

[0031] although Figure 1 The example of the computer-assisted system 100 shown in FIG includes only one device 110 having two articulation arms 120, but one of ordinary skill in the art will appreciate that the computer-assisted system 100 may include any number of devices having articulation arms and / or end effectors of similar and / or different designs than the device 110. In some examples, each of the devices may include fewer or more articulation arms and / or end effectors.

[0032] The computer-assisted system 100 also includes an operating table 170. Like the one or more articulation arms 120, the operating table 170 supports articulated movement of a tabletop 180 relative to the base of the operating table 170. In some examples, the articulated movement of the tabletop 180 includes supporting changes in height, tilt, slide, Trendelenburg orientation, etc. of the tabletop 180. Although not shown, the operating table 170 may include one or more control inputs, such as an operating table command unit, for controlling the position and / or orientation of the tabletop 180.

[0033] Operating table 170 is also coupled to control unit 130 via a corresponding interface. The interface may include one or more wireless links, cables, connectors, and / or buses, and may also include one or more networks having one or more network switches and / or routing devices. In some embodiments, operating table 170 may be coupled to a control unit different from control unit 130.

[0034] The control unit 130 can also be coupled to an operator workstation 190 via an interface. The operator workstation 190 can be used by an operator (e.g., a surgeon) to control the movement and / or operation of the joint arm 120 and the end effector 122. To support the operation of the joint arm 120 and the end effector 122, the operator workstation 190 includes, but is not limited to, a display system 192 for displaying an image of at least a portion of one or more of the joint arm 120 and / or the end effector 122. For example, the display system 192 can be used when it is impractical and / or impossible for the operator to see the joint arm 120 and / or the end effector 122 while using them. In some embodiments, the display system 192 displays video images from a video capture device (e.g., an endoscope) that is controlled by one of the joint arms 120 or a third joint arm (not shown).

[0035] The operator workstation 190 may also include a console workspace having one or more input controls 195 (or "master controls 195") that may be used to operate the device 110, the articulated arm 120, and / or the end effector 122. Each of the input controls 195 may be coupled to the distal end of the associated articulated arm 120 such that movement of the input control 195 may be detected by the operator workstation 190 and transmitted to the control unit 130. To provide improved ergonomics, the console workspace may also include one or more resting pieces, such as armrests 197, on which the operator may rest their arms while manipulating the input controls 195. In some examples, the display system 192 and the input controls 195 may be used by the operator to remotely operate the articulated arm 120 and / or the end effector 122 mounted thereon. In some examples, the input controls 195 include any type of device that can be manually operated by a human user, For example Joysticks, trackballs, button clusters, and / or other types of tactile devices typically equipped with multiple degrees of freedom. Position, force, and / or tactile feedback devices (not shown) can be used to transmit position, force, and / or tactile sensations from the instrument back to the operator's hand via input controls 195. In some embodiments, the device 110, operator workstation 190, and control unit 130 correspond to da Surgical system.

[0036] In some embodiments, other configurations and / or architectures are used with computer-assisted system 100. In some examples, control unit 130 is included as part of operator workstation 190 and / or apparatus 110. In some embodiments, computer-assisted system 100 is present in an operating room and / or interventional room. In some embodiments, an additional workstation 190 is present for controlling an additional arm that can be attached to apparatus 110. Additionally, in some embodiments, workstation 190 can have controls for controlling operating table 170.

[0037] Figure 2 is a simplified diagram showing a side view of the end effector 220 and imaging device 230 of the computer-assisted system 200 in the workplace 202 according to some embodiments. For example, the computer-assisted system 200 may be used with Figure 1 The workspace 202 indicates that one or more end effectors 220 are based on, for example, the computer-aided system 100 of FIG. Figure 1 An operator of the workstation 190 in the embodiment enters an area where various tasks are performed on the material 204. In the medical example, the workspace 202 material 204 is a cavity 206 and a portion of the internal patient anatomy 204, for example, created by blowing gas into the area surrounding the internal patient anatomy 204. Figure 1 In the illustrated embodiment, the end effector 220 and the imaging device 230 are positioned within the lumen 206 , although in other embodiments, additional end effectors 220 and / or imaging devices 230 may be positioned within the lumen 206 .

[0038] The imaging device 230 can be any camera or optical device that can be mounted on the articulated arm 232 of the computer-assisted system 200 and used in the workplace 202. For example, in some embodiments, the imaging device 230 can include an endoscopic camera or other minimally invasive surgical imaging device having a field of view 236 within the workplace 202. Figure 2 In the example shown, the field of view 236 is depicted two-dimensionally as a triangular area within the workplace 202, but in reality the field of view 236 is typically a three-dimensional area, such as a pyramid, cone, or frustum. In some embodiments, the imaging device 230 is coupled to the articulated arm 232 via a multi-axis wrist joint 234 that enables the imaging device 230 to be oriented in multiple directions within the workplace 202. Thus, in such embodiments, the imaging device 230 can be used to provide an image of the end effector 220 and the workplace 202, such as the material 204 ( For example , in the medical example, direct visual observation of internal patient anatomy 204) and / or objects in the workplace 202. In other embodiments, the imaging device 230 may be configured to include a plurality of imaging devices, such as a plurality of imaging devices, and a plurality of imaging devices. Figure 1 and Figure 2 Any technically feasible joint and link configuration other than that shown is coupled to engagement arm 232 .

[0039] The end effector 220 can be any instrument, tool, or other device that can be mounted on the articulation arm 222 of the computer-assisted system 200 and used in the workplace 202. For example, in some embodiments, the end effector 220 can include a specific minimally invasive surgical instrument, such as a surgical stapler, a suction irrigator, an electrocautery device for delivering energy, a clamp, a cutting mechanism, etc. In such embodiments, the end effector 220 is used to perform one or more operations on the material 204. In some embodiments, the end effector 220 is coupled to the articulation arm 222 via an articulation joint (e.g., an articulation wrist 224). Figure 1 In the illustrated embodiment, the end effector 220 is coupled to a link 226 via a joint wrist 224, and the link 226 is coupled to another link of the joint arm 222 extending outside the workplace 202 via a joint 228. In other embodiments, the end effector 220 can be coupled to the joint arm 222 by any other technically feasible joint and link configuration. Figure 3 and Figure 4 One embodiment of the end effector 220 and articulation wrist 224 is described.

[0040] Figure 3 2 is a simplified diagram illustrating an instrument 300 including an end effector 220 and an articulation wrist 224 according to some embodiments. Figure 3 The directions "proximal" and "distal" depicted and used herein help describe the relative orientation and position of the components of the end effector 220. Distal generally refers to the distance along the kinematic chain from the base of the computer-assisted system 200 (e.g., Figure 1 The term "proximal" generally refers to an element that is distal to the computer-assisted device 110 in the embodiment of the present invention and / or closer to the work area during the intended operational use of the end effector 220. Proximal generally refers to an element that is closer to the base of the computer-assisted system 200 and / or one of the articulated arms of the computer-assisted system 200 along the kinematic chain.

[0041] like Figure 3 As shown, the instrument 300 includes, but is not limited to, an end effector 220 and an engagement wrist 224 that couples the end effector 220 to an elongated shaft 310. Thus, the shaft 310 couples the end effector 220 and engagement wrist 224 at the distal end of the shaft 310 to an engagement arm and / or computer-assisted device, such as a drive system 340, at the proximal end of the shaft 310. Depending on the particular procedure for which the end effector 220 is being used, the shaft 310 may be inserted through an opening ( For example , body wall incision, natural orifice, etc.) to place the end effector 220 at a remote surgical site within the patient's anatomy, such as Figure 2 Near the workplace 202. Figure 2 In the illustrated embodiment, the end effector 220 is generally consistent with a double-jaw gripper-type end effector. However, one of ordinary skill in the art will appreciate that the end effector 220 can be configured as any other suitable tool, device, surgical instrument, etc. that can be used by the computer-assisted system 200.

[0042] In some embodiments, the end effector 220 relies on multiple degrees of freedom (DOF) during operation. Depending on the configuration of the end effector 220, the engagement arm 222, and / or the specific drive system 340 to which the end effector 220 is coupled, various DOFs are possible for positioning, orienting, and / or manipulating the end effector 220. In some examples, the shaft 310 is inserted in a distal direction and / or retracted in a proximal direction to provide an insertion DOF that is used to control how deeply the end effector 220 is positioned within the patient's anatomy. In some examples, the shaft 310 is rotatable about the longitudinal axis 312 to provide a roll DOF that is used to rotate the end effector 220. In some examples, additional flexibility in the position and / or orientation of the end effector 220 is provided by the engagement wrist 224 that is used to couple the end effector 220 to the distal end of the shaft 310. In some examples, the articulation wrist 224 includes one or more rotational joints 330, such as one or more roll joints, pitch joints, or yaw joints that provide one or more "roll," "pitch," and "yaw" DOFs, respectively. In such examples, such rotational joints 330 can be used to control the orientation of the end effector 220 relative to the longitudinal axis of the shaft 310. In some examples, the one or more rotational joints include a pitch and yaw joint; a roll, pitch, and yaw joint; a roll, pitch, and roll joint; and the like. In some examples, the end effector 220 can also include a clamping DOF for controlling the opening and closing of the jaws of the end effector 220 and / or an actuation DOF for controlling the operation, retraction, and / or extension of a cutting mechanism or stapling mechanism included in the end effector 220.

[0043] Typically, a drive system 340 associated with the end effector 220 is located at the proximal end of the shaft 310. The drive system 340 includes one or more components for introducing forces and / or torques to the end effector 220, which can be used to manipulate the aforementioned DOFs supported by the end effector 220. In some examples, the drive system 340 includes one or more motors, solenoids, servo mechanisms, active actuators, hydraulic devices, pneumatic devices, etc., which are driven based on signals from a control unit (e.g., Figure 1The drive system 340 operates based on signals received from the control unit 130 (e.g., a controller 130). In some examples, the signals include one or more currents, voltages, pulse-width modulated waveforms, etc. In some examples, the drive system 340 includes one or more shafts, gears, pulleys, rods, belts, etc., coupled to corresponding motors, solenoids, servos, active actuators, hydraulics, pneumatics, etc., as part of the articulation arm 222 to which the end effector 220 is mounted. In some examples, one or more drive mechanisms 350, such as disks, shafts, gears, pulleys, rods, belts, etc., are used to receive forces and / or torques from the motors, solenoids, servos, active actuators, hydraulics, pneumatics, etc., and apply these forces and / or torques to adjust various degrees of freedom (DOFs) of the end effector 220. In some examples, the shaft 310 is hollow, and the various drive mechanisms 350 are transmitted from the drive system 340 along the interior of the shaft 310 to corresponding degrees of freedom (DOFs) in the end effector 220 and / or articulation wrist 224.

[0044] Figure 4 is a simplified perspective view of the end effector 220 and the articulation wrist 224 according to some embodiments. Figure 4 , the distal end of the end effector 220 is depicted so that additional details of the end effector 220, the engagement wrist 224, and the drive mechanism 350 are visible. In more detail, the end effector 220 includes opposing jaws 410, which are shown in an open position. The jaws 410 are configured to move between an open position and a closed position, such that the end effector 220 is used to clamp and release tissue and / or other structures, such as sutures, located at a surgical site during a procedure. Alternatively, in some examples, the end effector 220 is configured as a surgical stapler, and the jaws 410 are configured to move between an open position and a closed position, such that the end effector 220 can be used to install one or more surgical staples. In some examples, the jaws 410 operate together as a single unit, with both jaws 410 opening and / or closing simultaneously. In some examples, the jaws 410 can be opened and / or closed independently, such that, for example, one jaw 410 remains stable while the other jaw 410 opens and / or closes.

[0045] In some examples, commanded motion of one control point included in the engagement arm 222 (e.g., the distal portion or tip of the jaw 410 of the end effector 220) is achieved by rotation and / or translation of one or more different control points included in the engagement arm 222, such as one or more joints of the engagement wrist 224, one or both ends of the shaft 310, and / or other joints or links included in the engagement arm 222. For example, in some embodiments, rotation (roll) of the jaw 410 about the axis of symmetry 412 can be generated by rotation 414 of the shaft 310 about the engagement wrist 224. However, in such cases, when rotation 414 of the jaw 410 is achieved by rotation 414 of the shaft 310 about the engagement wrist 224, each point of the shaft 310 is translated along an arc by a certain distance. Thus, in some cases, translation of the engagement wrist 224, portions of the shaft 310, and / or other portions of the engagement arm 222 occurs in conjunction with commanded rotation of the jaw 410 about the axis of symmetry 412.

[0046] Back to Figure 2 , portions of the end effector 220 are depicted as extending outside the field of view 236 of the imaging device 230. In this case, the movement of all portions of the end effector 220 within the workplace 202 and relative to the material 204 cannot be easily observed by an operator of the computer-assisted system 200 viewing the workplace 202 and the material 204 via the imaging device 230. For example, in some cases, portions of the end effector 220 or the engagement arms 222 may extend outside the field of view 236 when the operator of the computer-assisted system 200 is zooming in on the field of view 236 to confirm that certain materials have been properly captured by the end effector 220 and / or that a particular instrument associated with a different engagement arm (not shown) is precisely positioned in the proper orientation.

[0047] As described above, certain commanded motions of a control point of the end effector 220 (e.g., a distal portion or tip of the end effector 220) may be performed in conjunction with translation of one or more control points included in the engagement arm 222. Thus, commanded motion of a control point within the field of view 236 may result in translation of one or more control points of the engagement arm 222 that are outside of the field of view 236, which may be undesirable in many situations. According to various embodiments, when commanded motions of control points within the field of view 236 are performed, translational motion of portions of the end effector 220 and / or engagement arm 222 that are not within the field of view 236 of the imaging device 230 is limited or prevented. Figures 5 to 7 An example of such an implementation is described.

[0048] Figure 5 is a simplified diagram showing a portion of the end effector 520 extending outside the field of view 536 of the imaging device according to some embodiments. For example, the end effector 520 may be connected to Figure 2 The end effector 220 is consistent with the end effector 220, and the field of view 536 can be consistent with the end effector 220. Figure 2 The field of view is consistent with 236. Figure 5 In the example shown, the field of view 536 is depicted in terms of the "camera's perspective" and thus illustrates the field of view that may be captured by an imaging device (e.g., Figure 2 The content viewed by the imaging device 230). Figure 5 In the example shown, the field of view 536 includes a portion of the workplace 502, and a material 504 disposed near or within the workplace 502 ( For example , in the medical example, the patient's anatomy) and the portion of the end effector 520 disposed within the workplace 502. The end effector 520 is coupled to the articulation arm 522 via the articulation wrist joint 524 through the shaft 510. The shaft 510 may include various drive mechanisms 550 that are coupled to the articulation arm 522. Figure 3 The drive system 340 is consistent with the drive system (not shown).

[0049] According to various embodiments, in the event that a control point associated with the end effector 520 and / or engagement arm 522 is determined to be disposed outside of the field of view 536, translation of such control point is prevented when commanded movement of the end effector 520 and / or engagement arm 522 would otherwise cause such translation. Figure 5 In the illustrated example, the first distal portion 526 and the second distal portion 528 of the end effector 520 extend outside the field of view 536, and therefore, an operator controlling the movement of the end effector 520 cannot view the first distal portion 526 or the second distal portion 528. In embodiments, a computer-assisted system including the end effector 520 and the engagement arm 522 determines that a first control point 526A associated with the first distal portion 526 and a second control point 528A associated with the second distal portion 528 are positioned outside the field of view 536. In response, the computer-assisted system places the first control point 526A and the second control point 528A in a locked state. Consequently, commands for the end effector 520 and / or the engagement arm 522 are modified such that the modified commands do not translate the first control point 526A and the second control point 528A.

[0050] In some embodiments, where the first and second control points 526A, 528A are in a locked state and should not translate on command, other control points associated with the end effector 520 and / or engagement arm 522 that are not in a locked state may translate on command. Figure 5, examples of such control points include control point 532 associated with and / or co-located with a first rotational joint of joint wrist joint 524, control point 534 associated with and / or co-located with a second rotational joint of joint wrist joint 524, and control point 538 associated with and / or co-located with a third rotational joint or the end of axis 510 of joint wrist joint 524. Thus, in some examples, translation and / or rotation of control point 532, control point 534, and / or control point 538 is responsive to a control input via an input control (e.g., Figure 1 In some examples, such commands are modified so that the commanded translation and / or rotation of control point 532, control point 534, and / or control point 538 are adjusted so that first control point 526A and second control point 528A do not translate. For example, multiple joints included in joint arm 522 may be commanded to move to a combination of joint positions or a first pose that causes control point 532 to translate within field of view 536. Upon determining that such a command causes a control point in a locked state (e.g., control point 526A and / or control point 526B) to translate, the command for joint arm 522 to cause such translation is modified and / or not implemented. When modified, multiple joints included in joint arm 522 may be commanded to move to a combination of joint positions or a second pose that causes control point 532 to translate without translating control point 526A and / or control point 526B.

[0051] It is worth noting that in a repositionable structure such as articulated arm 522, uncommanded translation of a control point in a locked state may occur in some cases when a command that would cause translation of the control point is modified so that such translation is not achieved. For example, in some cases, uncommanded translation of a control point in a locked state may occur in response to commanded motion of other components of the repositionable structure. In another example, in some cases, uncommanded translation of a control point in a locked state may occur in response to factors external to the computer-assisted system that includes articulated arm 522 (e.g., movement of the workspace 502 relative to articulated arm 522, external forces applied to articulated arm 522, etc.). In a medical example, patient movement, such as due to breathing, heartbeat, etc., may cause uncommanded translation of a control point in a locked state. Typically, such uncommanded translation is relatively small and of the same order as other uncommanded motion of the control point that may occur during normal operation of the computer-assisted system.

[0052] In some embodiments, a control point in a locked state may be subject to rotation induced by a command input by an operator. In an example, rotation of a control point in a locked state may be achieved when the rotation is about an axis passing through the locked control point and does not cause the locked control point (or other locked control points) to translate. For example, in one such example, a control point 538 associated with an end of the shaft 510 may be located outside the field of view 536 and, in this case, be in a locked state. In this example, the longitudinal axis 512 of the shaft 510 passes through the control point 538. Therefore, rotation of the control point 538 about the longitudinal axis 512 does not cause translation of the control point 538 and can occur while the control point 538 is in the locked state. Therefore, in this example, when the control point 538 is in the locked state, the shaft 510 and the control point 538 can rotate about the longitudinal axis 512.

[0053] In some embodiments, a control point is in a locked state when it is determined to be located outside the field of view 536. In some examples, the computer-assisted system determines that the control point is outside the field of view 536 based on forward kinematics, computer vision analysis, and / or manual input, and / or a combination of any of these techniques. A person skilled in the art will readily appreciate that any combination of these techniques may be employed to determine whether a control point is outside the field of view 536.

[0054] In some examples, the computer-assisted system determines that a control point is outside the field of view 536 based on the forward kinematics of the imaging device generating the field of view 536 and the forward kinematics of the articulated arm 522 and / or other joints coupled to the end effector 520. In such examples, the forward kinematics of the articulated arm 522 are used to determine the positions of various control points of the articulated arm 522 and / or the end effector 520. For example, the positions of various control points can be determined in a coordinate system shared by the imaging device generating the field of view 536. Similarly, the forward kinematics of the articulated arm (not shown) associated with the imaging device can be used to determine the position and extent of the field of view 536. For example, the position and extent of the field of view 536 can be determined in a coordinate system shared by the articulated arm 522. In the shared coordinate system, the positions of various control points of the articulated arm 522 and the end effector 520 can be determined relative to the position and extent of the field of view 536. In such an example, the positions of various control points of the engagement arm 522 and / or end effector 520 disposed within the field of view 536 may be determined when such control points are not visible, such as when obscured by other instruments within the field of view 536 or by the material 504 .

[0055] In some examples, the computer-assisted system determines that a control point is outside the field of view 536 based on a computer vision analysis of the workplace 502 as viewed by an imaging device generating the field of view 536. In such examples, conventional computer vision algorithms can be employed to identify specific control points of the engagement arm 522 and / or end effector 520 that are disposed within the field of view 536. Based on the identified control points within the field of view 536, the computer-assisted system can then determine control points of the engagement arm 522 and / or end effector 520 that are disposed outside the field of view 536.

[0056] In some examples, the computer-assisted system determines that a control point is outside the field of view 536 based on one or more manual user inputs. In such examples, the manual user input can indicate a particular joint or control point of the joint arm 522 and / or the end effector 520 that should be in a locked state. Thus, in such examples, an operator of the computer-assisted system can cause a control point of the joint arm 522 and / or the end effector 520 that is disposed within the field of view 536 to be in a locked state. In some examples, such manual input is provided by an operator of the computer-assisted device via an input control of an operator workstation of the computer-assisted device, such as pressing a Figure 1 In some examples, such manual input is generated by an operator via a display system (e.g., Figure 1 The user interface generated by the display system 192 in the display system is generated. For example, the operator can generate manual input via a touch screen included in such a display system. In some examples, the operator generates manual input via voice commands and / or gestures.

[0057] Figure 6 is a simplified diagram showing the end effector 520 disposed within the field of view 636 and coupled to a control point outside the field of view 636, according to some embodiments. Figure 6 In the example shown, the field of view 636 includes the material 504 and a portion of the workspace 502 disposed near or within the workspace 502. As shown, the end effector 520 is coupled to one or more control points associated with the articulated arm 522 that are disposed outside the field of view 636. The control points disposed outside the field of view 636 include control points 532, 534, and 538 associated with the rotational joint and axis 510 of the articulated wrist joint 524. Thus, in Figure 6In the example shown, control points 532, 534, and 538 are in a locked state and cannot be commanded to translate. In contrast, control points within field of view 636 include first control point 526A associated with first distal portion 526, second control point 528A associated with second distal portion 528, and control point 632A associated with base portion 632 of end effector 520, which are not in a locked state.

[0058] exist Figure 6 In the example shown, actuation commands for the engagement arm 522 that cause movement (rotation and / or translation) of the first control point 526A, the second control point 528A, and / or the control point 632A within the field of view 636 and that do not cause translation of the control point 532, the control point 534, or the control point 538 may be normally implemented. Thus, actuation commands for the engagement arm 522 that may be normally implemented include actuation commands that cause the jaw 410 to open and / or close, the control point 632A to translate within the field of view 636, and the control point 532, the control point 534, and / or the control point 538 to rotate about the longitudinal axis 512 of the shaft 510. In the example, actuation commands for the engagement arm 522 that cause the end effector 520 to rotate 601 about the control point 532 may be normally implemented until it is determined that the control point 632A is outside the field of view 636. After such a determination, actuation commands for the joint arm 522 that cause further rotation of the end effector 520 about the control point 532 are modified and / or not implemented. In contrast, actuation commands for the joint arm 522 that cause translation of the control points 532, 534, and / or 538 are not normally implemented. Instead, such commands can be modified so that translation of the control points 532, 534, and / or 538 does not occur. In an example, actuation commands for the joint arm 522 that cause rotation (roll) of the first and second control points 526A, 528A about the axis of symmetry 412 can be normally implemented, while actuation commands for the joint arm 522 that cause translation of the control points 532, 534, and / or 538 can be modified and / or not implemented. When modified, various joints included in the joint arm 522 can be commanded to move to poses or combinations of joint positions that are different from the poses or combinations of joint positions indicated in the unmodified commands.

[0059] In some embodiments, tactile feedback is provided to the operator in response to the operator generating one or more commands for the articulated arm 522 that cause translation of one or more control points in a locked state. In an example, such tactile feedback is applied to the input control that generates the command to translate the locked control point, e.g. Figure 1 195. In an example, such tactile feedback may include vibration of a particular input control with a specified amplitude, intensity, and / or duration.

[0060] In some embodiments, the perceived intensity of the tactile feedback provided to the operator is related to the actual movement of one or more control points of the end effector 520 or the joint arm 522 that does not match the commanded movement. Thus, when the operator inputs a commanded movement of the joint arm 522 and / or the end effector 520 via an input control and modifies the commanded movement to avoid translation of one or more locked control points, tactile feedback is provided to the operator. In an example, the perceived intensity of the tactile feedback provided to the operator is based on the amount by which the actual movement of one or more control points differs from the commanded movement of the one or more control points. In an example, the amplitude, intensity, and / or duration of the tactile feedback increases as the actual movement differs from the commanded movement. In an example, the perceived intensity of the tactile feedback increases (e.g., weighted or scaled) at a rate proportional to the amount by which the actual movement differs from the commanded movement. In an example, separate tactile feedback is provided to the operator for each of the multiple degrees of freedom. Thus, in such an example, a different perceived intensity of tactile feedback is provided to the operator for each degree of freedom for which the corresponding actual movement differs from the commanded movement. In another example, a single perceived intensity of haptic feedback is provided to the operator based on a combination of the actual and commanded motions for each degree of freedom for which the corresponding actual motion differs from the commanded motion. In such an example, the perceived intensity of the haptic feedback is proportional to the combination of the differences between the actual and commanded motions for each degree of freedom (e.g., the vector sum of the differences associated with the individual degrees of freedom).

[0061] Figure 7 is a simplified diagram of an exemplary method 700 for local kinematic locking of out-of-line-of-sight control points, according to some embodiments. According to some embodiments, method 700 may include one or more of processes 701 through 704, which may be implemented at least in part in the form of executable code stored on a non-transitory, tangible, machine-readable medium, which when executed on one or more processors (e.g., Figure 1 When running on the processor 140 in the control unit 130 of the computer readable medium, the computer readable medium may cause one or more processors to perform one or more of the processes 701 to 704.

[0062] At process 701, it is determined whether there is a request for a relocatable structure (e.g., Figure 5Any control point of the joint arm 522 in the computer-assisted system (e.g., the joint arm 522 in the computer-assisted system) that is located outside the field of view of the imaging device associated with the computer-assisted system. As described above, such a determination can be made based on forward kinematics, computer vision analysis, and / or one or more manual inputs performed by an operator of the computer-assisted system, and / or any combination of these techniques. In an example, one or more control points determined to be located outside the field of view of the imaging device are placed in a locked state. In such an example, as described below, a command that causes a translation of a control point in the locked state can be modified so that such translation is not achieved.

[0063] At process 702 , one or more actuation commands are received by the computer-assisted system to cause at least one translation of a control point in a locked state. In an example, the one or more actuation commands may be received via operator input, such as by manipulating one of the input controls 195 .

[0064] At process 703, one or more modification commands are generated based on the one or more actuation commands received in process 702. In an example, the modification commands do not cause translation of the one or more control points in the locked state. In an example, the one or more modification commands are generated by modifying the one or more actuation commands.

[0065] At process 704, one or more modification commands are applied by the computer-assisted system. In an example, the one or more modification commands cause one or more joints of the computer-assisted system to be actuated, wherein such actuation does not result in translation of one or more control points in a locked state. In another example, the one or more modification commands prevent one or more joints of the computer-assisted system from being actuated, such that translation of one or more control points in a locked state does not result in translation. In an example, when the modification command is applied, haptic feedback is generated at process 704. After completing process 704, method 700 returns to process 701.

[0066] While exemplary embodiments have been shown and described, a wide range of modifications, variations, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be employed without the corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the present invention should be limited solely by the appended claims, and as appropriate, the claims should be interpreted broadly and in a manner consistent with the scope of the embodiments disclosed herein.

[0067] Any and all combinations of any claim elements recited in any claim and / or any elements described in this application, in any manner, are within the contemplated scope of the invention and protection.

[0068] The descriptions of the various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0069] Aspects of the present embodiment can be implemented as a system, method or computer program product. Therefore, various aspects of the present disclosure can take the following forms: a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.) or an embodiment of a combination of software and hardware aspects, which can all be collectively referred to as "modules", "systems" or "computers" in this article. In addition, any hardware and / or software technology, processing, function, components, engines, modules or systems described in this disclosure can be implemented as circuits or circuit groups. In addition, various aspects of the present disclosure can take the form of a computer program product contained in one or more computer-readable media, and the computer-readable medium has a computer-readable program code contained thereon.

[0070] Any combination of one or more computer-readable media can be utilized.Computer-readable media can be a computer-readable signal medium or a computer-readable storage medium.Computer-readable storage media can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, equipment or devices, or any suitable combination of the foregoing.More specific examples (non-exhaustive list) of computer-readable storage media will include the following: an electrical connection with one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.In the context of this document, a computer-readable storage medium can be any tangible medium that can contain or store a program used by an instruction execution system, device or device or used in combination with an instruction execution system, device or device.

[0071] Aspects of the present disclosure are described above with reference to the flowchart illustrations and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It will be understood that each frame in the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine. Instructions enable the function / action specified in the flowchart and / or block diagram frame or multiple frames to be implemented when the processor of the computer or other programmable data processing device is executed. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, an application-specific processor or a field programmable gate array.

[0072] The flow chart and block diagram in the figure illustrate the architecture, function and operation of the possible implementation of the system, method and computer program product according to each embodiment of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent the part of the module, segment or code comprising one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the function pointed out in the frame may not occur in the order pointed out in the accompanying drawings. For example, depending on the function involved, the two frames shown in succession can actually be performed substantially simultaneously, or the frames can sometimes be performed in the opposite order. It will also be noted that each frame in the block diagram and / or flow chart illustration and the combination of the frames in the block diagram and / or flow chart illustration can be realized by a system based on dedicated hardware or a combination of dedicated hardware and computer instructions that performs the specified function or action.

[0073] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, the scope of which is to be determined by the claims that follow.

Claims

1. A computer-aided system comprising: a repositionable structure comprising one or more joints coupled to the end effector; as well as control unit, Wherein, the control unit is configured to: determining that a control point associated with the end effector is not within a field of view of an imaging device capturing images of a workspace; receiving an actuation command for translating the control point if the control point is not within the field of view of the imaging device; generating, based on the actuation command, a modification command that does not translate the control point; and One or more joints of the end effector or the repositionable structure are actuated based on the modification command.

2. The computer-aided system according to claim 1, wherein: The modification command causes the control point to rotate.

3. The computer-aided system according to claim 1, wherein: The control point is included in the end effector.

4. The computer-aided system according to claim 1, wherein: The control points correspond to joints of the end effector.

5. The computer-aided system according to claim 1, wherein: The control point corresponds to a distal portion of the end effector.

6. The computer-aided system according to claim 1, wherein: The actuation command causes a specific operation to be performed by the end effector.

7. The computer-aided system according to claim 6, wherein: The specific operation includes one of: a stapling operation, a clamping operation, a cutting operation, an energy delivery operation, a translation of at least a portion of the end effector, or a rotation of at least a portion of the end effector.

8. The computer-aided system according to claim 6, wherein: The modification command causes the specific operation to be performed by the end effector.

9. The computer-aided system according to claim 6, wherein: The modification command prevents the particular operation from being performed by the end effector.

10. The computer-aided system according to any one of claims 1 to 9, wherein: The control unit also applies tactile feedback to input controls of the computer-assisted system.

11. The computer-aided system according to claim 10, wherein: The haptic feedback has a perceived intensity based on a difference between a first movement of the control point caused by the actuation command and a second movement of the control point caused by the modification command.

12. The computer-aided system according to any one of claims 1 to 9, wherein: The end effector includes one of a surgical stapler, a suction irrigator, an electrocautery device, a clamp, or a cutting mechanism.

13. The computer-aided system according to any one of claims 1 to 9, wherein: In order to determine that the control point is not within the field of view of the imaging device, the control unit is configured to determine a current position of the control point based on forward kinematics of the repositionable structure.

14. The computer-aided system according to any one of claims 1 to 9, wherein: In order to determine that the control point is not within the field of view of the imaging device, the control unit is configured to receive a user input indicating the control point.

15. The computer-aided system according to any one of claims 1 to 9, wherein: In order to determine that the control point is not within the field of view of the imaging device, the control unit is configured to perform a computer vision analysis of the field of view based on information generated by the imaging device.

16. The computer-aided system according to any one of claims 1 to 9, wherein: To determine that the control point is not within the field of view of the imaging device, the control unit is configured to utilize two or more of the following: determining the current position of the control point based on forward kinematics of the repositionable structure, receiving user input indicating the control point, or performing a computer vision analysis of the field of view based on information generated by the imaging device.

17. The computer-aided system according to any one of claims 1 to 9, wherein: The end effector includes a laparoscopic instrument.

18. A method for operating a computer-assisted device, the computer-assisted device comprising a repositionable structure comprising one or more joints coupled to an end effector, the method comprising: determining, by a control unit, that a control point associated with the end effector is not within a field of view of an imaging device capturing images of a workplace; receiving, by the control unit, an actuation command for translating the control point if the control point is not within the field of view of the imaging device; Based on the actuation command, the control unit generates a modification command that does not cause the control point to translate; as well as One or more joints of the end effector or the repositionable structure are actuated by the control unit based on the modification command using one or more motors, solenoids, servo mechanisms, or actuators.

19. The method according to claim 18, wherein The modification command causes the control point to rotate.

20. The method according to claim 18, wherein The control point is included in the end effector.

21. The method according to claim 18, wherein The control points correspond to joints of the end effector.

22. The method according to claim 18, wherein The control point corresponds to a distal portion of the end effector.

23. The method according to claim 18, wherein The actuation command causes a specific operation to be performed by the end effector.

24. The method according to claim 23, wherein The specific operation includes one of: a stapling operation, a clamping operation, a cutting operation, an energy delivery operation, a translation of at least a portion of the end effector, or a rotation of at least a portion of the end effector.

25. The method according to claim 23, wherein The modification command causes the specific operation to be performed by the end effector.

26. The method according to claim 23, wherein The modification command prevents the particular operation from being performed by the end effector.

27. The method of claim 18, further comprising: Haptic feedback is applied to input controls of the method.

28. The method according to claim 27, wherein The haptic feedback has a perceived intensity based on a difference between a first movement of the control point caused by the actuation command and a second movement of the control point caused by the modification command.

29. The method according to claim 18, wherein The end effector includes one of a surgical stapler, a suction irrigator, an electrocautery device, a clamp, or a cutting mechanism.

30. The method of claim 18, wherein Determining that the control point is not within the field of view of the imaging device includes determining a current position of the control point based on forward kinematics of the repositionable structure.

31. The method according to claim 18, wherein Determining that the control point is not within the field of view of the imaging device includes receiving user input indicating the control point.

32. The method of claim 18, wherein: Determining that the control point is not within the field of view of the imaging device includes performing a computer vision analysis of the field of view based on information generated by the imaging device.

33. The method of claim 18, wherein: Determining that the control point is not within the field of view of the imaging device includes two or more of: determining a current position of the control point based on forward kinematics of the repositionable structure, receiving user input indicating the control point, or performing a computer vision analysis of the field of view based on information generated by the imaging device.

34. The method of claim 18, wherein The end effector includes a laparoscopic instrument.

35. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions adapted, when executed by one or more processors associated with a computer-assisted apparatus, to cause the computer-assisted apparatus to perform the method of any one of claims 18 to 34.

Images