System and method for recentering an imaging device and an input control device

Abstract

A system and method of recentering an imaging device and input control devices includes a medical device having one or more end effectors, an imaging device, one or more input control devices for remotely operating the one or more end effectors, and a control unit including one or more processors coupled to the end effectors, the imaging device, and the input control devices. The control unit, in response to a recentering request, suspends remote operating control of the end effectors by the input control devices, determines a field of view recentering movement for the imaging device so that the end effectors are contained within a field of view space of the imaging device, determines one or more input control device recentering movements to provide a position and orientation harmony between each of the input control devices and a corresponding end effector of the end effectors, executes the field of view recentering movement and the input control device recentering movements, and resumes remote operating control of the end effectors by the input control devices.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 201910658710.6, filed March 17, 2015, entitled “System and Method for Recentering Imaging Devices and Input Controls,” which is a divisional application of Chinese Patent Application No. 201580023718.5 (PCT / US2015 / 021105), filed March 17, 2015, entitled “System and Method for Recentering Imaging Devices and Input Controls.”

[0002] Related Applications

[0003] This disclosure claims priority to U.S. Provisional Patent Application No. 61 / 954,191, filed March 17, 2014, entitled “System and Method for Recentering Imaging Devices and Input Controls,” which is incorporated by reference herein in its entirety. TECHNICAL FIELD

[0004] The present disclosure relates generally to the remote operation of devices with articulated arms and more particularly to recentering imaging devices and input controls. BACKGROUND

[0005] Increasingly, devices are being replaced with autonomous and semi-autonomous electronic devices. This is especially true in hospitals today with large arrays of autonomous and semi-autonomous electronic devices found in operating rooms, interventional suites, intensive care units, emergency rooms, etc. For example, glass thermometers and mercury thermometers are being replaced with electronic thermometers, intravenous drip lines now include electronic monitors and flow regulators, and traditional hand-held surgical instruments are being replaced by computer-assisted medical devices.

[0006] These electronic devices provide advantages and challenges to the workers operating them. Many of these electronic devices can be capable of autonomous or semi-autonomous movement of one or more articulated arms and / or end effectors. It is also conventional for the electronic device to be operated via remote operation using one or more input control devices on an operator console to control movement and / or operation of the articulated arms and / or end effectors. When the electronic device is remotely operated from the operator console and / or the end effectors are in use in areas not directly visible to the operator, such as during computer-assisted surgery when the end effectors are hidden by a patient's anatomy, the electronic device can include an imaging device that captures a region of interest and uses a display system to display the region of interest to the operator. When the operator controls the articulated arms and / or end effectors, the operator often attempts to keep the end effectors in view of the imaging device so that operation of the end effectors can be observed on the display system. In addition, the position and orientation of the input control devices are often matched to the end effectors so that when the input control devices are moved, the end effectors "follow" those movements.

[0007] When the imaging device and / or the end effectors move, it is possible that the operator can lose sight of one or more of the end effectors and / or lose track of the spatial relationship between the imaging device and the end effectors. This can be further complicated when the operator of the electronic device switches control to additional articulated arms and / or end effectors that can be parked in other areas around the region of interest, and / or when the end effectors are partially or completely blocked by other objects in the region of interest. To regain visualization of the end effectors (i.e., to place the end effectors within the view volume of the imaging device), the operator can have to perform a series of re-centering movements with the imaging device to find an imaging device pose (position and orientation) that includes the end effectors. This series of movements can become tedious, time consuming, and / or impractical.

[0008] In addition, when the imaging device moves and / or the input control devices are switched to additional articulated arms and / or end effectors, the spatial orientation between the imaging device and the end effectors can change. This can result in a discordance between the position and / or orientation of the end effectors as displayed by the display system and the corresponding position and / or orientation of the input control devices for those end effectors. In some cases, this can be corrected by the operator by activating clutches for the input control devices and then repositioning and / or reorienting the input control devices to match the end effector position and / or orientation as displayed on the display system. As with movement of the imaging device, these repositioning and / or reorienting operations can also become tedious, time consuming, and / or impractical.

[0009] Accordingly, improved methods and systems for visually reacquiring an end effector and / or repositioning and / or reorienting an input control device to match the end effector are desirable. SUMMARY

[0010] Consistent with some embodiments, a computer-assisted medical device includes one or more end effectors, an imaging device, one or more input control devices for remotely operating the one or more end effectors, and a control unit including one or more processors coupled to the end effectors, the imaging device, and the input control devices. The control unit, in response to a recentering request, suspends remote operating control of the end effectors by the input control devices, determines a view recentering movement for the imaging device so that the end effectors are contained within a view space of the imaging device, determines one or more input control device recentering movements to provide position and orientation harmony between each of the input control devices and a corresponding end effector of the end effectors, executes the view recentering movement and the input control device recentering movements, and resumes remote operating control of the end effectors by the input control devices.

[0011] Consistent with some embodiments, a method of controlling motion in a medical device includes, in response to a recentering request, suspending remote operating control of one or more end effectors of the medical device by one or more input control devices of the medical device, determining a view recentering movement for the imaging device so that the end effectors are contained within a view space of an imaging device of the medical device, determining one or more input control device recentering movements to provide position and orientation harmony between each of the input control devices and a corresponding end effector of the end effectors, executing the view recentering movement and the input control device recentering movements, and resuming remote operating control of the end effectors by the input control devices.

[0012] Consistent with some embodiments, a method of controlling motion in a medical device includes, in response to a recentering request, suspending remote operating control of one or more end effectors of the medical device by one or more input control devices of the medical device, determining a view recentering movement for the imaging device so that the end effectors are contained within a view space of an imaging device of the medical device, executing the view recentering movement, and resuming remote operating control of the end effectors by the input control devices.

[0013] Consistent with some embodiments, a method of determining a preferred working distance for an imaging device of a medical device includes detecting a start of a repositioning movement for the imaging device of the medical device; detecting an end of the repositioning movement; determining a current working distance based on a first distance between the imaging device and one or more targets associated with one or more end effectors of the medical device that are within a viewing volume of the imaging device at the end of the repositioning movement, the first distance being measured in a view direction of the imaging device; and aggregating the current working distance with previously obtained current working distances to determine the preferred working distance.

[0014] Consistent with some embodiments, a method of controlling motion in a medical device includes pausing remote operator control of one or more end effectors of the medical device by one or more input control devices of the medical device in response to a recentering request; determining one or more input control device recentering movements to provide position and orientation harmony between each of the input control devices and a corresponding end effector of the end effectors; performing the input control device recentering movements; and resuming remote operator control of the end effectors by the input control devices.

[0015] Consistent with some embodiments, a method of determining an ergonomic center for an operator console of a medical device includes detecting a start of a repositioning movement for one or more input control devices of the medical device; detecting an end of the repositioning movement; determining positions of one or more control points associated with the input control devices at the end of the repositioning movement; aggregating the positions to determine an input control device center point; and aggregating the input control device center point with previously obtained input control device center points to determine the ergonomic center.

[0016] Consistent with some embodiments, a non-transitory machine-readable medium includes a plurality of machine-readable instructions. When the machine-readable instructions are executed by one or more processors associated with a medical device, they cause the one or more processors to perform a method. The method includes pausing remote operator control of one or more end effectors of the medical device by one or more input control devices of the medical device in response to a recentering request; determining a view recentering movement for the imaging device so that the end effectors are contained within a view volume of an imaging device of the medical device; determining one or more input control device recentering movements to provide position and orientation harmony between each of the input control devices and a corresponding end effector of the end effectors; performing the view recentering movement and the input control device recentering movements; and resuming remote operator control of the end effectors by the input control devices.

[0017] Consistent with some embodiments, a method of controlling motion of an imaging device coupled to a medical device includes detecting activation of an imaging device motion mode, and determining whether one or more motion input control devices are being used. When the one or more motion input control devices are being used, controlling a pose of the imaging device based on the one or more motion input control devices. When the one or more motion input control devices are not being used for a timeout period, recentering the imaging device. The recentering the imaging device includes determining a field of view recentering movement for the imaging device so that one or more end effectors of the medical device are contained within a field of view space of the imaging device, and performing the field of view recentering movement. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a simplified schematic of a computer-assisted system in accordance with some embodiments.

[0019] Figure 2 is a simplified schematic of a method of recentering end effectors and input control devices in accordance with some embodiments.

[0020] Figure 3A and Figure 3B are simplified schematics of imaging views before and after a view recentering operation in accordance with some embodiments.

[0021] Figure 4A and Figure 4B are simplified schematics of imaging views and side views, respectively, after a view recentering operation in accordance with some embodiments.

[0022] Figure 5 is a simplified schematic of a method of view recentering in accordance with some embodiments.

[0023] Figure 6 is a simplified schematic of a method of determining a preferred working distance for an imaging device in accordance with some embodiments.

[0024] Figure 7 is a simplified schematic showing the relationship between end effectors in an image on a display system and corresponding input control devices in a console workspace after an input control device recentering operation in accordance with some embodiments.

[0025] Figure 8 is a simplified schematic of a method of input control device recentering in accordance with some embodiments.

[0026] Figure 9is a simplified schematic of a method of determining an ergonomic center / human control center of an input control device according to some embodiments.

[0027] Figure 10 is a simplified schematic of a method of controlling an imaging device according to some embodiments.

[0028] In the drawings, elements having the same name or reference numeral have the same or similar functionality. DETAILED DESCRIPTION

[0029] In the following description, specific details are set forth describing some embodiments consistent with the present disclosure. However, persons skilled in the art will appreciate that some embodiments can be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative only and not limiting of the disclosure. Other elements, of course, can be utilized and other techniques can be employed without departing from the scope and spirit of the disclosure. In addition, to the extent the following description is directed to a particular implementation of implementations consistent with the present disclosure, it is understood that the disclosure can be practiced with other implementations as well without departing from the spirit and scope of the disclosure.

[0030] Figure 1 is a simplified schematic of a computer-assisted system 100 according to some embodiments. As shown in Figure 1 the computer-assisted system 100 includes a device 110 having one or more movable arms 120 or articulated arms 120. Each of the one or more articulated arms 120 can support one or more end effectors 125. In some embodiments, the device 110 can be consistent with a computer-assisted surgical device. The one or more end effectors 125 can include surgical instruments, imaging devices, and / or the like. In some examples, the surgical instruments can include forceps, graspers, retractors, cauterizing tools, suction tools, suturing devices, and / or the like. In some examples, the imaging devices can include endoscopes, cameras, stereoscopic devices, and / or the like.

[0031] The device 110 is coupled to the control unit 130 via an interface. The interface can include one or more cables, connectors, and / or buses, and can further include one or more networks with one or more network switches and / or routing devices. The control unit 130 includes a processor 140 coupled to a memory 150. The operation of the control unit 130 is controlled by the processor 140. And while the control unit 130 is shown with only one processor 140, it should be understood that the processor 140 can represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and / or the like in the control unit 130. The control unit 130 can be implemented as a standalone subsystem and / or board added to a computing device and / or as a virtual machine.

[0032] The memory 150 can be used to store software executed by the control unit 130 and / or one or more data structures used during the operation of the control unit 130. The memory 150 can include one or more types of machine-readable media. Some common forms of machine-readable media can include floppy diskettes, flexible disks, hard disks, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tapes, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other medium from which a processor or computer is adapted to read.

[0033] As shown, the memory 150 includes a motion control application 160 that can be used to support autonomous and / or semi-autonomous control of the device 110. The motion control application 160 can include one or more application programming interfaces (APIs) for receiving position, motion, and / or other sensor information from the device 110, exchanging position, motion, and / or collision avoidance information about other devices with other control units, and / or planning and / or assisting in planning motion of the device 110, the articulated arm 120, and / or the end effector 125 of the device 110. And while the motion control application 160 is depicted as a software application, the motion control application 160 can be implemented using hardware, software, and / or a combination of hardware and software.

[0034] The control unit 130 can further be coupled to an operator console 170 via an interface. The operator console 170 can be used by an operator, such as a surgeon, to control movement and / or operation of the articulated arms 120 and the end effectors 125. To support operation of the articulated arms 120, the operator console 170 includes a display system 180 for displaying images of at least portions of one or more of the articulated arms 120 and / or the end effectors 125. The display system 180 can be used, for example, when the operator is seeing the articulated arms 120 and / or end effectors 125 being used is not practical and / or possible. The operator console 170 can further include a console workspace with one or more input control devices or master control devices 195 that can be used to operate the device 110, the articulated arms 120, and / or the end effectors 125. Each of the input control devices 195 can be coupled to a distal end of its own articulated arm so that movement of the input control device 195 can be detected by the operator console 170 and communicated to the control unit 130. To provide improved ergonomics, the console workspace can also include one or more rests, such as an arm rest 190, against which an operator can rest his arms while manipulating the input control devices 195. In some examples, the display system 180 and the input control devices 195 can be used by an operator to remotely operate the articulated arms 120 and / or the end effectors 125. In some embodiments, the device 110, the operator console 170, and the control unit 130 can correspond to the da Vinci® Surgical System commercially available from Intuitive Surgical, Inc. of Sunnyvale, California. Surgical system.

[0035] In some embodiments, other configurations and / or arrangements can be used with the computer-assisted system 100. In some examples, the control unit 130 can be included as part of the operator console 170 and / or the device 110. In some embodiments, the computer-assisted system 100 can be found in an operating room and / or interventional suite. And while the computer-assisted system 100 includes only one device 110 with two articulated arms 120, those of ordinary skill in the art will appreciate that the computer-assisted system 100 can include any number of devices with articulated arms and / or end effectors of similar and / or different design than the device 110. In some examples, each device can include fewer or more articulated arms 120 and / or end effectors 125.

[0036] Figure 2is a simplified schematic of a method 200 of recentering end effectors and input control devices according to some embodiments. One or more of the processes 210-280 of the method 200 can be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors (e.g., the processors 140 in the control unit 130), can cause the one or more processors to perform one or more of the processes 210-280. In some embodiments, the method 200 can be performed by an application, such as the motion control application 160. In some embodiments, the method 200 can be used to recenter one or more of the end effectors 125 in an image captured by an imaging device and displayed on the display system 180 and / or to recenter one or more of the input control devices 195 in the console workspace so that the position and / or orientation of the input control devices 195 correspond to the position and / or orientation of the end effectors 125 displayed in the image.

[0037] At process 210, a request to recenter is detected. In some examples, an operator of the electronic device can manually trigger the request to recenter using one or more input control devices such as switches, pedals, levels, voice recognition, and / or the like. In some examples, the request can be issued as a temporal input that triggers recentering and / or as a continuous input that activates recentering until it is complete and / or the input is withdrawn. In some examples, the request to recenter can be made automatically in response to a change in system state. In some examples, the change in system state can include a change in the association between an input control device and a teleoperated end effector. In some examples, the change in system state can include a change in the association between an input control device and a teleoperated end effector where one or more end effectors are detected to be outside the field of view of an imaging device. In some examples, the change in system state can include a change in the mode of an imaging device that results in one or more end effectors being outside the field of view of the imaging device (e.g., a change in digital zoom, a change in telephoto angle, and / or the like). In some examples, the request to recenter can also include a designation of the articulated arms and end effectors to be recentered. In some examples, the detection of the recentering request can be confirmed by appropriate feedback to the operator, such as a unique sound, a message on the console, an indicator, and / or the like.

[0038] At process 220, operator control of one or more end effectors is suspended. The ability of one or more of the end effectors of the teleoperated electronic device to be recentered can be suspended prior to recentering can begin. Suspension of operator control allows the recentering operation to proceed without interference from operator commanded motion.

[0039] At process 230, a desired view recentering movement is determined. The view recentering movement is determined using, for example, the sensed joint positions in the articulated arm and end effector coupled to the articulated arm, and one or more kinematic models of the articulated arm and end effector. In some examples, this can include determining a pose (e.g., position and / or orientation) of one or more end effectors of interest associated with the electronic device being controlled. In some examples, each determined pose can be mapped to a common coordinate system, such as a world coordinate system and / or a view coordinate system. Using knowledge of the geometry of the pose for the preferred working distance of the imaging device and the preferred working distance, a desired pose for the imaging system to place the end effector within the view space of the imaging device is determined. The pose of the imaging device and the one or more kinematic models can then be used to determine a desired view recentering movement of the imaging device.

[0040] At process 240, a desired input control device recentering movement is determined. The pose for the end effector determined during process 230 can be mapped to a coordinate system of the console workspace in which the input control device corresponding to the end effector is located. The pose can be mapped using knowledge of the preferred ergonomic center of the console workspace and a scale factor between the distance of the workspace used by the end effector and the distance of the console workspace containing the input control device. The mapped pose and one or more kinematic models of the input control device can then be used to determine a corresponding input control device recentering movement of the input control device. In some embodiments, two input control device recentering movements are determined, one corresponding to a left input control device associated with a first end effector of the end effectors and the other corresponding to a right input control device associated with a second end effector of the end effectors. In some embodiments, corresponding recentering movements for other numbers of input control devices can also be determined.

[0041] At process 250, it is determined whether the field of view recentering movement and / or the input control device recentering movement is valid. Using the kinematic model of the imaging device and the desired recentering movement for the imaging device determined during process 230, it is determined whether the desired recentering movement for the imaging device is valid. In some examples, this validity determination can include checking one or more constraints on factors including movement of the imaging device, position of other articulated arms, other end effectors, and / or devices in the workspace of the electronic device and / or the ability of the imaging device to obtain suitable images of the end effectors. Using the kinematic model of the input control device and the desired recentering movement for the input control device determined during process 240, it is determined whether the desired recentering movement for the input control device is valid. In some examples, this validity determination can include checking one or more constraints on factors including movement of the input control device, position of portions of the operator workstations in the console workspace, and / or ergonomic considerations for the operator of the input control device. When the recentering movement is determined to be valid, the recentering movement is executed using process 260. When any recentering movement is determined to be invalid, an error is indicated using process 270.

[0042] At process 260, the field of view recentering movement and the input control device recentering movement are coordinated. One or more movement commands are sent to one or more actuators of the articulated arm coupled to the imaging device to command and / or instruct the imaging device to perform the field of view recentering movement. One or more movement commands are also sent to one or more actuators of the articulated arm coupled to the input control device to command and / or instruct the input control device to perform the input control device recentering movement. The movement commands for the imaging device and the input control device are typically coordinated. In some examples, the coordination can allow for simultaneous recentering of the imaging device and the input control device. In some examples, the coordination can be performed so that at least some positions and / or orientations are maintained in harmony between the end effector within the field of view of the imaging device and the pose of the input control device during the recentering movement. In some examples, process 260 can also include providing audio and / or visual feedback to the operator indicating that the recentering operation is occurring. In some examples, the audio feedback can include a unique sound, a spoken phrase, and / or the like. Upon completion of the recentering movement, operator control is restored using process 280.

[0043] At process 270, an error is indicated. When the determined recentering movement is determined to be invalid, the operator is notified. In some examples, the notification can include any suitable audio and / or visual feedback. In some examples, the audio feedback can include the playing of a unique sound. After the error is indicated, operator control is restored using process 280.

[0044] At process 280, operator control of the end effector is resumed. Whether a recentering movement was performed using process 260 or an error was indicated using process 270, control of the end effector using the input control device is returned to the operator. When an error was indicated, recentering of the imaging device and / or the input control device can become the responsibility of the operator. After a period of time in which the end effector and / or the imaging device are controlled by the operator, another recentering operation can be detected by using process 210.

[0045] As discussed above and further emphasized here, Figure 2 These are merely examples which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many modifications, alternatives, and variations. According to some embodiments, additional conditions can result in early termination of the method 200, such as by returning operator control using process 280 and / or by a pause in device operation. In some examples, additional conditions can include manual intervention or override from the operator using one or more control devices or articulated arms on the operator console, detection of operator disengagement from the operator console using one or more safety interlock devices, a positional tracking error in the articulated arms and / or input control devices, a system failure, and / or the like.

[0046] Figure 3A And Figure 3B are simplified diagrams of imaging views before and after a field of view recentering operation according to some embodiments. As shown in Figure 3A is shown a workspace including three articulated arms before execution of a field of view recentering operation. A first articulated arm ends with a gripper type end effector 310. The gripper type end effector 310 includes two gripper fingers 312 and 314 and a pivot joint 316. A second articulated arm also ends with a gripper type end effector 320 that includes two gripper fingers 322 and 334 and a pivot joint 326. A third articulated arm includes a single finger end effector 330 that contains an end point 332 and a reference point 334. In some examples, the reference point 334 can correspond to a rotatable joint. In some examples, the single finger end effector 330 can represent a cauterization tool, an aspiration tool, and / or the like. In some examples, the articulated arms can be representative examples of the articulated arms 120 and the gripper type and / or single finger end effectors 310, 320, and / or 330 can be representative examples of the end effectors 125.

[0047] And Figure 3Adepicted is field of view space 340. In some examples, field of view space 340 can correspond to an image captured by an imaging device. As shown, field of view space 340 contains end effector 320 of the gripper type, a portion of end effector 310 of the gripper type, and does not contain single finger end effector 330. In some examples, Figure 3A may correspond to an image taken when the operator is controlling end effector 310 and / or 320.

[0048] In some examples, this can create a problem when the operator desires to switch to controlling end effectors 310 and 330 instead of end effectors 310 and 320. For example, because end effector 330 is not within field of view space 340, end effector 330 is not visible in the image of field of view space 340, and the operator can not remember where end effector 330 is located. In some examples, the operator can manually re-center field of view space 340 to place end effectors 310 and 330 within field of view space 340. In some examples, the operator can trigger an automated re-centering by using a method similar to method 200, and designate end effectors 310 and 330 as the end effectors about which the re-centering is to occur.

[0049] Figure 3B field of view space 350 of end effectors 310 and 330 after the re-centering. By using a field of view re-centering movement, the imaging device used to capture end effectors 310 and 330 is repositioned and / or reoriented to contain the pose of end effectors 310 and 330. The re-centering movement changes field of view space 340 from before the field of view re-centering movement to field of view space 350 after the field of view re-centering movement occurs. This field of view re-centering movement results in field of view space 350 containing gripper fingers 312 and 314, pivot joint 316, end point 332, and reference point 334. Field of view space 350 is also centered about gripper fingers 312 and 314, pivot joint 316, end point 332, and reference point 334.

[0050] Figure 4A and Figure 4B are simplified schematic diagrams of an imaging view and a side view, respectively, after a field of view re-centering operation according to some embodiments. Figure 4A and Figure 4B show the use of targets on end effectors 310 and 330 to center field of view space 350 on end effectors 310 and 330. This is in Figure 4AAn image can be shown from the field of view space 350 that can be captured by the imaging device. In some examples, when using a field of view coordinate system, the field of view space 350 can include an x-axis from left to right in the field of view space 350, a y-axis in an upward viewing direction, and a z-axis in a field of view direction.

[0051] To help re-center the end effectors 310 and 330 in the field of view space 350, one or more targets on each end effector 310 and / or 330 are selected. In some embodiments, each target can be associated with the tip of each finger of the end effectors 310 and / or 330, as well as any joints and / or reference points of interest as shown in Figure 4A In some embodiments, other criteria can be used to select the targets, such as only associated targets on the tips of the fingers and / or at other locations on the end effectors 310 and / or 330 and / or associated articulated arms. As shown in Figure 4A Three targets are used on the gripper-type end effector 310, and two targets are used on the single-finger end effector 330, as shown in

[0052] In some examples, each of the targets 412-416 and / or 432-434 can be modeled as a virtual enclosing sphere with a center at the corresponding tip of the finger and / or at or near the center of the corresponding joint and / or reference point. In some examples, the radius of each virtual sphere is large enough to capture at least a volume of the corresponding portion of the end effector associated with the respective target point. In some examples, the radius can be two to three times the volume of the corresponding portion of the end effector, such that the field of view space 350 can capture the corresponding end effector as well as a margin of space around the corresponding end effector. This helps prevent having the end effector placed right on the edge of the field of view space 350. In some examples, the size of the radius can be set to account for kinematic uncertainty in the location of the target point.

[0053] In some examples, a centroid 440 of the center points of each of the targets 412-416 and / or 432-434 can be calculated. The centroid 440 can then be used as a center point for the field of view space 350. The working distance between the centroid 440 and the imaging device can then be adjusted so that the field of view space 350 includes each of the targets 412-416 and / or 432-434.

[0054] Figure 4B A corresponding side view of the field of view space 350 is shown. Figure 4B The side view of the field of view space 350 shows a viewing frustum that widens as the field of view space 350 moves away from the imaging device 450. In some examples, the angular width of the frustum can be determined from the optical properties of the imaging device 450. In some examples, the imaging device 450 can be an endoscope inserted through a cannula 460 into a patient. In some examples, the imaging device 450 can be a stereoscope. In some examples, the cannula 460 can be positioned near a remote center for the imaging device 450 so that roll, pitch, and yaw rotations of the imaging device 450 are centered about the remote center. As shown, the field of view space 350 is oriented with the z-axis of the field of view coordinate system pointing in the direction of the field of view. Figure 4B Further shown, the imaging device 450 is oriented with the centroid 440 in the z-direction of the field of view coordinate system along the direction of the field of view. The centroid 440 can also be located at the average depth in the z-direction of each of the targets 412-416 and / or 332-334. The centroid 440 is also located at the working distance 480 from the tip 470 of the imaging device 450.

[0055] In some embodiments, the working distance 480 can be selected based on one or more criteria. The process begins by determining the centroid 440 and using the direction from a reference point on the imaging device to the centroid 440 as the direction of the field of view or z-axis. In some examples, the reference point can correspond to the cannula 460 when the imaging device is straight between the cannula 460 and the tip 470. In some examples, one or more kinematic models of the imaging device can be used to determine the location of the reference point relative to the cannula 460. In some examples, the reference point can be associated with the tip 470. The maximum x-axis and / or y-axis extent of each of the targets 412-416 and / or 432-434 is then used to determine a respective minimum standoff distance for each of the targets 412-416 and / or 432-434 so that the targets 412-416 and / or 432-434 are within the frustum of the field of view space 350. The maximum minimum standoff distance can then be selected as the working distance 480 so as to ensure that the volume associated with each of the targets 412-416 and / or 432-434 is contained within the field of view space 350. In some examples, when the working distance is specified and is greater than the maximum minimum standoff distance, the working distance 480 can be increased to a preferred working distance for the imaging device 450. In some examples, the working distance 480 can also be constrained to be within the minimum and maximum focal distances of the imaging device 450.

[0056] Once the field of view direction / field of view coordinate system z-axis and working distance 480 are determined, a field of view recentering move for the imaging device 450 can be determined. The field of view recentering move can include adjusting the pitch and yaw of the imaging device 450 to align with the field of view direction, and adjusting the amount of insertion and / or retraction of the tip 470 relative to the sleeve 460 based on the working distance 480. In some examples, the field of view recentering move can be analyzed to determine if it is valid. In some examples, this can include determining if the articulated arm to which the imaging device 450 is attached can perform the field of view recentering move. In some examples, the articulated arm can not be able to perform the field of view recentering move due to joint limits, maximum movement limits placed on the field of view recentering move, and / or collision avoidance with other articulated arms (e.g., the articulated arms 310, 320, and / or 330), patient anatomy, and / or other objects in the workspace. In some examples, the maximum movement limits can include pitch and yaw angle limits that limit the pitch and yaw motion below 30 degrees and / or prevent insertion of the tip 470 beyond its pre-move position. In some examples, the field of view recentering move can be determined to be invalid when any constraints placed on the movement of the imaging device 450 can cause any target to no longer be contained in the frustum of the field of view space 350.

[0057] In some examples, the field of view recentering move can be planned as a multi-step move that includes retracting the imaging device 450 away from the centroid 440, performing a pitch and / or yaw orientation to align with the field of view direction, and then inserting the tip 470 to the working distance 480 from the centroid 440. In some examples, when the field of view recentering move includes zooming in, the multi-step move can include performing a pitch and / or yaw orientation to align with the field of view direction before inserting the tip 470 to the working distance 480 from the centroid 440. In some examples, when the field of view recentering move includes zooming out, the multi-step move can include retracting the imaging device to the working distance 480 before performing a pitch and / or yaw orientation. In some examples, the multi-step move can help reduce the likelihood of collision of the tip 470 with the end effectors of the articulated arms 310, 320, and / or 330, patient anatomy, and / or other objects in the workspace. In some examples, the field of view recentering move can also include rolling the imaging device 450 so that the upward looking direction / field of view coordinate system y-axis aligns with the world coordinate system. In some examples, the field of view recentering move can be determined by using an iterative motion planning operation that optimizes the pitch, yaw, and insertion of the imaging device 450 based on the accuracy limits in the joints that control the articulated arm of the imaging device 450, thereby minimizing orientation and / or positioning errors of the imaging device 450.

[0058] In some embodiments, when the field recentering move is determined to be invalid, an alternative field recentering move is determined in which the tip 470 is retracted to a minimum insertion depth. In some examples, the minimum insertion depth can correspond to a depth beyond which the imaging device can become partially obstructed by one or more portions of the articulated arm used to position and / or orient the imaging device 450. In some examples, the portions of the articulated arm of the imaging device that can partially obstruct the imaging device can correspond to the sleeve 460. In some examples, the minimum insertion depth can correspond to a point, i.e., a predetermined distance from the remote center for the imaging device. In some examples, the predetermined distance can be based on the length of the sleeve 460. In some examples, the predetermined distance can range in length from 2 centimeters to 9 centimeters. As the tip 470 is retracted into the sleeve 460, the field of view direction of the imaging device 450 is then set to point toward the centroid 440. The maximum x-axis and / or y-axis range for each of the targets 412-416 and / or 432-434 is then checked to see if they fall within the field of view space 350. When each of the targets 412-416 and / or 432-434 does not fall within the field of view space 350, the alternative field recentering move is also determined to be invalid. As with the field recentering move, additional checks for the validity of the alternative field recentering move can include determining whether the articulated arm to which the imaging device 450 is attached can perform the alternative field recentering move. In some examples, the articulated arm can not be able to perform the alternative field recentering move due to joint limitations, maximum movement limits placed on the field recentering move, and / or collision avoidance with other articulated arms (e.g., the articulated arms 310, 320, and / or 330) and / or patient anatomy. When the alternative field recentering move is invalid, the field recentering is aborted and an appropriate error is indicated.

[0059] Figure 5 FIG. 5 is a simplified schematic of a method 500 of field recentering in accordance with some embodiments. One or more of the processes 510-580 of the method 500 can be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors (e.g., the processors 140 in the control unit 130), can cause the one or more processors to perform one or more of the processes 510-580. In some embodiments, the method 500 can be performed by an application, such as the motion control application 160. In some embodiments, the method 500 can be used to recenter the end effector 125 and / or one or more of the end effectors 310-330 in the field of view space of an imaging device, such as the imaging device 450, so that corresponding images can be displayed on the display system 180.

[0060] At process 510, a view center point is determined. In some examples, the view center point can correspond to a center of mass of one or more end effectors to be recentered in an image captured by an imaging device, such as imaging device 450. In some examples, the center of mass can be determined by taking a center of mass of one or more targets, such as targets 412-416 and / or 432-434. In some examples, sensors associated with the articulated arms of end effectors 310 and / or 330 can be used to determine the positions of the joints in the articulated arms. These joint positions, in combination with one or more kinematic models of end effectors 310 and / or 330 and their articulated arms, can be used to determine the positions of end effectors 310 and / or 330, which can then be used to determine the center of mass. Figure 3A 、 Figure 3B 、 Figure 4A and Figure 4B In some examples, the center of mass can be determined by taking a center of mass of one or more targets, such as targets 412-416 and / or 432-434. In some examples, sensors associated with the articulated arms of end effectors 310 and / or 330 can be used to determine the positions of the joints in the articulated arms. These joint positions, in combination with one or more kinematic models of end effectors 310 and / or 330 and their articulated arms, can be used to determine the positions of end effectors 310 and / or 330, which can then be used to determine the center of mass.

[0061] At process 520, a working distance is determined. In some examples, the working distance can be determined by determining how far the targets of the end effectors should be so that each target is within the field of view space of the imaging device. In some examples, the working distance can be determined by determining the maximum x-axis and / or y-axis range perpendicular to the field of view direction for each target and then determining the respective minimum view distance for each target so that the target is within the frustum of the field of view space. The maximum minimum view distance can then be selected as the working distance so as to ensure that each target is contained in the field of view space. In some examples, when the working distance is specified and is greater than the maximum minimum view distance, the working distance can be increased to a preferred working distance for the imaging device. In some examples, the preferred working distance can be set by an operator of the imaging device. In some examples, the working distance can also be constrained to be within a minimum focal distance and a maximum focal distance for the imaging device.

[0062] At process 530, a desired imaging device position and orientation is determined. The imaging device is oriented by a vector between a reference point on the imaging device and the view center determined during process 510. In some examples, the reference point can correspond to a remote center when the imaging device is constrained with respect to movement about the remote center, such as the hub 460 of the imaging device 450, when the imaging device is straight between the remote center and the tip of the imaging device. In some examples, one or more kinematic models of the imaging device can be used to determine the position of the reference point. In some examples, the reference point can be associated with the tip of the imaging device. In some examples, the orientation vector can be determined by arranging the tip of the imaging device with the view center while preserving the roll position of the imaging device, and then using the field of view direction of the imaging device as the orientation vector. The position of the tip of the imaging device is then determined based on positioning the tip of the imaging device opposite the field of view direction at a distance from the view center as determined during process 520.

[0063] At process 540, it is determined whether the desired imaging device position and orientation is valid. In some examples, this can include determining whether the articulated arm to which the imaging device is attached can perform a view recentering movement from its current position and orientation to the imaging device position and orientation determined during process 530. In some examples, the articulated arm can not be able to perform the view recentering movement due to joint limits, maximum movement limits placed on the view recentering movement, and / or collision avoidance with other articulated arms, patient anatomy, and / or other objects in the workspace. In some examples, the maximum movement limits can include pitch and yaw angle limits that limit pitch and yaw motion to 30 degrees or less and / or prevent the imaging device from exceeding its pre- movement position insertion. In some examples, the view recentering movement can be determined to be invalid when any constraints placed on movement of the imaging device can cause any targets to no longer be contained in the frustum of the view volume. When the desired imaging device position and orientation is valid, the imaging device is moved to the desired imaging device position and orientation using process 550. When the desired imaging device position and orientation is invalid, an alternative imaging device position and orientation is determined using process 560.

[0064] At process 550, the imaging device is moved. The imaging device is moved by planning appropriate motions for the imaging device and the articulated arms attached to the imaging device, and then executing the planned motions by sending one or more commands to actuators in the articulated arms. In some examples, the motion planning can include a multi-step movement that includes retracting the imaging device away from the center of view point, performing pitch and / or yaw orientations to align the field of view direction, orienting the imaging device toward the center of view point, and then inserting the imaging device to the working distance from the center of view point. In some examples, when the imaging device movement includes zooming in, the multi-step movement can include performing pitch and / or yaw orientations to align the field of view direction before inserting the imaging device to the working distance. In some examples, when the imaging device movement includes zooming out, the multi-step movement can include retracting the imaging device to the working distance before performing pitch and / or yaw orientations. In some examples, the multi-step movement can help reduce the likelihood of end effector collisions of the imaging device with other articulated arms, patient anatomy, and / or other objects in the workspace. In some examples, when the imaging device is to be retracted, as determined during process 560, the insertion step can be omitted. In some examples, the planned motions can also include rolling the imaging device so that the upward looking direction for the imaging device aligns with the world coordinate system. In some examples, the kinematic model of one or more of the articulated arms associated with the imaging device can be used to assist in the motion planning. In some examples, the planned motions can be determined by using an iterative motion planning operation that optimizes the pitch, yaw, and insertion and / or retraction of the imaging device based on joint control accuracy limits associated with the articulated arms to which the imaging device is attached, thereby minimizing orientation and / or positioning errors of the imaging device. Once the imaging device is moved, the recentering operation is complete.

[0065] At process 560, an alternative imaging device position and orientation is determined. When the desired imaging device position and orientation determined during process 540 is invalid, an alternative imaging device position and orientation is determined in which the imaging device is retracted away from the view center point. In some examples, the alternative imaging device position and orientation includes retracting the imaging device to a minimum available insertion depth and ignoring the working distance determined during process 520. In some examples, the minimum insertion depth can correspond to a depth beyond which the imaging device can become partially obstructed by one or more portions of an articulating arm used to position and / or orient the imaging device. In some examples, the portions of the articulating arm that can partially obstruct the imaging device can correspond to a sleeve, such as sleeve 460. In some examples, the minimum insertion depth can correspond to a point a predetermined distance from a remote center for the imaging device. In some examples, the predetermined distance can be based on a length of the sleeve. In some examples, the predetermined distance can be from two to nine centimeters in length. The alternative imaging device orientation then includes orienting the imaging device toward the view center point using a similar approach as used during process 530.

[0066] At process 570, it is determined whether the alternative imaging device position and orientation is valid. In some examples, this can include determining whether the articulating arm to which the imaging device is attached can perform a view center recentering movement from its current position and orientation to the alternative imaging device position and orientation determined during process 560. In some examples, the articulating arm can not be able to perform the view center recentering movement due to joint limits, maximum movement limits placed on the view center recentering movement, and / or collision avoidance with other articulating arms, patient anatomy, and / or other objects in the workspace. In some examples, the maximum movement limits can include pitch and yaw angle limits that limit pitch and yaw motion to 30 degrees or less. When the alternative imaging device position and orientation is valid, the imaging device is moved to the alternative imaging device position and orientation using process 550. When the alternative imaging device position and orientation is invalid, an error is indicated using process 580.

[0067] At process 580, an error is indicated. When the determined and alternative imaging device position and orientation are determined to be invalid, the operator is notified. In some examples, the notification can include any suitable audio and / or visual feedback. In some examples, the audio feedback can include the playing of a unique sound.

[0068] As discussed above and further emphasized here, the Figure 5The examples are merely illustrative and should not be taken as limiting the scope of the claims. Those skilled in the art will recognize many modifications, alternatives and variations. According to some embodiments, additional conditions and / or safety factors can be considered during the method 500 and more specifically during the process 550 when the imaging device is in automated movement.

[0069] In some embodiments, one or more preventative measures can be used to reduce and / or prevent contact and / or interference between the imaging device and the patient's anatomy and / or other obstacles proximate to the imaging device. In some examples, one or more pre-operative and / or intra-operative images of the patient's anatomy can be used to identify one or more no-fly zones that the imaging device should not enter. In some examples, the forces and / or torques on one or more of the joints used to manipulate the imaging device can be monitored using suitable sensors to determine that unexpected forces and / or torques can indicate unacceptable contact of the imaging device with the patient's anatomy and / or other obstacles. In some examples, errors between commanded and actual positions and / or velocities of the imaging device and / or joints used to manipulate the imaging device can be monitored to determine if the errors exceed a configurable threshold. In some examples, the configurable threshold can be different for each joint. In some examples, the errors can be low-pass filtered and / or smoothed to avoid false positive detections that can result from otherwise acceptable temporary conditions. In some examples, one or more contacts located proximate to the distal end of the imaging device can be monitored to determine if the imaging device is in contact with the patient's anatomy and / or other obstacles. In some examples, a determination that the imaging device is contacting and / or interfering with the patient's anatomy can result in an early termination of movement of the imaging device and / or activation of one or more visual and / or audio warnings.

[0070] In some embodiments, one or more interlock devices can be used to ensure that an operator is present to view the re-centering movement. In some examples, one or more input controls such as a head-in sensor can be used to determine that an operator is present at the operator console and in position to view images from the imaging device. In some examples, an illumination sensor can be used to determine that images from the imaging device are being displayed for the operator on a viewer of the operator console. In some examples, a determination by one or more of the interlock devices that the operator is absent and / or that images on the imaging device are lost can result in an early termination of movement of the imaging device and / or activation of one or more visual and / or audio warnings.

[0071] In some embodiments, the motions planned and executed during process 550 can be designed to set upper limits on the velocity and / or acceleration of the imaging device and / or one or more joints used to manipulate the imaging device. In some examples, the velocity and / or acceleration can be limited so that an operator monitoring the recentering motion can have sufficient time to react to potential undesirable motion in the imaging device and override and / or terminate the recentering motion. In some examples, the velocity and / or acceleration can be limited so that positive feed torques in the joints used to manipulate the imaging device are kept at a sufficiently minimum level that allows motion in the imaging device to overcome expected inertia, viscous friction, and / or the like, without allowing movements that can result in excessive forceful contact with the patient's anatomy, other end effectors in proximity to the imaging device, and / or other unexpected obstacles. In some examples, feedback torques in the joints used to manipulate the imaging device can be limited to a minimum sufficient to overcome expected sources of resistance, such as a sterile drape, friction in a sleeve seal, and / or the like.

[0072] Figure 6 FIG. 6 is a simplified schematic of a method 600 of determining a preferred working distance for an imaging device, according to some embodiments. One or more of the processes 610-660 of the method 600 can be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors (e.g., the processor 140 in the control unit 130), can cause the one or more processors to perform one or more of the processes 610-660. In some embodiments, the method 600 can be performed by an application, such as the motion control application 160. In some embodiments, the method 600 can be used to determine a preferred working distance between an imaging device and a view center point. In some examples, the preferred working distance can be the preferred working distance used during the process 520. In some embodiments, the method 600 can be used to monitor manual repositioning operations of an imaging device by an operator to learn a preferred working distance for the operator.

[0073] At process 610, the start of motion for the imaging device is detected. As the operator operates the device having one or more articulated arms and an imaging device, the repositioning movements of the imaging device can be monitored. In some examples, the motion of the imaging device can be associated with the tip of the imaging device, such as tip 470. In some examples, the motion of interest can be associated with manual repositioning of the imaging device by the operator. By monitoring the manual repositioning of the imaging device, it can be possible to learn the preferred distance of the operator between the imaging device and the one or more end effectors captured in images taken by the imaging device. In some examples, each manual repositioning operation can be detected by activation of a repositioning and / or reorientation control for the imaging device. In some examples, upon detection of the start of manual repositioning, the current position and / or orientation of the imaging device can be recorded.

[0074] At process 620, the end of motion for the imaging device is detected. Once motion of the imaging device is detected during process 610, the motion is monitored until its end. In some examples, the end of motion can be detected by a lack of movement in the imaging device. In some examples, the lack of motion can be detected by determining that the velocity of the imaging device falls below a minimum threshold. In some examples, the lack of motion can be detected by determining that the velocity of the imaging device remains below a minimum threshold for a predetermined period of time. In some examples, the end of motion can be associated with the end of manual repositioning, as indicated by deactivation of the repositioning and / or reorientation control. In some examples, upon detection of the end of motion, the current position and / or orientation of the imaging device can be recorded.

[0075] At process 630, it is determined whether sufficient motion was detected in the imaging device. By using the current position and / or orientation values recorded during processes 610 and 620, the amount of motion of the imaging device can be determined. In some examples, the amount of motion can be the distance between the start position and the end position, such as the Euclidean distance. In some examples, the amount of motion can be further based on the angular change between the start orientation and the end orientation. In some examples, the angular change can be converted to a distance by determining the sine and / or cosine of the angular change and multiplying one of them by the distance from the working distance of the imaging device prior to detecting the start of motion during process 610. When the amount of motion exceeds a minimum threshold, such as around 0.5 cm, process 640 begins to determine a new preferred working distance. When the amount of motion does not exceed the minimum threshold, method 600 can return to process 610 to detect future motion in the imaging device.

[0076] At process 640, the z-distances are determined for the points of interest. In some examples, the working distance of the imaging device can be represented based on the perpendicular distance from the imaging device to one or more points of interest along the field of view direction. In some examples, when the points of interest are mapped to the field of view coordinate system of the imaging device, the z-value of each of the points of interest can represent the corresponding z-distance. In some examples, the points of interest can correspond to the center of one or more targets on one or more end effectors. In some examples, the end effectors can be selected by an operator and / or automatically selected based on the end effectors being determined to be visible in the image captured by the imaging device. In Figure 4A and Figure 4B In examples of the target 412-416, 422-426, and / or 432-434, the target can be selected from the group consisting of:

[0077] At process 650, the current working distance is determined. In some examples, the current working distance can be determined by aggregating each of the z-distances determined during process 640. In some examples, the aggregation can include an average, a median, a minimum, a maximum, and / or the like. In some examples, the z-coordinate of the centroid of the points of interest, such as the centroid 440, can be used to determine the current working distance.

[0078] At process 660, the current working distance is aggregated with previous working distance values. The current working distance determined during process 650 is aggregated with previous working distance values to determine a preferred working distance. In some examples, the current working distance determined during process 650 can be weighted based on the amount of motion between the start and end of the motion of the imaging device, such that greater motion has a greater impact on the preferred working distance. In some examples, the aggregation can include determining a moving average, a windowed average over a predetermined period of time, an exponential smoothing, and / or the like. In some examples, the preferred working distance can be initialized to a default value. In some examples, the default value can be based on a minimum focal length and / or a maximum focal length for the imaging device. In some examples, the default value can be set to around 7 cm. In some embodiments, multiple preferred working distances can be determined based on the context of the detected motion. In some examples, the context can include maintaining different preferred working distances for different operators, different procedures, different stages of a procedure, digital zoom settings, focal length settings, stereo disparity settings, and / or the like. Once the aggregation is performed, the method 600 can repeat to include additional motion in the imaging device in the aggregation of the preferred working distance.

[0079] Figure 7is a simplified schematic diagram showing the relationship between the end effectors in the image on the display system and the corresponding input control devices in the console workspace after an input control device recentering operation according to some embodiments. In some examples, the input control device recentering operation can correspond to the input control device recentering that occurs during the method 200 as part of the recentering. In some examples, one of the goals of the recentering operation is to maintain the position and / or orientation harmony between the end effectors in the field of view space of the imaging device during the field of view recentering and the input control devices corresponding to the end effectors. In some examples, the input control device recentering includes changing the position and / or orientation of each input control device to correspond to the position and / or orientation of the respective end effectors.

[0080] Figure 7 the upper portion of Figure 3B and Figure 4A the field of view recentering movement of the end effectors 310 and 330 after they can be captured into the image displayed on the display system 180. The image captured using the imaging device 450 can be displayed on the display system 180 as the image shown within the bounds 710 of the display system 180. For clarity, additional portions of the end effectors 310 and 330 and their articulated arms are shown in Figure 7 even though they would not appear on the display system 180, and any objects that can be partially or fully obstructing the end effectors are also removed from the image. Also shown is a view center point 720 that can correspond to the centroid 440. In some examples, to facilitate the recentering of the input control devices, each of the points of interest on the end effectors 310 and 330 can also be mapped to the field of view coordinate system as depicted by the x v axis, the y v axis, and the z v axis. In some examples, the points of interest can correspond to the targets 412-416 and / or 432-434.

[0081] Figure 7The lower portion shows a console workspace containing input control devices 760 and 770, respectively corresponding to end effectors 310 and 330. Input control devices 760 and 770 can be coupled to the main body 730 of the operator's workbench via their own articulated arms. In some examples, the console workspace can be positioned relative to an armrest 740. In some examples, the operator's workbench can correspond to an operator's workbench 170, and the armrest 740 can correspond to an armrest 190. Because each operator may prefer different heights of the armrest 740, different sizes and lengths of arms, wrists, and / or hands, and / or different preferences for elbow placement and / or flexion, an ergonomic center 750 can be defined within the console workspace. In some examples, the console workspace coordinate system can be as follows: c axis, y c axis and z c The axis is defined as shown.

[0082] In some embodiments, the positional and / or orientational harmony between the end effectors 310 and 330 and the input control devices 760 and 770 can be determined based on a mapping between control points on the input control devices 760 and 770 and corresponding points on the end effectors 310 and 330. More specifically, as Figure 7 As shown in the example, control points 762 and 764 on the finger loops of the input control device 760 can be mapped to targets 412 and 414, respectively, so that the gripping fingers 312 and 314 open and close as the operator opens and closes the distance between control points 762 and 764 during remote operation. Furthermore, control point 766 on the input control device 760 can be mapped to target point 416 so that pivot joint 316 can move accordingly as pivot point 766 moves during remote operation. Similarly, control points 772 and 774 on the input control device 770 can be mapped to targets 432 and 434, respectively.

[0083] To maintain positional and / or orientational harmony between the end effectors 310 and 330 and the input controls 760 and 770, respectively, the input control recentering operation repositions and / or reorients the input controls 760 and 770 around the ergonomic center 750 to approximately correspond to the position and / or orientation of the end effectors 310 and 330 within the field of view space corresponding to the image with boundary 710. Therefore, as Figure 7As shown in FIG. 7, input control device 760 is positioned to the lower left portion of the console workspace and oriented in a right upper direction that matches the position and orientation of end effector 310. Similarly, input control device 770 is positioned in the upper right portion of the console workspace and oriented in a left lower direction that matches the position and orientation of end effector 330. To maintain position and / or orientation harmony, the view and console stereo viewer workspace coordinate systems are generally aligned in the left-right (x c and x v ) directions, the up-down (y c and y v ) directions, and the in-out (z c and z v ) directions. Generally, because the operator hand movements of the input control devices can be translated into corresponding movements of end effectors 310 and / or 330, this provides intuitive operation of end effectors 310 and / or 330 during teleoperation.

[0084] In some embodiments, position and / or orientation harmony between end effectors 310 and 330 and input control devices 760 and 770, respectively, can be maintained by mapping the target points 412-416 and / or 432-434 of end effectors 310 and 330 from the view coordinate system to the console workspace coordinate system, and then using one or more actuators in the articulated arms associated with input control devices 760 and 770 to position and / or orient the corresponding control points 762-766 and / or 772-774 at the mapped locations in the console workspace coordinate system. In some examples, this can be accomplished by using translation and scale transformations. In some examples, one or more translation transformations can be used to map the view center point 720 to the ergonomic center 740. Once the view center point 720 and the ergonomic center 740 are aligned, distances in the view coordinate system can be scaled to corresponding distances in the console workspace coordinate system. In some examples, one or more scale factors used for scaling can be set by an operator of the operator console. In some examples, one or more scale factors can be set based on the relative dimensions of the image boundary 710 and the console workspace. Once each point 312-316 and / or 332-334 of the end effector is mapped to determine the position of the corresponding control point 762-766 and / or 772-774, motion planning for input control devices 760 and 770 can be developed and executed.

[0085] In some embodiments, the positions of each of the control points 762-766 and / or 772-774 can be constrained prior to developing and executing the motion plan. In some examples, the positions and / or orientations of the control points 762-766 and / or 772-774 can be constrained by ranges of motion limits of the joints in the corresponding articulated arms, so as to: maintain a minimum distance and / or a maximum distance between the input control devices 760 and 770; avoid collisions with the arm rest 740 and / or other portions of the operator worktable; prevent left / right cross of the input control devices 760 and 770; avoid undesirable positions and / or orientations of the input control devices 760 and 770; account for the position accuracy (e.g., on the order of 1 cm) of the targets 412-416 and / or 432-434 and / or the control points 762-766 and / or 772-774, and / or the like.

[0086] While Figure 7 not shown in FIG. 8, the front-to-back positioning of the input control devices 760 and / or 770 is matched to the depth of the corresponding end effectors 310 and / or 330. Thus, the z v coordinates of the targets 412-416 and / or 432-434 are correspondingly shifted and scaled to determine the z c coordinates of the control points 762-766 and / or 772-774. Thus, consistent with the side view relationship shown in FIG. 7, the control points 672 and 674 can be positioned closer to the operator than the control points 762-766. Figure 4B

[0087] Figure 8 is a simplified schematic diagram of a method 800 of input control device recentering in accordance with some embodiments. One or more of the processes 810-860 of the method 800 can be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors (e.g., the processors 140 in the control unit 130), can cause the one or more processors to perform one or more of the processes 810-860. In some embodiments, the method 800 can be performed by an application such as the motion control application 160. In some embodiments, the method 800 can be used to recenter one or more of the input control devices 195, 760, and / or 770 in the console workspace to maintain harmony with the positions and / or orientations of the corresponding end effectors 125, 310, 320, and / or 330 as displayed in images captured by an imaging device such as the imaging device 450 and as displayed on the display system 180.

[0088] ​At process 810, the end effector position is determined. In some examples, sensors associated with the articulated arm linked to the end effector can be used to determine the position of a joint within the articulated arm. These joint positions, combined with one or more kinematic models of the articulated arm and the end effector, can be used to determine the end effector position. In some examples, one or more images of the end effector can be used to determine the end effector position. Figure 3A , Figure 3B , Figure 4A , Figure 4B and Figure 7 In the example, the end effector may correspond to end effectors 310 and 330, wherein the positions of end effectors 310 and / or 330 are characterized by targets 412-416 and / or 432-434.

[0089] At process 820, the end effector position is mapped to the field-of-view coordinate system. The field-of-view coordinate system for the imaging device is determined using sensors associated with the articulated arm associated with the imaging device, and one or more kinematic models of the articulated arm associated with the imaging device. The end effector position determined during process 810 is then mapped to the field-of-view coordinate system. This mapping helps determine the x and y positions of the end effector in the image captured by the imaging device, and the z position of the end effector, indicating how far the end effector is from the imaging device in the field-of-view direction. Figure 7 In the example, the end effector position in the field-of-view coordinate system can correspond to the x-axis of target 412-416 and / or 432-434. v y v and z v Coordinate values.

[0090] At process 830, the end effector position is shifted with respect to an ergonomic center. To help maintain position and / or orientation harmony between the end effector and one or more input control devices of the operator console, the field coordinate system is mapped to the console workspace coordinate system. In some examples, the mapping between the field coordinate system and the console workspace coordinate system begins with associating a center point in the field coordinate system with a center point in the console workspace coordinate system. In some examples, the centroid of the end effector position can be selected as the center point in the field coordinate system. In some examples, the ergonomic center of the console workspace can be selected as the center point of the console workspace coordinate system. In some examples, when the origin of the field coordinate system and / or the console workspace coordinate system does not coincide with the selected center point, the two center points can be associated by using one or more transformational transforms. In some examples, the ergonomic center of the console workspace can be pre-selected by an operator of the operator console and / or by the geometry of the operator console and its input control devices. In some examples, the ergonomic center can move when one or more rests, such as arm rests on the console workstation, are repositioned. In some examples, the ergonomic center can be learned by monitoring the operation of the operator workstation as discussed in further detail below. Figure 9 In examples where the operator workstation is a console, the ergonomic center can be learned by monitoring the operation of the operator console as discussed in further detail below. Figure 7 In the example of FIG. 7, process 830 corresponds to aligning the centroid 720 with the ergonomic center 750.

[0091] At process 840, the end effector position is scaled with respect to the ergonomic center to determine control point positions. Because the scale of the field coordinate system and the console workspace coordinate system are typically different, the position of the end effector in the field coordinate system relative to the center point in the field coordinate system is scaled with respect to the ergonomic center in the console workspace coordinate system. The scaling converts the relative distance between the end effector position and the center point in the field coordinate system into a corresponding relative distance between the input control device position and the ergonomic center in the console workspace coordinate system. Each scaled point from the field coordinate system then becomes a control point in the console workspace coordinate system. In some examples, one or more scale factors used for scaling can be set by an operator of the operator workstation. In some examples, one or more scale factors can be set based on the respective dimensions of the images captured in the field coordinate system and the dimensions of the console workspace. In some examples, the scale factors can be set by an operator of the operator workstation. In some examples, the scale factors can be set by an operator of the operator workstation. Figure 7 In the example of FIG. 7, the scaling of process 840 converts the respective x v , y v , and z v distances into x c , y c , and z cdistances so that the positions of the targets 412-416 and / or 432-434 are converted to the positions of the control points 762-766 and / or 772-774, respectively.

[0092] At process 850, the control point positions are constrained. In some examples, the mapping of points associated with the end effector positions in the field of view coordinate system to the control point positions in the control console workspace coordinate system can not result in suitable positions and / or orientations for the input control devices, such as the input control devices 195, 760, and / or 770. In some embodiments, the position of each control point mapped during processes 830 and / or 940 can be constrained. In some examples, the position and / or orientation of a control point can be constrained by the range of movement limitations of the joints in the corresponding articulated arm, so as to: maintain a minimum distance and / or a maximum distance between control points of different input control devices; avoid collisions with arm rests and / or other parts of the operator's workstation; prevent left / right cross-over of input control devices; avoid undesirable positions and / or orientations of input control devices; account for limitations on the position accuracy of points of the end effector and / or control points of the input control devices (e.g., on the order of 1 cm), and / or the like.

[0093] At process 860, the input control devices are moved to the control point positions. Using one or more kinematic models of the articulated arms associated with the input control devices, a motion plan is determined for moving the control points on the input control devices from their previous positions to the control point positions determined using processes 830-850. In some examples, when the desired motion of an input control device and the control point position can result in a collision and / or a near-collision between the articulated arms associated with the input control device, the motion plan can include a multi-segment plan with intermediate control position points that avoid the collision and / or near-collision. The motion plan can then be implemented by sending one or more commands to the actuators associated with the articulated arms. In some examples, when no suitable motion plan can be determined, an error is indicated.

[0094] Figure 9is a simplified schematic of a method 900 of determining an ergonomic center for an input control device according to some embodiments. One or more of the processes 910-950 of the method 900 can be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that, when run by one or more processors (e.g., the processors 140 in the control unit 130), can cause the one or more processors to perform one or more of the processes 910-950. In some embodiments, the method 900 can be performed by an application such as the motion control application 160. In some embodiments, the method 900 can be used to determine an ergonomic center for one or more input control devices in a console workspace. In some embodiments, the method 900 can be used to monitor manual repositioning operations of input control devices to learn a preferred ergonomic center for an operator.

[0095] At process 910, a start of an input control device repositioning movement is detected. During operation of a teleoperational device using an operator workbench, an operator can periodically reposition one or more input control devices to a more comfortable and / or ergonomic position. In some examples, this can be triggered by the operator engaging a clutch, where the clutch disengages movement of an end effector being remotely operated by the respective input control device from the input control device. In some examples, detection of engagement of the clutch indicates a start of an input control device repositioning movement. In some examples, when the start of an input control device repositioning movement is detected, a current position and / or orientation of the input control device can be recorded for one or more control points of the input control device.

[0096] At process 920, each input control device repositioning movement is detected. When the operator completes an input control device repositioning movement, the clutch is disengaged, and remote operation of the articulated arm and end effector is resumed. In some examples, detection of disengagement of the clutch indicates an end of an input control device repositioning movement. In some examples, when the end of an input control device repositioning movement is detected, a current position and / or orientation of the input control device can be recorded based on one or more control points of the input control device.

[0097] At process 930, it is determined whether sufficient motion was detected in the input control device between the start and end of the input control device repositioning movement. Using the current position and / or orientation values recorded during processes 910 and 920, the amount of motion of the input control device can be determined. In some examples, the amount of motion can be a distance between the start and end positions, such as a Euclidean distance. In some examples, the amount of motion can be a set of one or more distances between the start and end positions of one or more control points. In some examples, the set can be a sum, a weighted sum, an average, and / or the like. When the amount of motion exceeds a minimum threshold, such as around 2 cm, the input control device center is determined at process 940. When the amount of motion does not exceed the minimum threshold, the method 900 returns to process 910 to detect future input control device repositioning movements.

[0098] At process 940, the input control device center is determined. Using the end position of the input control device recorded during process 920, the center of the input control device is determined. In some examples, the center of the input control device can be determined using a set of the end positions of one or more control points of the input control device, such as a centroid.

[0099] At process 950, the input control device center is added to a set of previous input control device centers. The input control device center determined during process 940 is added to a set of previous input control device centers to determine an ergonomic center. In some examples, the input control device center determined during process 940 can be weighted based on the amount of motion between the start and end of the input control device repositioning movement so that larger movements have a greater impact on the ergonomic center. In some examples, the set can include determining a moving average, a windowed average over a predetermined period of time, an exponential smoothing, and / or the like. In some examples, the ergonomic center can be initialized to a default value. In some examples, the default value can be based on the geometry of the input control device, the console workspace, and / or the expected physiology of the operator. In some embodiments, multiple ergonomic centers can be determined based on the context of the detected motion. In some examples, the context can include maintaining different ergonomic centers for different operators, different procedures, different stages of a procedure, different end effectors being remotely operated by the input control device, and / or the like. Once the set is performed, the method 900 can repeat to include additional input control device repositioning movements in the collection of ergonomic centers. In some examples, the ergonomic center can be adjusted to account for the position of one or more rests in the console workspace, such as arm rests.

[0100] Figure 10is a simplified schematic of a method 1000 of controlling an imaging device according to some embodiments. One or more of the processes 1005-1050 of the method 1000 can be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that, when run by one or more processors (e.g., the processors 140 in the control unit 130), can cause the one or more processors to perform one or more of the processes 1005-1050. In some embodiments, the method 1000 can be performed by an application such as the motion control application 160. In some embodiments, the method 1000 can be used to combine manual control of an imaging device (such as the imaging device 450) using one or more input control devices in a control console workspace with automated re-centering of the imaging device. In some embodiments, variations in the processes are all possible. In some examples, the processes 1020-1035 can be performed in a different order and / or substantially in parallel.

[0101] At the process 1005, activation of an imaging device motion mode is detected. In some examples, an operator of the electronic device can manually trigger activation of the imaging device motion mode using one or more input control devices such as switches, buttons, pedals, levels, voice recognition, and / or the like. In some examples, a request can be issued as a temporal input to trigger the imaging device motion mode and / or as a continuous input to activate the imaging device motion mode.

[0102] At the process 1010, the imaging device motion mode is entered. In some examples, operator control of one or more end effectors is suspended prior to entering the imaging device motion mode. In some examples, one or more motion input control devices such as one or more master control devices 195 can be decoupled from control of the one or more end effectors. In some examples, decoupling can occur due to a limited number of operator controls for controlling devices attached to the distal end of an articulated arm and / or due to limiting operator control and / or the ability to remotely operate one or more of the end effectors of the electronic device. Suspension of control by the operator allows the imaging device to move without interference from motion of the one or more end effectors commanded by the operator.

[0103] At process 1015, it is determined whether one or more of the motion input controls are being used. In some examples, a timeout period can begin when the imaging device motion mode is entered during process 1010. During the timeout period, the one or more motion input controls can be monitored to determine whether the operator is attempting to use the one or more motion input controls to manually control the position and / or orientation of the imaging device. In some examples, the timeout period can have a configurable length, such as around 0.5 seconds. In some examples, use of the one or more motion input controls can be determined based on whether the operator moves the one or more motion input controls more than a threshold distance, whether the operator rotates the one or more motion input controls through more than a threshold angle, and / or some combination of both. In some examples, the threshold distance can be 5-10 mm. In some examples, the threshold angle can be 5 degrees or more. When the timeout period ends without use of the one or more motion input controls, recentering begins from process 1020. When use of the one or more input controls is detected during the timeout period, manual control of the imaging device begins with process 1040.

[0104] At process 1020, recentering of the imaging device is performed. In some examples, processes similar to processes 510-580 of method 500 can be used to perform recentering of the imaging device during process 1020. In some examples, when the imaging device is being recentered during process 1020, the one or more motion input controls can be automatically moved to maintain position and / or orientation harmony between the one or more motion input controls and the imaging device. In some examples, processes similar to processes 810-860 of method 800 can be modified to maintain position and / or orientation harmony between the one or more motion input controls and the imaging device, where the position and / or orientation of the imaging device is being replaced by the position and / or orientation of the end effector.

[0105] At process 1025, it is determined whether one or more of the motion input control devices is being used. In some examples, use of one or more motion input control devices can correspond to deliberate motion of the one or more motion input control devices by the operator and / or sufficient resistance to change in position and / or orientation of the one or more motion input control devices by the operator when the position and / or orientation between the one or more motion input control devices and the imaging device is being maintained. In some examples, deliberate motion can be detected by using a similar method to that used during process 1015. In some examples, resistance by the operator can be detected by determining a difference between commanded position and / or orientation and actual position and / or orientation of the motion input control device that exceeds a threshold distance and / or a threshold angle. In some examples, the threshold distance can be around 1 cm to 3 cm. In some examples, the threshold angle can be 5 degrees or higher. When use of the one or more motion input control devices is detected, manual control of the imaging device begins with process 1040. When use of the one or more input control devices is not detected, recentering continues with process 1030.

[0106] At process 1030, it is determined whether recentering is complete. The recentering performed by process 1020 is monitored to determine whether the motion planned as part of the recentering is complete with the imaging device having the desired pose. When recentering is complete, manual control of the imaging device begins with process 1040. When recentering is not complete, recentering begins with process 1035.

[0107] At process 1035, it is determined whether deactivation of the imaging device motion mode is detected. In some examples, the operator can indicate deactivation of the imaging device motion mode by using one or more input control devices such as a switch, a button, a pedal, a level, voice recognition, and / or the like. In some examples, when activation of the imaging device motion mode was activated during process 1005 by using an instantaneous input, a compensatory instantaneous input can be used to deactivate the imaging device motion mode. In some examples, when activation of the imaging device motion mode was activated during process 1005 by using a continuous input, removal of the continuous input such as removing a foot from a pedal can be used to deactivate the imaging device motion mode. In some examples, one or more of a precaution, a safety feature, and / or an interlock associated with method 500 and / or process 550 can be used to determine that deactivation of the imaging device motion mode should occur. When deactivation of the imaging device motion mode is not detected, recentering continues by repeating processes 1020-1035. When deactivation of the imaging device motion mode is detected, the imaging device motion mode is exited using process 1050.

[0108] At process 1040, the imaging device is controlled based on the motion input control devices. In some examples, the motion input control devices can be used to manually control the position and / or orientation of the imaging device. In some examples, the imaging device can be moved to maintain a position and / or orientation harmony between the one or more motion input control devices and the imaging device. In some examples, the motion input control devices can be used to remotely operate the imaging device by reflecting changes in the position and / or orientation of the motion input control devices to corresponding changes in the position and / or orientation of the imaging device. In some examples, one or more kinematic models of the motion input control devices, the imaging device, and / or the articulated arm to which the imaging device is attached can be used to translate changes in the motion input control devices to corresponding changes in the imaging device. In some examples, the one or more kinematic models can be used to determine one or more coordinate transformation matrices that map changes in the motion input control devices to corresponding changes in the imaging device. In some examples, the coordinate transformation matrices can implement one or more shift transformations and / or scale transformations. In some examples, changes in the position and / or orientation of the imaging device can be performed by sending one or more commands to actuators in the articulated arm to which the imaging device is attached.

[0109] At process 1045, it is determined whether deactivation of the imaging device motion mode is detected. Using a process similar to process 1035, it is determined whether to exit the imaging device motion mode. When deactivation of the imaging device motion mode is not detected, manual control of the imaging device continues by repeating process 1040. When deactivation of the imaging device motion mode is detected, the imaging device motion mode is exited using process 1050.

[0110] At process 1050, the imaging device motion mode is exited. When the imaging device motion mode is deactivated during processes 1035 and / or 1045, the imaging device motion mode is exited. In some examples, when the imaging device motion mode is exited, any motion of the imaging device due to recentering of process 1020 is ended and the one or more motion input control devices are disengaged from controlling the position and / or orientation of the imaging device. In some examples, when the imaging device motion mode is exited, manual control and / or recentering control of the imaging device is ended. In some examples, when the imaging device motion mode is exited, the electronic device can return to a mode in which the one or more motion input control devices become dormant and / or resume control of one or more end effectors of the electronic device.

[0111] Some examples of control units, such as control unit 130, can include a non-transitory, tangible, machine-readable medium having executable code that, when run by one or more processors (e.g., processor 140), can cause the one or more processors to perform the processes of methods 200, 500, 600, 800, 900, and / or 1000. Some common forms of machine-readable media that can include the processes of methods 200, 500, 600, 800, 900, and / or 1000 are, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tapes, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other medium from which a processor or computer is adapted to read.

[0112] While illustrative embodiments have been shown and described, a wide range of modifications, changes and substitutes are foreseen in the foregoing disclosure, and in some cases, some features of the embodiments can be employed without others. Those of ordinary skill in the art will appreciate variations, modifications, and alternatives. Thus, the full scope of the present application should be determined by the appended claims, and appropriately, the claims should be construed broadly and in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A computer-assisted medical device comprising: an input control device for remotely operating an imaging device; and a control unit comprising one or more processors coupled to the imaging device and the input control device; wherein the control unit is configured to: detect a start of a repositioning movement for the imaging device; detect an end of the repositioning movement; determine a current working distance based on one or more first distances between the imaging device and one or more targets associated with one or more end effectors supported by the computer-assisted medical device and within a viewing volume of the imaging device at the end of the repositioning movement, the one or more first distances being measured in a field of view direction of the imaging device; aggregate the current working distance with a previous working distance determined from a previous repositioning movement for the imaging device to determine a preferred working distance for the imaging device; determine a viewing center point of the imaging device; and determine a desired imaging device position and orientation based on the viewing center point and the preferred working distance.

2. The computer-assisted medical device of claim 1, wherein to determine the current working distance, the control unit is configured to: determine a second distance between the imaging device and a center of mass of the one or more targets.

3. The computer-assisted medical device of claim 1, wherein a first target of the one or more targets is associated with a tip, a joint, or a reference point of a first end effector of the one or more end effectors.

4. The computer-assisted medical device of claim 1, wherein the control unit is further configured to: identify the one or more end effectors based on an operator selection or based on whether the end effectors are visible in one or more images captured by the imaging device.

5. The computer-assisted medical device of any one of claims 1-4, wherein the control unit is further configured to determine whether a sufficient motion of the imaging device occurred between the start and the end of the repositioning movement.

6. The computer-assisted medical device of claim 5, wherein to determine whether a sufficient motion of the imaging device occurred between the start and the end of the repositioning movement, the control unit is configured to: determine a distance between a position of the imaging device at the start of the repositioning movement and a position of the imaging device at the end of the repositioning movement or an angular change between an orientation of the imaging device at the start of the repositioning movement and an orientation of the imaging device at the end of the repositioning movement.

7. The computer-assisted medical device of any one of claims 1-4, wherein the control unit is further configured to: weight the current working distance based on an amount of motion of the imaging device between the start and the end of the repositioning movement. ​ 8. The computer-assisted medical device of any of claims 1-4, wherein to detect an end of the repositioning movement, the control unit is configured to: determine that a velocity of the imaging device is below a threshold; or determine that a velocity of the imaging device remains below a threshold for a predetermined period of time.

9. A machine-readable medium comprising a plurality of machine-readable instructions adapted to cause one or more processors associated with a medical device to perform a method when executed by the one or more processors, the method comprising: detecting a start of a repositioning movement of an imaging device for the medical device; detecting an end of the repositioning movement; determining a current working distance based on one or more first distances between the imaging device and one or more targets associated with one or more end effectors supported by the medical device and within a field of view of the imaging device at the end of the repositioning movement, the one or more first distances measured in a direction of view of the imaging device; collecting the current working distance with a previous working distance determined from a previous repositioning movement of the imaging device to determine a preferred working distance for the imaging device; determining a view center point for the imaging device; and determining a desired imaging device position and orientation based on the view center point and the preferred working distance.

10. The machine-readable medium of claim 9, wherein determining the current working distance comprises: determining a second distance between the imaging device and a centroid of the one or more targets.

11. The machine-readable medium of claim 9 or 10, further comprising: weighting the current working distance based on an amount of motion of the imaging device between the start and the end of the repositioning movement. ​

Images