System and method for recentering an imaging device and an input control device

Abstract

Systems and methods for recentering an imaging device and an input control device include a medical device having one or more end effectors, an imaging device, one or more input control devices for remotely operating the one or more end effectors, and a control unit including one or more processors coupled to the end effectors, the imaging device, and the input control devices. The control unit, in response to a recentering request, suspends remote operating control of the end effectors by the input control devices, determines a field of view recentering movement for the imaging device so that the end effectors are contained within a field of view space of the imaging device, determines one or more input control device recentering movements to provide a position and orientation harmony between each of the input control devices and a corresponding end effector of the end effectors, executes the field of view recentering movement and the input control device recentering movements, and resumes remote operating control of the end effectors by the input control devices.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 202211259251.2, filed on March 17, 2015, entitled "System and method for recentering an imaging device and an input control device". The aforementioned application is a divisional application of Chinese Patent Application No. 201910658710.6, filed on March 17, 2015, entitled "System and method for recentering an imaging device and an input control device". The aforementioned application is a divisional application of Chinese Patent Application No. 201580023718.5 (PCT / US2015 / 021105), filed on March 17, 2015, entitled "System and method for recentering an imaging device and an input control device".

[0002] Related applications This disclosure claims priority to U.S. Provisional Patent Application No. 61 / 954,191, filed March 17, 2014, entitled “System and Method for Recentering Imaging Devices and Input Controls,” which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to the remote operation of devices with articulated arms and more specifically to the recentering of imaging devices and input control devices. Background Technology

[0004] More and more devices are being replaced by autonomous and semi-autonomous electronic devices. This is especially true in hospitals today, where large arrays of autonomous and semi-autonomous electronic devices are found in operating rooms, interventional suites, intensive care units, emergency rooms, and more. For example, glass and mercury thermometers are being replaced by electronic thermometers, intravenous drip lines now include electronic monitors and flow regulators, and traditional handheld surgical instruments are being replaced by computer-aided medical devices.

[0005] These electronic devices offer both advantages and challenges to the operators. Many of these devices can enable autonomous or semi-autonomous movement of one or more articulated arms and / or end effectors. It is also common practice to operate the electronics remotely via one or more input controls on an operator's workbench to control the movement and / or operation of the articulated arms and / or end effectors. When the electronics are remotely operated from the operator's workbench and / or the end effector is being used in an area not directly visible to the operator, such as during computer-assisted surgery when the end effector is hidden by the patient's anatomy, the electronics may include an imaging device that captures the region of interest and displays it to the operator using a display system. When the operator controls the articulated arms and / or end effectors, the operator typically attempts to keep the end effectors within the sight of the imaging device so that the operation of the end effectors can be observed on the display system. Furthermore, the position and orientation of the input controls are typically matched to the end effector so that when the input controls move, the end effector “follows” those movements.

[0006] When the imaging device and / or end effector moves, it is possible that the operator may lose sight of one or more end effectors and / or lose tracking of the spatial relationship between the imaging device and the end effector. This can be further complicated when the operator switches control of the electronics to additional articulated arms and / or end effectors that can stop in other areas around the region of interest, and / or when the end effector is partially or completely obstructed by other objects in the region of interest. To regain visualization of the end effector (i.e., to place the end effector within the view volume of the imaging device), the operator may have to perform a series of recentering movements with respect to the imaging device to find the appropriate pose (position and orientation) of the imaging device, including the end effector. This series of movements can become cumbersome, time-consuming, and / or impractical.

[0007] Furthermore, the spatial orientation between the imaging device and the end effector can change when the imaging device moves and / or the input control device switches to the additional articulated arm and / or end effector. This can result in a discrepancy between the position and / or orientation of the end effector as displayed by the display system and the corresponding position and / or orientation of the input control device for those end effectors. In some cases, this can be corrected by the operator by activating the clutch for the input control device and then repositioning and / or reorienting the input control device to match the position and / or orientation of the end effector as displayed on the display system. Similar to the movement of the imaging device, these repositioning and / or reorienting operations can also become cumbersome, time-consuming, and / or impractical.

[0008] Therefore, improved methods and systems for visually reacquiring the end effector and / or repositioning and / or reorienting input control devices to match the end effector are desired. Summary of the Invention

[0009] Consistent with some embodiments, a computer-assisted medical device includes one or more end effectors, an imaging device, one or more input controls for remotely operating the one or more end effectors, and a control unit including one or more processors coupled to the end effectors, the imaging device, and the input controls. In response to a recentering request, the control unit suspends remote operational control of the end effectors by the input controls, determines a recentering movement for the imaging device's field of view so that the end effector is included within the imaging device's view space, determines recentering movements by the one or more input controls to provide positional and orientational harmony between each input control and its corresponding end effector, performs the recentering movement and the input control recentering movement, and resumes remote operational control of the end effectors by the input controls.

[0010] Consistent with some embodiments, a method of controlling movement in a medical device includes: suspending remote operational control of one or more input controls of the medical device on one or more end effectors of the medical device in response to a recentering request; determining a field-of-view recentering movement for the imaging device such that the end effector is included within the field-of-view space of the imaging device of the medical device; determining a recentering movement of one or more input controls to provide positional and orientational harmony between each of the input controls and the corresponding end effector of the end effector; performing the field-of-view recentering movement and the input control recentering movement; and resuming remote operational control of the input controls on the end effector.

[0011] Consistent with some embodiments, a method of controlling movement in a medical device includes: suspending remote operational control of one or more input controls of the medical device on one or more end effectors of the medical device in response to a recentering request; determining a field-of-view recentering movement for the imaging device such that the end effector is included within the field-of-view space of the imaging device of the medical device, performing the field-of-view recentering movement, and resuming remote operational control of the end effector by the input controls.

[0012] Consistent with some embodiments, a method for determining a preferred working distance of an imaging device for a medical device includes: detecting the start of a repositioning movement of the imaging device for the medical device; detecting the end of the repositioning movement; determining a current working distance based on a first distance between the imaging device and one or more targets associated with one or more end effectors of the medical device within the viewing volume of the imaging device at the end of the repositioning movement, the first distance being measured in the view direction of the imaging device; and combining the current working distance with a previously obtained current working distance to determine the preferred working distance.

[0013] Consistent with some embodiments, a method of controlling movement in a medical device includes: suspending remote operational control of one or more input controls of the medical device on one or more end effectors of the medical device in response to a recentering request; determining a recentering movement of the one or more input controls to provide positional and orientational harmony between each of the input controls and the corresponding end effector of the end effector; performing the recentering movement of the input controls; and resuming remote operational control of the input controls on the end effector.

[0014] Consistent with some embodiments, a method for determining the ergonomic center of an operator's workbench for a medical device includes: detecting the start of a repositioning movement of one or more input controls for the medical device; detecting the end of the repositioning movement; determining the position of one or more control points associated with the input controls at the end of the repositioning movement; aggregating the positions to determine a center point of the input controls; and aggregating the center point of the input controls with a previously obtained center point of the input controls to determine the ergonomic center.

[0015] Consistent with some embodiments, a non-transitory machine-readable medium includes a plurality of machine-readable instructions. When executed by one or more processors associated with a medical device, the machine-readable instructions cause the one or more processors to perform a method. The method includes: suspending remote operational control of one or more input controls of the medical device on one or more end effectors of the medical device in response to a recentering request; determining a field-of-view recentering movement for the imaging device such that the end effector is included within the field-of-view space of the imaging device of the medical device; determining a recentering movement of one or more input controls to provide positional and orientational harmony between each of the input controls and the corresponding end effector of the end effector; performing the field-of-view recentering movement and the input control recentering movement; and resuming remote operational control of the input controls on the end effector.

[0016] Consistent with some embodiments, a method for controlling the movement of an imaging apparatus coupled to a medical device includes: detecting the activation of a motion mode of the imaging apparatus; and determining whether one or more motion input control devices are being used. When the one or more motion input control devices are being used, the posture of the imaging apparatus is controlled based on the one or more motion input control devices. When the one or more motion input control devices are not used for a timeout period, the imaging apparatus is recentered. Recentering the imaging apparatus includes: determining a field-of-view recentering movement for the imaging apparatus such that one or more end effectors of the medical device are included within the field of view of the imaging apparatus; and performing the field-of-view recentering movement. Attached Figure Description

[0017] Figure 1 This is a simplified schematic diagram of a computer-aided system according to some embodiments.

[0018] Figure 2 This is a simplified schematic diagram of a method for recentering an end effector and an input control device according to some embodiments.

[0019] Figure 3A and Figure 3B This is a simplified schematic diagram of the imaging view before and after a view recentering operation, according to some embodiments.

[0020] Figure 4A and Figure 4B These are simplified schematic diagrams of the imaging view and side view after a field-of-view recentering operation, according to some embodiments.

[0021] Figure 5 This is a simplified schematic diagram of a method for recentering the field of view according to some embodiments.

[0022] Figure 6 This is a simplified schematic diagram of a method for determining a preferred working distance for an imaging apparatus according to some embodiments.

[0023] Figure 7 This is a simplified schematic diagram illustrating the relationship between the end effector in the image on the display system and the corresponding input control device in the console workspace after an input control device recentering operation, according to some embodiments.

[0024] Figure 8 This is a simplified schematic diagram of a method for recentering an input control device according to some embodiments.

[0025] Figure 9This is a simplified schematic diagram of a method for determining the ergonomic center / human-machine control center of an input control device according to some embodiments.

[0026] Figure 10 This is a simplified schematic diagram of a method for controlling an imaging device according to some embodiments.

[0027] In the accompanying drawings, elements with the same name have the same or similar functions. Detailed Implementation

[0028] In the following description, specific details of some embodiments consistent with this disclosure are set forth. However, it will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative and not restrictive. Those skilled in the art will recognize other elements that, while not specifically described herein, are also within the scope and spirit of this disclosure. Furthermore, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless otherwise specifically described, or if one or more features would render the embodiment nonfunctional.

[0029] Figure 1 This is a simplified schematic diagram of a computer-aided system 100 according to some embodiments. Figure 1 As shown, the computer-assisted system 100 includes a device 110 having one or more movable arms 120 or articulated arms 120. Each of the one or more articulated arms 120 may support one or more end effectors 125. In some embodiments, the device 110 may be consistent with a computer-assisted surgical apparatus. The one or more end effectors 125 may include surgical instruments, imaging devices, and / or the like. In some examples, surgical instruments may include forceps, grippers, retractors, cauterization tools, suction tools, suture devices, and / or the like. In some examples, imaging devices may include endoscopes, cameras, stereoscopic devices, and / or the like.

[0030] Device 110 is coupled to control unit 130 via an interface. The interface may include one or more cables, connectors, and / or buses, and may further include one or more networks with one or more network switching and / or routing devices. Control unit 130 includes a processor 140 coupled to memory 150. Operation of control unit 130 is controlled by processor 140. While control unit 130 is shown with only one processor 140, it should be understood that processor 140 may represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and / or the like within control unit 130. Control unit 130 may be implemented as a separate subsystem and / or board added to a computing device or implemented as a virtual machine.

[0031] Memory 150 may be used to store software executed by control unit 130 and / or one or more data structures used during operation of control unit 130. Memory 150 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, floppy hard disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper cards, any other physical media with perforations, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media suitable for a processor or computer to read from.

[0032] As shown in the figure, memory 150 includes a motion control application 160 that can be used to support autonomous and / or semi-autonomous control of device 110. Motion control application 160 may include one or more application programming interfaces (APIs) for receiving position, motion, and / or other sensor information from device 110, exchanging position, motion, and / or collision avoidance information with other control units, and / or planning and / or assisting in planning the motion of device 110, its articulated arm 120, and / or end effector 125. While motion control application 160 is depicted as a software application, it can be implemented using hardware, software, and / or a combination of both.

[0033] The control unit 130 may be further coupled to the operator workbench 170 via an interface. The operator workbench 170 may be used by an operator (such as a surgeon) to control the movement and / or operation of the articulated arm 120 and the end effector 125. To support the operation of the articulated arm 120, the operator workbench 170 includes a display system 180 for displaying images of at least a portion of one or more of the articulated arm 120 and / or the end effector 125. For example, the display system 180 may be used when the operator sees that the articulated arm 120 and / or the end effector 125 in use is impractical and / or impossible. The operator workbench 170 may further include a console workspace with one or more input controls or a master control 195, which may be used to operate the device 110, the articulated arm 120, and / or the end effector 125. Each of the input controls 195 may be coupled to the distal end of its own articulated arm so that movement of the input control 195 can be detected by the operator workbench 170 and transmitted to the control unit 130. To provide improved ergonomics, the console workspace may also include one or more supports, such as arm supports 190, on which the operator can rest their arm while manipulating the input controls 195. In some examples, the display system 180 and the input controls 195 may be used by the operator to remotely operate the articulated arm 120 and / or the end effector 125. In some embodiments, the device 110, operator workbench 170, and control unit 130 may correspond to the da Vinci® surgical system, commercially available from Intuitive Surgical, Inc., Sunnyvale, California.

[0034] In some embodiments, other configurations and / or constructions may be used with the computer-assisted system 100. In some examples, the control unit 130 may be included as part of the operator workbench 170 and / or device 110. In some embodiments, the computer-assisted system 100 may be found in an operating room and / or interventional suite. And while the computer-assisted system 100 includes only one device 110 with two articulated arms 120, those skilled in the art will understand that the computer-assisted system 100 may include any number of devices with articulated arms and / or end effectors of similar and / or different designs from device 110. In some examples, each device may include fewer or more articulated arms 120 and / or end effectors 125.

[0035] Figure 2This is a simplified schematic diagram of a method 200 for recentering an end effector and an input control device according to some embodiments. One or more of the processes 210-280 of method 200 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when run by one or more processors (e.g., processor 140 in control unit 130), causes one or more processors to execute one or more of processes 210-280. In some embodiments, method 200 may be executed by an application (such as motion control application 160). In some embodiments, method 200 may be used to recenter one or more end effectors 125 in an image captured by an imaging device and displayed on a display system 180, and / or to recenter one or more input control devices 195 in a console workspace, such that the position and / or orientation of the input control devices 195 corresponds to the position and / or orientation of the end effectors 125 displayed in the image.

[0036] At process 210, a recentering request is detected. In some examples, the operator of the electronic device may manually trigger the request to recenter using one or more input control devices such as switches, pedals, levels, voice recognition, and / or the like. In some examples, the request may be issued as a temporary input to trigger recentering and / or as a continuous input to activate recentering until it is completed and / or the input is withdrawn. In some examples, the recentering request may be initiated automatically in response to a change in system state. In some examples, a change in system state may include a change in the association between the input control device and the remotely operated end effector. In some examples, a change in system state may include a change in the association between the input control device and the remotely operated end effector, wherein one or more end effectors are detected to be outside the field of view of the imaging device. In some examples, a change in system state may include a change in the mode of the imaging device that causes one or more end effectors to be outside the field of view of the imaging device (e.g., a change in digital zoom, a change in the far-angle view, and / or similar). In some examples, the recentering request may also include the specification of the articulated arm and the end effector to be recentered. In some examples, the detection of a recentering request can be confirmed through appropriate feedback to the operator, such as a distinctive voice, messages on the console, indicators, and / or the like.

[0037] At process 220, operator control of one or more end effectors is suspended. The ability of one or more end effectors to be controlled by operators and / or remotely operated electronic devices is suspended before the relocation center can begin. Suspension of operator control allows relocation center operation to continue without interference from movement commands from operators.

[0038] At process 230, the desired recentering movement of the field of view is determined. This recentering movement is determined using, for example, sensed joint positions in the articulated arm and the end effector coupled to the articulated arm, and one or more kinematic models of the articulated arm and the end effector. In some examples, this may include determining the pose (e.g., position and / or orientation) of one or more end effectors of interest associated with the controlled electronics. In some examples, each determined pose may be mapped to a common coordinate system, such as the world coordinate system and / or the view coordinate system. Using the geometry of the pose for a preferred working distance of the imaging apparatus and knowledge of that preferred working distance, the desired pose for placing the end effector within the field of view space of the imaging apparatus is determined. The pose of the imaging apparatus and one or more kinematic models can then be used to determine the desired recentering movement of the imaging apparatus.

[0039] At process 240, the desired input control device recentering movement is determined. The pose for the end effector determined during process 230 can be mapped to the coordinate system of the console workspace where the input control device corresponding to the end effector is located. The pose can be mapped using knowledge of the preferred ergonomic center of the console workspace and a scaling factor between the distance of the workspace used by the end effector and the distance of the console workspace containing the input control device. The mapped pose and one or more kinematic models of the input control device can then be used to determine the corresponding input control device recentering movement. In some embodiments, two input control device recentering movements are determined, one corresponding to the left input control device associated with the first end effector of the end effector and the other corresponding to the right input control device associated with the second end effector of the end effector. In some embodiments, corresponding recentering movements of other numbers of input control devices may also be determined.

[0040] At process 250, it is determined whether the field-of-view recentering movement and / or the input control device recentering movement are valid. The validity of the desired recentering movement for the imaging device is determined using a kinematic model of the imaging device and the desired recentering movement for the imaging device determined during process 230. In some examples, this validity determination may include examining one or more constraints on the following factors: movement of the imaging device, position of other articulated arms, other end effectors, and / or the ability of the device and / or the imaging device to obtain a suitable image from the end effector within the workspace of the electronic device. The validity of the desired recentering movement for the input control device is determined using a kinematic model of the input control device and the desired recentering movement for the input control device determined during process 240. In some examples, this validity determination may include examining one or more constraints on the following factors: movement of the input control device, position of the operator's workbench portion within the console workspace, and / or ergonomic considerations for the operator of the input control device. When the recentering movement is determined to be valid, the recentering movement is performed using process 260. When any recentering movement is determined to be invalid, use procedure 270 to indicate an error.

[0041] At process 260, the field-of-view recentering movement and the input control recentering movement are coordinated. One or more movement commands are sent to one or more actuators of the articulated arm coupled to the imaging device to command and / or instruct the imaging device to perform the field-of-view recentering movement. One or more movement commands are also sent to one or more actuators of the articulated arm coupled to the input control device to command and / or instruct the input control device to perform the input control device recentering movement. Movement commands for the imaging device and the input control device are typically coordinated. In some examples, coordination may allow the imaging device and the input control device to recenter simultaneously. In some examples, coordination may be performed so that at least some position and / or orientation harmony is maintained between the end effector and the input control device's posture within the imaging device's field of view during the recentering movement. In some examples, process 260 may also include providing audio and / or visual feedback to the operator indicating that the recentering operation is occurring. In some examples, audio feedback may include a distinctive sound, spoken phrase, and / or the like. Upon completion of the recentering movement, operator control is restored using process 280.

[0042] At process 270, an error is indicated. The operator is notified when the determined recentering movement is determined to be invalid. In some examples, the notification may include any appropriate audio and / or visual feedback. In some examples, audio feedback may include the playback of a distinctive sound. After the error is indicated, operator control is restored using process 280.

[0043] At process 280, operator control of the end effector is restored. Whether a recentering movement was performed using process 260 or an error was indicated using process 270, control of the end effector using the input control device is returned to the operator. When an error is indicated, recentering of the imaging device and / or the input control device may become the operator's responsibility. After a period of time when the operator has control of the end effector and / or the imaging device, another recentering operation can be detected using process 210.

[0044] As discussed above and further emphasized here, Figure 2 This is merely an example and should not unduly limit the scope of the claims. Those skilled in the art will recognize many variations, alternatives, and modifications. According to some embodiments, additional conditions may cause early termination of method 200, such as by returning to operator control of process 280 and / or by pausing device operation. In some examples, additional conditions may include manual interference or override from an operator using one or more control devices or articulated arms on an operator workbench, detection of operator disengagement from an operator workbench using one or more safety interlock devices, position tracking errors in the articulated arms and / or input control devices, system malfunctions, and / or the like.

[0045] Figure 3A and Figure 3B This is a simplified schematic diagram of the imaging view before and after a field-of-view recentering operation, according to some embodiments. Figure 3A The diagram illustrates a workspace comprising three articulated arms prior to the execution of a field-of-view recentering operation. The first articulated arm terminates with a gripper-type end effector 310. The gripper-type end effector 310 includes two gripping fingers 312 and 314 and a pivot joint 316. The second articulated arm also terminates with a gripper-type end effector 320, which includes two gripping fingers 322 and 334 and a pivot joint 326. The third articulated arm includes a single finger end effector 330, which contains an end point 332 and a reference point 334. In some examples, the reference point 334 may correspond to a rotatable joint. In some examples, the single finger end effector 330 may represent a cauterizing tool, a suction tool, and / or the like. In some examples, the articulated arm may be a representative example of the articulated arm 120, and the end effector 310, 320 and / or 330 of the gripper type and / or single finger can be a representative example of the end effector 125.

[0046] and Figure 3AThe image depicts a field of view space 340. In some examples, the field of view space 340 may correspond to an image captured by an imaging device. As shown, the field of view space 340 includes a portion of a gripper-type end effector 320 and a gripper-type end effector 310, but does not include a single finger-shaped end effector 330. In some examples, Figure 3A This can correspond to an image captured when the operator is controlling the end effector 310 and / or 320.

[0047] In some examples, problems may arise when the operator expects to switch control of end effectors 310 and 330 instead of end effectors 310 and 320. For example, because end effector 330 is not within the field of view 340, it is not visible in the image of the field of view 340, and the operator may not remember its location. In some examples, the operator can manually recenter the field of view 340 to place end effectors 310 and 330 within it. In some examples, the operator can trigger automated recentering using a method similar to method 200, specifying end effectors 310 and 330 as the end effectors to be recentered with respect to them.

[0048] Figure 3B The field of view 350 of end effectors 310 and 330 after recentering is shown. By using a field of view recentering movement, the imaging apparatus for capturing end effectors 310 and 330 is repositioned and / or reoriented to an orientation containing end effectors 310 and 330. The recentering movement changes the field of view 340 from before the recentering movement to the field of view 350 after the recentering movement occurs. This recentering movement results in the field of view 350 including clamping fingers 312 and 314, pivot joint 316, endpoint 332, and reference point 334. The field of view 350 is also centered with respect to clamping fingers 312 and 314, pivot joint 316, endpoint 332, and reference point 334.

[0049] Figure 4A and Figure 4B These are simplified schematic diagrams of the imaging view and side view after a field-of-view recentering operation, according to some embodiments. Figure 4A and Figure 4B The use of targets on end effectors 310 and 330 is shown so that the field of view 350 is centered on end effectors 310 and 330. This is in Figure 4AThe image captured by the imaging device using the field of view space 350 is shown in the middle. In some examples, when using a field of view coordinate system, the field of view space 350 may include an x-axis from left to right, a y-axis in the upward viewing direction, and a z-axis in the field of view direction.

[0050] To aid in recentering the end effectors 310 and 330 within the field of view 350, one or more targets are selected on each end effector 310 and / or 330. In some embodiments, each target may be associated with the end of each finger of the end effector 310 and / or 330, as well as... Figure 4A Any joints and / or reference points of interest shown are associated. In some embodiments, other criteria may be used to select targets, such as associated targets only at the ends of the finger members and / or at other locations on the end effectors 310 and / or 330 and / or associated articulated arms. Figure 4A As shown, three targets are used on a gripper-type end effector 310, and two targets are used on a single-finger end effector 330. The three targets on the gripper-type end effector 310 include targets 412 and 414 centered on the ends of gripping fingers 312 and 314, respectively, and target point 416 centered on pivot joint 316. The two targets on the single-finger end effector 330 include target point 432 centered on end point 332 and target point 434 centered on reference point 334.

[0051] In some examples, each of targets 412-416 and / or 432-434 can be modeled as a virtual surrounding sphere, the center of which is located at or near the corresponding end of the finger and / or the center of the corresponding joint and / or reference point. In some examples, the radius of each virtual sphere is large enough to capture at least the volume of the corresponding portion of the end effector associated with the corresponding target point. In some examples, the radius can be two to three times the volume of the corresponding portion of the end effector, such that the field of view 350 can capture the corresponding end effector and the space margin around it. This helps prevent the end effector from being placed exactly on the edge of the field of view 350. In some examples, the radius can be set to account for kinematic uncertainties in the position of the target point.

[0052] In some examples, the centroid 440 of the center point of each target 412-416 and / or 432-434 can be calculated. The centroid 440 can then be used as the center point of the field of view 350. The working distance between the centroid 440 and the imaging device can then be adjusted so that the field of view 350 includes each of the targets 412-416 and / or 432-434.

[0053] Figure 4B The corresponding side view of the field of view space 350 is shown. Figure 4B The side view shows that the field of view 350 is a viewing frustum that widens as it moves away from the imaging device 450. In some examples, the angular width of the frustum can be determined from the optical properties of the imaging device 450. In some examples, the imaging device 450 may be an endoscope inserted into the patient via a cannula 460. In some examples, the imaging device 450 may be a stereoscope. In some examples, the cannula 460 may be positioned near a remote center for the imaging device 450 so that the roll, pitch, and yaw rotations of the imaging device 450 are centered on the remote center. Figure 4B As further shown, the imaging device 450 is oriented with its centroid 440 along the field of view in the z-direction of the field of view coordinate system. The centroid 440 may also be located at the average depth in the z-direction of each of the targets 412-416 and / or 332-334. The centroid 440 is also located at a working distance 480 from the end 470 of the imaging device 450.

[0054] In some embodiments, the working distance 480 may be selected based on one or more criteria. The process begins by determining the centroid 440 and using the direction from a reference point on the imaging device to the centroid 440 as the field of view direction or z-axis direction. In some examples, the reference point may correspond to the sleeve 460 when the imaging device is straight between the sleeve 460 and the end cap 470. In some examples, one or more kinematic models of the imaging device may be used to determine the position of the reference point relative to the sleeve 460. In some examples, the reference point may be associated with the end cap 470. The maximum x-axis and / or y-axis range for each of the targets 412-416 and / or 432-434 is then used to determine the corresponding minimum line of sight for each of the targets 412-416 and / or 432-434, so that the targets 412-416 and / or 432-434 are within the frustum body of the field of view space 350. The maximum minimum viewing distance can then be selected as the working distance 480 to ensure that the volume associated with each of targets 412-416 and / or 432-434 is contained within the field of view 350. In some examples, when the working distance is specified and is greater than the maximum minimum viewing distance, the working distance 480 can be increased to a preferred working distance for the imaging device 450. In some examples, the working distance 480 can also be constrained within the minimum and maximum focal lengths of the imaging device 450.

[0055] Once the field of view orientation / field of view coordinate system z-axis and working distance 480 are determined, the field of view recentering movement for imaging device 450 can be determined. Field of view recentering movement may include adjusting the pitch and yaw of imaging device 450 to align with the field of view orientation, and adjusting the insertion and / or retraction of end cap 470 relative to cannula 460 based on working distance 480. In some examples, the field of view recentering movement may be analyzed to determine its effectiveness. In some examples, this may include determining whether the articulated arm to which imaging device 450 is attached can perform the field of view recentering movement. In some examples, the articulated arm may not be able to perform the field of view recentering movement due to connector limitations, maximum movement limits set on the field of view recentering movement, and / or collision avoidance with other articulated arms (e.g., articulated arms 310, 320, and / or 330), the patient anatomy, and / or other objects in the workspace. In some examples, maximum movement limits may include pitch and yaw limits that restrict pitch and yaw motion to less than 30 degrees and / or prevent insertion of the end face 470 beyond its pre-movement position. In some examples, field-of-view recentering movements may be determined to be invalid when any constraints imposed on the movement of the imaging device 450 could result in any target no longer being included in the frustum of the field-of-view space 350.

[0056] In some examples, the field-of-view recentering movement can be planned as a multi-step movement, which includes retracting the imaging device 450 away from the center of mass 440, performing pitch and / or yaw orientation to align the field of view, and then inserting the end cap 470 to a working distance 480 from the center of mass 440. In some examples, when the field-of-view recentering movement includes magnification, the multi-step movement may include performing pitch and / or yaw orientation to align the field of view before inserting the end cap 470 to a working distance 480 from the center of mass 440. In some examples, when the field-of-view recentering movement includes reduction, the multi-step movement may include retracting the imaging device to a working distance 480 before performing pitch and / or yaw orientation. In some examples, the multi-step movement can help reduce the likelihood of the end cap 470 colliding with the end effectors of the articulated arms 310, 320, and / or 330, the patient anatomy, and / or other objects in the workspace. In some examples, field-of-view recentering movement may also include rolling the imaging device 450 to align the upward-looking direction / field-of-view coordinate system y-axis with the world coordinate system. In some examples, the field-of-view recentering movement can be determined by using iterative motion planning operations, where the iterative motion planning operations optimize the pitch, yaw, and insertion of the imaging device 450 based on accuracy constraints in the articulated arm of the joint-controlled imaging device 450, thereby minimizing orientation and / or positioning errors of the imaging device 450.

[0057] In some embodiments, when the field-of-view recentering movement is determined to be invalid, an alternative field-of-view recentering movement is determined, wherein the end 470 is retracted to a minimum insertion depth. In some examples, the minimum insertion depth may correspond to a depth beyond which the imaging device may become partially obstructed by one or more portions of the articulated arm used for positioning and / or orienting the imaging device 450. In some examples, the portion of the articulated arm that may partially obstruct the imaging device may correspond to the sleeve 460. In some examples, the minimum insertion depth may correspond to a point, i.e., a predetermined distance from the remote center of the imaging device. In some examples, the predetermined distance may be based on the length of the sleeve 460. In some examples, the predetermined distance may range from 2 cm to 9 cm in length. As the end 470 retracts to the sleeve 460, the field-of-view orientation of the imaging device 450 is then set to point toward the centroid 440. The maximum x-axis and / or y-axis ranges for each of the targets 412-416 and / or 432-434 are then examined to see if they fall within the field-of-view space 350. Alternative visual field recentering movements are also determined to be invalid when each of targets 412-416 and / or 432-434 does not fall within the visual field space 350. Similar to visual field recentering movements, additional checks on the validity of alternative visual field recentering movements may include determining whether the articulated arms to which the imaging device 450 is attached can perform alternative visual field recentering movements. In some examples, the articulated arms may not be able to perform alternative visual field recentering movements due to connector limitations, maximum movement limits set on visual field recentering movements, and / or collision avoidance with other articulated arms (e.g., articulated arms 310, 320, and / or 330) and / or patient anatomy. When alternative visual field recentering movements are invalid, visual field recentering is aborted and an appropriate error is indicated.

[0058] Figure 5 This is a simplified schematic diagram of a method 500 for recentering a field of view according to some embodiments. One or more of the processes 510-580 of method 500 may be implemented at least in part in the form of executable code stored on a non-transitory, tangible, machine-readable medium, which, when run by one or more processors (e.g., processor 140 in control unit 130), causes one or more processors to execute one or more of processes 510-580. In some embodiments, method 500 may be executed by an application (such as motion control application 160). In some embodiments, method 500 may be used to recenter one or more of end effectors 125 and / or end effectors 310-330 in the field of view space of an imaging device (such as imaging device 450) so that the corresponding image can be displayed on display system 180.

[0059] At process 510, the view center point is determined. In some examples, the view center point may correspond to the centroid of one or more end effectors to be recentered in an image captured by an imaging device (such as imaging device 450). Figure 3A , Figure 3B , Figure 4A and Figure 4B In some examples, the end effector may correspond to end effectors 310 and 330, and the center of view may correspond to the center of mass 440. In some examples, the center of mass can be determined by obtaining the centers of mass of one or more targets such as targets 412-416 and / or 432-434. In some examples, sensors associated with the articulated arms of end effectors 310 and / or 330 can be used to determine the position of joints in the articulated arms. These joint positions, combined with one or more kinematic models of end effectors 310 and / or 330 and their articulated arms, can be used to determine the position of end effectors 310 and / or 330, which can then be used to determine the center of mass.

[0060] At process 520, the working distance is determined. In some examples, the working distance can be determined by determining how far the end effector's targets should be so that each target is within the field of view of the imaging device. In some examples, the working distance can be determined by determining the maximum x-axis and / or y-axis range perpendicular to the field of view for each target and then determining the corresponding minimum line of sight for each target so that the target is within the frustum of the field of view. The maximum minimum line of sight can then be selected as the working distance to ensure that each target is included in the field of view. In some examples, when the working distance is specified and is greater than the maximum minimum line of sight, the working distance can be increased to a preferred working distance for the imaging device. In some examples, the preferred working distance can be set by the operator of the imaging device. In some examples, the working distance can also be constrained within the minimum and maximum focal lengths for the imaging device.

[0061] At process 530, the desired imaging device position and orientation are determined. The imaging device is oriented using a vector between a reference point on the imaging device and the center of view determined during process 510. In some examples, the reference point may correspond to the remote center when the imaging device is in a straight line between the remote center and the end of the imaging device, as the imaging device is subject to movement constraints about a remote center (such as the sleeve 460 of the imaging device 450). In some examples, one or more kinematic models of the imaging device may be used to determine the position of the reference point. In some examples, the reference point may be associated with the end of the imaging device. In some examples, the orientation vector may be determined by aligning the end of the imaging device with the center of view while preserving the roll position of the imaging device, and then using the field of view direction of the imaging device as the orientation vector. The position of the end of the imaging device is then determined based on positioning the end of the imaging device away from the center of view in a direction opposite to the field of view direction at a working distance as determined during process 520.

[0062] At process 540, it is determined whether the desired imaging device position and orientation are valid. In some examples, this may include determining whether the articulated arm to which the imaging device is attached can perform a field-of-view recentering movement from its current position and orientation to the imaging device position and orientation determined during process 530. In some examples, the articulated arm may not be able to perform the field-of-view recentering movement due to joint limitations, maximum movement limits set on the field-of-view recentering movement, and / or collision avoidance with other articulated arms, patient anatomy, and / or other objects in the workspace. In some examples, maximum movement limits may include pitch and yaw limits that restrict pitch and yaw motion to 30 degrees or less and / or prevent the imaging device from being inserted beyond its pre-movement position. In some examples, the field-of-view recentering movement may be determined to be invalid when any constraints set on the movement of the imaging device could result in any target no longer being included in the frustum of the field of view space. When the desired imaging device position and orientation are valid, the imaging device is moved to the desired imaging device position and orientation using process 550. When the desired imaging device position and orientation are invalid, an alternative imaging device position and orientation are determined using process 560.

[0063] At process 550, the imaging device is moved. The imaging device is moved by planning appropriate movements for the imaging device and the articulated arm attached to it, and then the planned movements are executed by sending one or more commands to actuators in the articulated arm. In some examples, the movement plan may include multi-step movements, including retracting the imaging device away from the center of vision, performing pitch and / or yaw orientation to align the field of view so that the imaging device is oriented toward the center of vision, and then inserting the imaging device to a working distance from the center of vision. In some examples, when the imaging device movement includes magnification, the multi-step movement may include performing pitch and / or yaw orientation to align the field of view before inserting the imaging device to the working distance. In some examples, when the imaging device movement includes reduction, the multi-step movement may include retracting the imaging device to the working distance before performing pitch and / or yaw orientation. In some examples, multi-step movements may help reduce the likelihood of end-effector collisions between the imaging device and other articulated arms, patient anatomy, and / or other objects in the workspace. In some examples, the insertion step may be omitted when the imaging device is about to be retracted, as determined during process 560. In some examples, the planned motion may also include rolling the imaging device to align its upward viewing direction with the world coordinate system. In some examples, the kinematic model of one or more articulated arms associated with the imaging device may be used to assist motion planning. In some examples, the planned motion may be determined by using iterative motion planning operations, which optimize the pitch, yaw, and insertion and / or retraction of the imaging device based on accuracy constraints in the joint control of the articulated arms associated with the imaging device, thereby minimizing orientation and / or positioning errors of the imaging device. Once the imaging device has moved, the recentering operation is completed.

[0064] At process 560, an alternative imaging device position and orientation is determined. When the desired imaging device position and orientation determined during process 540 is invalid, an alternative imaging device position and orientation is determined, wherein the imaging device is retracted away from the center of vision. In some examples, the alternative imaging device position and orientation includes retracting the imaging device to a minimum usable insertion depth and ignoring the working distance determined during process 520. In some examples, the minimum insertion depth may correspond to a depth beyond which the imaging device may become partially obstructed by one or more portions of the articulated arm used for positioning and / or orienting the imaging device. In some examples, the portions of the articulated arm that may partially obstruct the imaging device may correspond to a sleeve, such as sleeve 460. In some examples, the minimum insertion depth may correspond to a point at a predetermined distance from the remote center of the imaging device. In some examples, the predetermined distance may be based on the length of the sleeve. In some examples, the predetermined distance may range in length from two centimeters to nine centimeters. The alternative imaging device orientation then includes orienting the imaging device toward the center of vision using a method similar to that used during process 530.

[0065] At process 570, it is determined whether the alternative imaging device position and orientation are valid. In some examples, this may include determining whether the articulated arm to which the imaging device is attached can perform a field-of-view recentering movement from its current position and orientation to the alternative imaging device position and orientation determined during process 560. In some examples, the articulated arm may not be able to perform the field-of-view recentering movement due to joint limitations, maximum movement limits set on the field-of-view recentering movement, and / or collision avoidance with other articulated arms, patient anatomy, and / or other objects in the workspace. In some examples, the maximum movement limits may include pitch and yaw limits that restrict pitch and yaw motion to 30 degrees or less. When the alternative imaging device position and orientation are valid, the imaging device is moved to the alternative imaging device position and orientation using process 550. When the alternative imaging device position and orientation are invalid, an error is indicated using process 580.

[0066] At process 580, an error is indicated. The operator is notified when the determined and alternative imaging device positions and orientations are determined to be invalid. In some examples, the notification may include any suitable audio and / or visual feedback. In some examples, audio feedback may include the playback of a distinctive sound.

[0067] As discussed above and further emphasized here, Figure 5This is merely an example and should not unduly limit the scope of the claims. Those skilled in the art will recognize many variations, alternatives, and modifications. According to some embodiments, additional conditions and / or safety factors may be considered during method 500 and more specifically during process 550 when the imaging apparatus is in automated movement.

[0068] In some embodiments, one or more preventative measures may be used to reduce and / or prevent contact and / or interference between the imaging device and the patient's anatomy and / or other obstacles approaching the imaging device. In some examples, one or more preoperative and / or intraoperative images of the patient's anatomy may be used to identify one or more non-fly zones where the imaging device should not enter. In some examples, forces and / or torques on one or more joints used to manipulate the imaging device may be monitored using appropriate sensors to determine if unexpected forces and / or torques indicate unacceptable contact between the imaging device and the patient's anatomy and / or other obstacles. In some examples, errors between the commanded position and / or velocity of the imaging device and / or joints used to manipulate the imaging device and / or joints and the actual position and / or velocity may be monitored to determine if the error exceeds a configurable threshold. In some examples, the configurable threshold may be different for each joint. In some examples, errors may be low-pass filtered and / or smoothed to avoid false positive detections that may be due to additional acceptable temporary conditions. In some examples, one or more contacts located near the distal end of the imaging device may be monitored to determine if the imaging device is in contact with the patient's anatomy and / or other obstacles. In some examples, determining that the imaging device is contacting and / or interfering with the patient's anatomy can lead to premature termination of the imaging device's movement and / or activation of one or more visual and / or audio warnings.

[0069] In some embodiments, one or more interlocking devices may be used to ensure the operator's presence to observe repositioning center movement. In some examples, one or more input control devices, such as head-in sensors, may be used to determine that the operator is present at the operator console and is observing the image from the imaging device in the appropriate position. In some examples, an illuminance sensor may be used to determine that the image from the imaging device is being displayed for the operator on the observer's console. In some examples, the detection of the operator's absence and / or the loss of the image on the imaging device by one or more interlocking devices may lead to the early termination of the imaging device's movement and / or the activation of one or more visual and / or audio warnings.

[0070] In some embodiments, the movements planned and executed during process 550 may be designed to set upper limits on the speed and / or acceleration of the imaging device and / or one or more joints used to manipulate the imaging device. In some examples, the speed and / or acceleration may be limited so that the operator monitoring the recentering movement has sufficient time to react to and override and / or terminate the recentering movement in response to potential undesirable movements in the imaging device. In some examples, the speed and / or acceleration may be limited so that the positive delivery torque in the joints used to manipulate the imaging device is kept at a sufficiently minimum level that allows for overcoming anticipated inertia, viscous friction, and / or movements in the imaging device that could result in excessive contact with the patient's anatomy, other end effectors near the imaging device, and / or other unintended obstacles. In some examples, the feedback torque in the joints used to manipulate the imaging device may be limited to a minimum sufficient to overcome anticipated sources of resistance, such as sterilization curtains, friction in cannula seals, and / or the like.

[0071] Figure 6 This is a simplified schematic diagram of a method 600 for determining a preferred working distance for an imaging apparatus according to some embodiments. One or more of the processes 610-660 of method 600 may be implemented at least in part in the form of executable code stored on a non-transitory, tangible, machine-readable medium, which, when run by one or more processors (e.g., processor 140 in control unit 130), causes one or more processors to execute one or more of the processes 610-660. In some embodiments, method 600 may be executed by an application such as motion control application 160. In some embodiments, method 600 may be used to determine a preferred working distance between the imaging apparatus and a center of vision. In some examples, the preferred working distance may be a preferred working distance used during process 520. In some embodiments, method 600 may be used by an operator monitoring manual repositioning operations of the imaging apparatus to learn a preferred working distance for the operator.

[0072] At process 610, the initiation of movement for the imaging device is detected. Repositioning movements of the imaging device can be monitored as the operator operates the device, which has one or more articulated arms and an imaging device. In some examples, the movement of the imaging device can be associated with an end of the imaging device (such as end 470). In some examples, the movement of interest can be associated with manual repositioning of the imaging device performed by the operator. By monitoring manual repositioning of the imaging device, it is feasible to learn the preferred distance between the imaging device and one or more end effectors captured by the imaging device in the acquired image. In some examples, each manual repositioning operation can be detected by activation of repositioning and / or reorientation control for the imaging device. In some examples, when the initiation of manual repositioning is detected, the current position and / or orientation of the imaging device can be recorded.

[0073] At process 620, the end of motion of the imaging device is detected. Once motion of the imaging device is detected during process 610, the motion is monitored until its end. In some examples, the end of motion can be detected by the lack of movement in the imaging device. In some examples, the lack of motion can be detected by determining that the speed of the imaging device has decreased below a minimum threshold. In some examples, the lack of motion can be detected by determining that the speed of the imaging device remains below the minimum threshold for a predetermined period of time. In some examples, the end of motion can be associated with the end of manual repositioning, as indicated by the deactivation of repositioning and / or reorientation controls. In some examples, when the end of motion is detected, the current position and / or orientation of the imaging device can be recorded.

[0074] At process 630, it is determined whether sufficient motion has been detected in the imaging device. The amount of motion of the imaging device can be determined using the current position and / or orientation values ​​recorded during processes 610 and 620. In some examples, the amount of motion can be the distance between the start and end positions, such as the Euclidean distance. In some examples, the amount of motion can be further based on the angular change between the start and end orientations. In some examples, the angular change can be converted into a distance by determining the sine and / or cosine of the angular change and multiplying one of them by a distance related to the working distance of the imaging device prior to the start of motion detection during process 610. When the amount of motion exceeds a minimum threshold, such as approximately 0.5 cm, a new preferred working distance is determined at process 640. When the amount of motion does not exceed the minimum threshold, method 600 can return to process 610 to detect future motion in the imaging device.

[0075] At process 640, the z-distance is determined as the point of interest. In some examples, the working distance of the imaging device can be represented based on the vertical distance from the imaging device to one or more points of interest along the field of view direction. In some examples, when the points of interest are mapped to the field of view coordinate system of the imaging device, the z-value of each point of interest can represent the corresponding z-distance. In some examples, the points of interest can correspond to the center of one or more targets on one or more end effectors. In some examples, the end effectors can be selected by the operator and / or automatically selected based on the end effectors determined to be visible in the images captured by the imaging device. Figure 4A and Figure 4B In the example, the target can be selected from targets 412-416, 422-426 and / or 432-434.

[0076] At process 650, the current working distance is determined. In some examples, the current working distance can be determined by aggregating each z-distance determined during process 640. In some examples, the aggregator may include the average, median, minimum, maximum, and / or similar values. In some examples, the z-coordinate of the centroid of the point of interest (such as centroid 440) can be used to determine the current working distance.

[0077] At process 660, the current working distance is compared with the set of previous working distances. During process 650, the set of current working distance and previous working distance values ​​is determined to determine the preferred working distance. In some examples, the current working distance determined during process 650 may be weighted based on the amount of movement between the start and end of the imaging device's movement, so that larger movements have a greater impact on the preferred working distance. In some examples, the set may include determining a moving average, a windowed average over a predetermined time period, exponential smoothing, and / or the like. In some examples, the preferred working distance may be initialized to a default value. In some examples, the default value may be based on the minimum and / or maximum focal length used for the imaging device. In some examples, the default value may be set to approximately 7 cm. In some embodiments, multiple preferred working distances may be determined based on the background of detected movement. In some examples, the background may include maintaining different preferred working distances for different operators, different programs, different stages of a program, digital zoom settings, focal length settings, stereo parallax settings, and / or the like. Once the set is executed, method 600 may repeatedly include additional movement within the imaging device in the set of preferred working distances.

[0078] Figure 7This is a simplified schematic diagram illustrating the relationship between an end effector in an image on a display system and a corresponding input control device in the console workspace after an input control device recentering operation, according to some embodiments. In some examples, the input control device recentering operation may correspond to an input control device recentering that occurs as part of recentering during method 200. In some examples, one of the objectives of the recentering operation is to maintain positional and / or orientational harmony between the end effector in the field of view of the imaging device and the corresponding input control device during field of view recentering. In some examples, input control device recentering includes changing the position and / or orientation of each input control device to correspond to the position and / or orientation of the respective end effector.

[0079] Figure 7 The upper part is shown in Figure 3B and Figure 4A After the field of view is recentered, the images of end effectors 310 and 330 can be captured when the images are displayed on display system 180. The images captured using imaging device 450 can be displayed on display system 180 as images shown within the boundaries 710 of display system 180. For clarity, in Figure 7 Additional portions of the end effectors 310 and 330 and their articulated arms are shown, even though they are not visible on the display system 180, and any objects that might be partially or completely obstructing the end effectors are removed from the image. A viewpoint 720, which may correspond to the centroid 440, is also shown. In some examples, each point of interest on the end effectors 310 and 330 may also be mapped to a point such as x, to facilitate recentering of the input control device. v axis, y v axis and z v The axes depict the field-of-view coordinate system. In some examples, the points of interest may correspond to targets 412-416 and / or 432-434.

[0080] Figure 7The lower portion shows a console workspace containing input control devices 760 and 770, respectively corresponding to end effectors 310 and 330. Input control devices 760 and 770 can be coupled to the main body 730 of the operator's workbench via their own articulated arms. In some examples, the console workspace can be positioned relative to an armrest 740. In some examples, the operator's workbench can correspond to an operator's workbench 170, and the armrest 740 can correspond to an armrest 190. Because each operator may prefer different heights of the armrest 740, different sizes and lengths of arms, wrists, and / or hands, and / or different preferences for elbow placement and / or flexion, an ergonomic center 750 can be defined within the console workspace. In some examples, the console workspace coordinate system can be as follows: c axis, y c axis and z c The axis is defined as shown.

[0081] In some embodiments, the positional and / or orientational harmony between the end effectors 310 and 330 and the input control devices 760 and 770 can be determined based on a mapping between control points on the input control devices 760 and 770 and corresponding points on the end effectors 310 and 330. More specifically, as Figure 7 As shown in the example, control points 762 and 764 on the finger loops of the input control device 760 can be mapped to targets 412 and 414, respectively, so that the gripping fingers 312 and 314 open and close as the operator opens and closes the distance between control points 762 and 764 during remote operation. Furthermore, control point 766 on the input control device 760 can be mapped to target point 416 so that pivot joint 316 can move accordingly as pivot point 766 moves during remote operation. Similarly, control points 772 and 774 on the input control device 770 can be mapped to targets 432 and 434, respectively.

[0082] To maintain positional and / or orientational harmony between the end effectors 310 and 330 and the input controls 760 and 770, respectively, the input control recentering operation repositions and / or reorients the input controls 760 and 770 around the ergonomic center 750 to approximately correspond to the position and / or orientation of the end effectors 310 and 330 within the field of view space corresponding to the image with boundary 710. Therefore, as Figure 7As shown, the input control device 760 is positioned in the lower left portion of the console workspace and oriented in an upper right direction to match the position and orientation of the end effector 310. Similarly, the input control device 770 is positioned in the upper right portion of the console workspace and oriented in a lower left direction to match the position and orientation of the end effector 330. To maintain positional and / or orientational harmony, the view and console stereoscopic viewer workspace coordinate systems are typically aligned left-right (x-y). c and x v Direction, up and down (y) c and y v ) direction and inside and outside (z c and z v Alignment in the direction of the input control device. Typically, since operator hand movements of the input control device can be translated into corresponding movements of the end effectors 310 and / or 330, this provides intuitive operation of the end effectors 310 and / or 330 during remote operation.

[0083] In some embodiments, the positional and / or orientational harmony between the end effectors 310 and 330 and the input controls 760 and 770 can be maintained by mapping targets 412-416 and / or 432-434 of the end effectors 310 and 330 from the field-of-view coordinate system to the console workspace coordinate system, and then using one or more actuators of the articulated arms associated with the input controls 760 and 770 to position and / or orient corresponding control points 762-766 and / or 772-774 at the mapped locations in the console workspace coordinate system. In some examples, this can be accomplished by using transformations and scaling. In some examples, one or more transformations can be used to map the view center point 720 to the ergonomic center 740. Once the view center point 720 and the ergonomic center 740 are aligned, the distance in the field-of-view coordinate system can be converted to the corresponding distance in the console workspace coordinate system. In some examples, one or more scale factors used for the conversion can be set by the operator at the operator's workstation. In some examples, one or more scale factors can be set based on the relative dimensions of the image boundary 710 and the console workspace. Once each point 312-316 and / or 332-334 of the end effector is mapped to determine the position of the corresponding control points 762-766 and / or 772-774, motion plans for input controls 760 and 770 can be developed and executed.

[0084] In some embodiments, the position of each control point 762-766 and / or 772-774 may be constrained before a motion plan is developed and executed. In some examples, the position and / or orientation of control points 762-766 and / or 772-774 may be constrained by the range of motion limitations of the joints in the corresponding articulated arms, thereby: maintaining a minimum and / or maximum distance between input controls 760 and 770; avoiding collisions with the arm support 740 and / or other parts of the operator's workbench; preventing left / right crisscrossing of input controls 760 and 770; avoiding undesired positions and / or orientations of input controls 760 and 770; taking into account the positional accuracy of targets 412-416 and / or 432-434 and / or control points 762-766 and / or 772-774 (e.g., about 1 cm) and / or similar.

[0085] Although Figure 7 Not shown, but the forward-to-back positioning of the input control devices 760 and / or 770 is matched to the depth of the corresponding end effectors 310 and / or 330. Therefore, the z-axis of targets 412-416 and / or 432-434... v The coordinates were shifted and transformed accordingly to determine the z-axis of control points 762-766 and / or 722-774. c Coordinates. Therefore, with Figure 4B The side view relationships shown are consistent, and control points 672 and 674 can be closer to the operator's position than control points 762-766.

[0086] Figure 8 This is a simplified schematic diagram of a method 800 for recentering an input control device according to some embodiments. One or more of the processes 810-860 of method 800 may be implemented at least in part in the form of executable code stored on a non-transitory, tangible, machine-readable medium, which, when run by one or more processors (e.g., processor 140 in control unit 130), causes one or more processors to execute one or more of the processes 810-860. In some embodiments, method 800 may be executed by an application such as motion control application 160. In some embodiments, method 800 may be used to recenter one or more input control devices 195, 760, and / or 770 in a console workspace to maintain harmony with the position and / or orientation of the corresponding end effectors 125, 310, 320, and / or 330, as displayed in an image captured by an imaging device (such as imaging device 450) and displayed on a display system 180.

[0087] At process 810, the end effector position is determined. In some examples, sensors associated with the articulated arm linked to the end effector can be used to determine the position of a joint within the articulated arm. These joint positions, combined with one or more kinematic models of the articulated arm and the end effector, can be used to determine the end effector position. In some examples, one or more images of the end effector can be used to determine the end effector position. Figure 3A , Figure 3B , Figure 4A , Figure 4B and Figure 7 In the example, the end effector may correspond to end effectors 310 and 330, wherein the positions of end effectors 310 and / or 330 are characterized by targets 412-416 and / or 432-434.

[0088] At process 820, the end effector position is mapped to the field-of-view coordinate system. The field-of-view coordinate system for the imaging device is determined using sensors associated with the articulated arm associated with the imaging device, and one or more kinematic models of the articulated arm associated with the imaging device. The end effector position determined during process 810 is then mapped to the field-of-view coordinate system. This mapping helps determine the x and y positions of the end effector in the image captured by the imaging device, and the z position of the end effector, indicating how far the end effector is from the imaging device in the field-of-view direction. Figure 7 In the example, the end effector position in the field-of-view coordinate system can correspond to the x-axis of target 412-416 and / or 432-434. v y v and z v Coordinate values.

[0089] At process 830, the end effector position shifts about the ergonomic center. To help maintain positional and / or orientational harmony between the end effector and one or more input controls of the operator console, the field-of-view coordinate system is mapped to the console workspace coordinate system. In some examples, the mapping between the field-of-view and console workspace coordinate systems begins by associating the center point in the field-of-view coordinate system with the center point in the console workspace coordinate system. In some examples, the centroid of the end effector position can be selected as the center point in the field-of-view coordinate system. In some examples, the ergonomic center of the console workspace can be selected as the center point in the console workspace coordinate system. In some examples, when the origins of the field-of-view coordinate system and / or the console workspace coordinate system do not coincide with the selected center point, the two center points can be associated using one or more transformations. In some examples, the ergonomic center of the console workspace can be pre-selected by the operator of the operator console and / or by the geometry of the operator console and its input controls. In some examples, the ergonomic center can move when one or more supports (such as arm supports on the console workbench) are repositioned. In some examples, it can be achieved by monitoring, such as about... Figure 9 Further detailed discussion of operator workstation operation will be provided in the ergonomics center. Figure 7 In the example, process 830 corresponds to aligning the centroid 720 with the ergonomic center 750.

[0090] At process 840, the end effector position is converted about the ergonomic center to determine the control point position. Because the scales of the field-of-view coordinate system and the console workspace coordinate system are typically different, the position of the end effector in the field-of-view coordinate system relative to its center point is converted about the ergonomic center in the console workspace coordinate system. This conversion transforms the relative distance between the end effector position and the center point in the field-of-view coordinate system into the corresponding relative distance between the input control device position and the ergonomic center in the console workspace coordinate system. Each converted point from the field-of-view coordinate system then becomes a control point in the console workspace coordinate system. In some examples, one or more scale factors used for the conversion can be set by the operator at the operator's workstation. In some examples, one or more scale factors can be set based on the corresponding size of the captured image in the field-of-view coordinate system and the size of the console workspace. Figure 7 In the example, the conversion in process 840 will change the corresponding x v y v and z v Convert the distances to x respectively c y c and z cThe distances are calculated so that the positions of targets 412-416 and / or 432-434 are converted into the positions of control points 762-766 and / or 772-774, respectively.

[0091] At process 850, the control point positions are constrained. In some examples, the mapping of points associated with the end effector positions in the field-of-view coordinate system to the control point positions in the console workspace coordinate system may not result in suitable positions and / or orientations for input controls (such as input controls 195, 760, and / or 770). In some embodiments, the position of each control point mapped during processes 830 and / or 940 may be constrained. In some examples, the position and / or orientation of the control points may be constrained by the range of movement limits of the joints in the corresponding articulated arms, thereby: maintaining minimum and / or maximum distances between control points of different input controls; avoiding collisions with the arm supports and / or other parts of the operator's workbench; preventing left / right cross-location of input controls; avoiding undesirable positions and / or orientations of input controls; taking into account limitations on the positional accuracy of the end effector points and / or control points of the input controls (e.g., approximately 1 cm) and / or similar limitations.

[0092] At process 860, the input control device is moved to the control point position. Using one or more kinematic models of the articulated arms associated with the input control device, a motion plan is determined based on the input control device to move the control point on the input control device from its previous position to the control point position determined using processes 830-850. In some examples, when the desired motion of the input control device and the control point position may result in a collision and / or near-collision between the articulated arms associated with the input control device, the motion plan may include a multi-segment plan with intermediate control position points to avoid the collision and / or near-collision. The motion plan can then be implemented by sending one or more commands to the actuators associated with the articulated arms. In some examples, an error is indicated when no suitable motion plan can be determined.

[0093] Figure 9This is a simplified schematic diagram of a method 900 for determining the ergonomic center for an input control device according to some embodiments. One or more of the processes 910-950 of method 900 may be implemented at least in part as executable code stored on a non-transitory, tangible, machine-readable medium, which, when run by one or more processors (e.g., processor 140 in control unit 130), causes one or more processors to execute one or more of the processes 910-950. In some embodiments, method 900 may be performed by an application such as motion control application 160. In some embodiments, method 900 may be used to determine the ergonomic center of one or more input control devices in a console workspace. In some embodiments, method 900 may be used to monitor manual repositioning operations of the input control devices to learn a preferred ergonomic center for the operator.

[0094] At process 910, the start of repositioning movement of the input control device is detected. During operation using the remote operating device at the operator's workstation, the operator can periodically reposition one or more input control devices to a more comfortable and / or ergonomic position. In some examples, this can be triggered by the operator engaging a clutch, where the clutch disengages from the end effector being remotely operated by the corresponding input control device and the movement of the input control device. In some examples, detecting clutch engagement indicates the start of input control device repositioning movement. In some examples, when the start of input control device repositioning movement is detected, the current position and / or orientation of the input control device can be recorded for one or more control points of the input control device.

[0095] At process 920, each input control device repositioning movement is detected. When the operator completes the input control device repositioning movement, the clutch disengages, and remote operation of the articulated arm and end effector is resumed. In some examples, detecting clutch disengagement indicates the end of the input control device repositioning movement. In some examples, when the end of the input control device repositioning movement is detected, the current position and / or orientation of the input control device may be recorded based on one or more control points of the input control device.

[0096] At process 930, it is determined whether sufficient motion was detected in the input control device between the start and end of a repositioning movement. The amount of motion of the input control device can be determined using the current position and / or orientation values ​​recorded during processes 910 and 920. In some examples, the amount of motion can be the distance between the start and end positions, such as Euclidean distance. In some examples, the amount of motion can be a set of one or more distances between the start and end positions of one or more control points. In some examples, the set can be a sum, a weighted sum, an average, and / or the like. When the amount of motion exceeds a minimum threshold (such as approximately 2 cm), the center of the input control device is determined at process 940. When the amount of motion does not exceed the minimum threshold, method 900 returns to process 910 to detect future repositioning movements of the input control device.

[0097] At process 940, the center of the input control device is determined. The center of the input control device is determined using the end position of the input control device recorded during process 920. In some examples, the center of the input control device can be determined using a set of end positions of one or more control points of the input control device (such as the centroid).

[0098] At process 950, the input control device center is compared with the previous set of input control device centers. The input control device center determined during process 940, compared with the previous set of input control device centers, is used to determine the ergonomic center. In some examples, the input control device center determined during process 940 may be weighted based on the amount of motion between the start and end of an input control device repositioning movement, so that larger movements have a greater impact on the ergonomic center. In some examples, the set may include determining a moving average, a windowed average over a predetermined time period, exponential smoothing, and / or the like. In some examples, the ergonomic center may be initialized to a default value. In some examples, the default value may be based on the geometry of the input control device, the console workspace, and / or the operator's expected physiological function. In some embodiments, multiple ergonomic centers may be determined based on the context of detected motion. In some examples, the context may include different ergonomic centers maintained for different operators, different procedures, different stages of a procedure, different end effectors being remotely operated by the input control device, and / or the like. Once the set is executed, method 900 may repeatedly include additional input control device repositioning movements within the set of ergonomic centers. In some examples, the ergonomic center can be adjusted to take into account the position of one or more supports (such as arm supports) in the console workspace.

[0099] Figure 10This is a simplified schematic diagram of a method 1000 for controlling an imaging apparatus according to some embodiments. One or more of the processes 1005-1050 of method 1000 may be implemented at least partially in the form of executable code stored on a non-transitory, tangible, machine-readable medium, which, when run by one or more processors (e.g., processor 140 in control unit 130), causes one or more processors to execute one or more of the processes 1005-1050. In some embodiments, method 1000 may be executed by an application such as motion control application 160. In some embodiments, method 1000 may be used to combine manual control of an imaging apparatus (such as imaging apparatus 450) with automated recentering of the imaging apparatus using one or more input control devices in a console workspace. In some embodiments, variations of the processes are possible. In some examples, processes 1020-1035 may be executed in different orders and / or substantially in parallel.

[0100] At process 1005, activation of the imaging device motion mode is detected. In some examples, the operator of the electronic device may manually trigger the activation of the imaging device motion mode using one or more input control devices (such as switches, buttons, pedals, levels, voice recognition, and / or the like). In some examples, a request may be issued as a temporary input to trigger the imaging device motion mode and / or as a continuous input to activate the imaging device motion mode.

[0101] At process 1010, the imaging device enters motion mode. In some examples, operator control of one or more end effectors is suspended before entering motion mode. In some examples, one or more motion input control devices (such as one or more master control devices 195) can be detached from control of one or more end effectors. In some examples, detachment may occur due to a limited number of operator controls for controlling devices attached to the distal end of the articulated arm and / or due to limitations on the capabilities of one or more end effectors for operator control and / or remote operation electronics. Suspension of operator-mandated control allows the imaging device to move without motion interference from one or more end effectors commanded by the operator.

[0102] At process 1015, it is determined whether one or more motion input controls are being used. In some examples, a timeout period may begin when the imaging device enters motion mode during process 1010. During the timeout period, one or more motion input controls may be monitored to determine if the operator is attempting to manually control the position and / or orientation of the imaging device using one or more motion input controls. In some examples, the timeout period may have a configurable length, such as approximately 0.5 seconds. In some examples, the use of one or more motion input controls may be determined based on whether the operator moves one or more motion input controls by more than a threshold distance, rotates one or more motion input controls by more than a threshold angle, and / or some combination of both. In some examples, the threshold distance may be 5-10 mm. In some examples, the threshold angle may be 5 degrees or greater. When the timeout period ends without one or more motion input controls being used, recentering begins from process 1020. When the use of one or more input controls is detected during the timeout period, manual control of the imaging device begins from process 1040.

[0103] At process 1020, recentering of the imaging device is performed. In some examples, processes 510-580 of method 500 can be used to perform recentering of the imaging device during process 1020. In some examples, while the imaging device is recentering during process 1020, one or more motion input controls can be automatically moved to maintain positional and / or orientational harmony between the one or more motion input controls and the imaging device. In some examples, processes 810-860 of method 800 can be modified to maintain positional and / or orientational harmony between the one or more motion input controls and the imaging device, wherein the position and / or orientation of the imaging device is being replaced by the position and / or orientation of the end effector.

[0104] At process 1025, it is determined whether one or more motion input controls are being used. In some examples, the use of one or more motion input controls can correspond to deliberate motion of one or more motion input controls by the operator and / or sufficient resistance by the operator to changes in the position and / or orientation of one or more motion input controls, while positional and / or orientation harmony between one or more motion input controls and the imaging device is being maintained. In some examples, deliberate motion can be detected using a method similar to that used during process 1015. In some examples, resistance by the operator can be detected by determining the difference between the commanded position and / or orientation of the motion input controls exceeding a threshold distance and / or a threshold angle and the actual position and / or orientation. In some examples, the threshold distance can be approximately 1 cm to 3 cm. In some examples, the threshold angle can be 5 degrees or higher. When it is detected that one or more motion input controls are not being used, recentering continues to process 1030. When it is detected that one or more input controls are being used, manual control of the imaging device begins to process 1040.

[0105] At process 1030, it is determined whether recentering is complete. The recentering performed by process 1020 is monitored to determine whether the motion planned as part of the recentering is completed using the imaging device with the desired posture. When recentering is complete, manual control of the imaging device begins at process 1040. If recentering is not complete, recentering begins at process 1035.

[0106] At process 1035, it is determined whether deactivation of the imaging device motion mode has been detected. In some examples, the operator may indicate deactivation of the imaging device motion mode using one or more input controls, such as switches, buttons, pedals, levels, voice recognition, and / or the like. In some examples, a compensatory momentary input may be used to deactivate the imaging device motion mode when a request to activate the imaging device motion mode is activated during process 1005 using a momentary input. In some examples, the removal of a continuous input (such as removing a foot from a pedal) may be used to deactivate the imaging device motion mode when a request to activate the imaging device motion mode is activated during process 1005 using a continuous input. In some examples, one or more precautions, safety features, and / or interlocks associated with method 500 and / or process 550 may be used to determine that deactivation of the imaging device motion mode should occur. When deactivation of the imaging device motion mode is not detected, recentering continues by repeating processes 1020-1035. When the deactivation of the imaging device motion mode is detected, process 1050 exits the imaging device motion mode.

[0107] At process 1040, the imaging device moves based on a motion input control device. In some examples, the motion input control device can be used to manually control the position and / or orientation of the imaging device. In some examples, the imaging device can be moved to maintain positional and / or orientational harmony between one or more motion input control devices and the imaging device. In some examples, the motion input control device can be used to remotely operate the imaging device by reflecting changes in the position and / or orientation of the motion input control device to corresponding changes in the position and / or orientation of the imaging device. In some examples, one or more kinematic models of the motion input control device, the imaging device, and / or the articulated arm to which the imaging device is attached can be used to translate changes in the motion input control device into corresponding changes in the imaging device. In some examples, one or more kinematic models can be used to determine one or more coordinate transformation matrices that map changes in the motion input control device to corresponding changes in the imaging device. In some examples, the coordinate transformation matrices can implement one or more shift transformations and / or scaling transformations. In some examples, changes in the position and / or orientation of the imaging device can be performed by sending one or more commands to an actuator in the articulated arm to which the imaging device is attached.

[0108] At process 1045, it is determined whether the deactivation of the imaging device motion mode has been detected. A process similar to process 1035 is used to determine whether to exit the imaging device motion mode. If no deactivation of the imaging device motion mode is detected, manual control of the imaging device continues by repeating process 1040. If deactivation of the imaging device motion mode is detected, process 1050 is used to exit the imaging device motion mode.

[0109] At process 1050, the imaging device motion mode is exited. The imaging device motion mode is also exited when it is deactivated during processes 1035 and / or 1045. In some examples, upon exiting the imaging device motion mode, any movement of the imaging device during recentering in process 1020 is terminated, and one or more motion input controls disengage from controlling the position and / or orientation of the imaging device. In some examples, upon exiting the imaging device motion mode, manual control and / or recentering control of the imaging device terminates. In some examples, upon exiting the imaging device motion mode, the electronics may return to a mode where one or more motion input controls become dormant and / or resume control of one or more end effectors of the electronics.

[0110] Some examples of control units, such as control unit 130, may include non-transitory, tangible machine-readable media having executable code that, when run by one or more processors (e.g., processor 140), can cause one or more processors to perform processes of methods 200, 500, 600, 800, 900, and / or 1000. Some common forms of machine-readable media that may include processes of methods 200, 500, 600, 800, 900, and / or 1000 are, for example, floppy disks, floppy hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper cards, any other physical media with perforations, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media from which a processor or computer is adapted to read.

[0111] While illustrative embodiments have been shown and described, a wide range of modifications, variations, and substitutions are contemplated in the foregoing disclosure, and in some cases, certain features of the embodiments may be employed without a corresponding use of other features. Many variations, substitutions, and modifications will be recognized by those skilled in the art. Therefore, the scope of the invention should be limited only by the appended claims, and it is appropriate that the claims be interpreted broadly and in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A computer-aided medical device, comprising: One or more end effectors; Imaging devices; One or more input control devices for remotely operating the one or more end effectors; and A control unit, comprising one or more processors coupled to the end effector, the imaging device, and the input control device; The control unit is configured to: In response to a relocation center request, the remote operation control of the end effector by the input control device is suspended; Determine the field-of-view recenter movement for the imaging device so that the end effector is included within the field-of-view space of the imaging device; Determine the recenter movement of one or more input control devices to provide positional and orientation harmony between each of the input control devices and the corresponding end effector of the end effector; Perform the field-of-view recentering movement and input control device recentering movement; as well as Restore the remote operation control of the end effector by the input control device.

2. The apparatus of claim 1, wherein the control unit is further configured to coordinate the execution of the field-of-view recentering movement and the input control device recentering movement.

3. The apparatus according to claim 1, wherein the field of view recentering movement and the input control device recentering movement are performed simultaneously.

4. The apparatus of claim 1, wherein the control unit is further configured to determine whether the field of view recentering movement and the input control device recentering movement are effective.

5. The apparatus of claim 4, wherein the control unit is further configured to indicate an error when one or more of the field-of-view recenter movement and the input control device recenter movement are invalid.

6. The apparatus of claim 1, wherein the imaging device is an endoscope.

7. The apparatus of claim 1, wherein, in order to determine the field-of-view recenter movement, the control unit is further configured to: Determine the view center point based on one or more targets associated with the end effector; Determine the working distance for the imaging device; and Based on the viewing center point and the working distance, the desired position and desired orientation for the imaging device are determined.

8. The apparatus of claim 7, wherein each of the targets is a virtual sphere focused on a point of interest associated with the corresponding end effector of the end effector.

9. The apparatus of claim 7, wherein the control unit is further configured to determine the field of view direction based on a vector between a remote center associated with the imaging device and the center of vision.

10. The apparatus of claim 7, wherein the control unit is further configured to determine the field of view direction based on a vector between a reference point associated with the imaging device and the center of view.

Images