Imaging device control in an observation system

Abstract

Techniques for imaging device control in a computer-assisted device are disclosed, the computer-assisted device including a repositionable structure configured to support an imaging device and a control system coupled to the repositionable structure. The control system is configured to: determine a position of a reference point offset from a display unit; convert the position of the reference point to a target position of another reference point offset from the imaging device; determine a movement command for the first repositionable structure that moves the imaging device such that the other reference point moves toward the target position; and cause actuation of the repositionable structure based on the movement command.

Description

[0001] Related applications This application claims the benefit of U.S. Provisional Application No. 62 / 123,936, filed on December 10, 2020, entitled “Imaging Device Control in Viewing Systems,” the contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure generally relates to an electronic device, and more specifically to the control of a device having a repositionable imaging device. Background Technology

[0003] Increasingly, devices are being replaced by computer-aided electronic devices. This is especially true in industrial, recreational, educational, and other settings. As an example in medicine, modern hospitals have a large number of electronic devices in operating rooms, interventional suites, intensive care units, emergency rooms, and / or similar wards. Many of these devices may be capable of autonomous or semi-autonomous movement. Personnel also typically use one or more input devices located at the user control system to control the movement and / or operation of the electronic devices. As a specific example, minimally invasive robotic remote surgery systems allow surgeons to operate on patients from the bedside or a remote location. Remote surgery generally refers to surgery performed using a surgical system where the surgeon uses some form of remote control (e.g., a servo mechanism) to manipulate the movement of surgical instruments, rather than directly holding and moving the instruments by hand.

[0004] When electronic devices are used to perform tasks at a work site, one or more imaging devices (e.g., endoscopes, optical cameras, and / or ultrasound probes) can acquire images of the work site, providing visual feedback to the operator monitoring and / or performing the task. The imaging devices can also be controllable to update the field of view of the work site provided to the operator via a display unit. For example, the imaging devices can be attached to a repositionable structure comprising two or more links connected together by one or more structures, wherein the repositionable structure is movable (including through internal reconfiguration) to update the position and / or orientation of the imaging devices at the work site. In such cases, the movement of the imaging devices can be controlled by the operator, other personnel, or automatically, enabling changes to the field of view of the work site.

[0005] One method for controlling an imaging device is to move the imaging device to follow the movement of the display unit. For example, the operator's head movements can be tracked via a sensor system and used to control the movement of the imaging device. Another method for controlling an imaging device is to move the imaging device to follow the movement of a hand-operated controller. For example, the operator can control a hand-operated controller with two manipulators that can be manipulated together as analog handles. However, this method of controlling the imaging device based on different input patterns has not yet been combined to control the same imaging device. Furthermore, if the imaging device cannot follow multiple input patterns due to range of motion limitations, collisions, and / or similar conditions associated with the imaging device and / or the repositionable structure on which the imaging device is mounted, the specific input pattern causing the problem cannot be easily identified, for example, to provide tactile feedback that prevents the operator from further attempting to move the specific input pattern in a way that cannot be followed.

[0006] Therefore, there is a need for methods and systems for controlling repositionable imaging devices. Summary of the Invention

[0007] According to some embodiments, the computer-aided device includes a first repositionable structure configured to support an imaging device and a control system coupled to the repositionable structure. The control system is configured to: determine the position of a second reference point offset from a display unit; convert the position of the second reference point into a target position of the first reference point offset from the imaging device; determine a movement command for the first repositionable structure, the movement command moving the imaging device such that the first reference point moves toward the target position; and cause actuation of the first repositionable structure based on the movement command.

[0008] According to some embodiments, a method is disclosed for operating a computer-aided device including a repositionable structure and one or more processors, wherein one or more processors are communicatively coupled to the repositionable structure, the method comprising: determining a position of a first reference point offset from a display unit; converting the position of the first reference point into a target position of a second reference point offset from an imaging device; determining a movement command for the repositionable structure, the movement command moving the imaging device such that the first reference point moves toward the target position, wherein the repositionable structure is configured to support the imaging device; and causing actuation of the repositionable structure based on the movement command.

[0009] Other embodiments include, but are not limited to, one or more non-transitory machine-readable media comprising a plurality of machine-readable instructions which, when executed by one or more processors, are adapted to cause one or more processors to perform any of the methods disclosed herein.

[0010] It should be understood that the foregoing and the following detailed description are substantially exemplary and explanatory, and are intended to provide an understanding of the present disclosure without limiting its scope. In this regard, additional aspects, features, and advantages of the present disclosure will become apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0011] Figure 1 This is a simplified diagram of examples of remote operating systems according to various embodiments.

[0012] Figure 2 This is a perspective view of an exemplary display system according to some embodiments.

[0013] Figure 3 The examples illustrate that the display unit may have degrees of freedom that do not match the degrees of freedom of the imaging device.

[0014] Figure 4 The mapping between reference points in the display unit workspace and reference points in the imaging device workspace is shown according to various embodiments.

[0015] Figure 5 The differences between the display unit reference point and the display unit baseline reference point, and between the imaging device reference point and the imaging device baseline reference point, according to various embodiments, are shown.

[0016] Figure 6 A simplified diagram of a method for controlling an imaging apparatus according to various embodiments is shown.

[0017] Figure 7 Various embodiments are shown. Figure 6 A more specific process of the method.

[0018] Figure 8 Various embodiments are shown. Figure 6 Another more specific process of the method. Detailed Implementation

[0019] This specification and accompanying drawings, which illustrate inventive aspects, embodiments, implementation methods, or modules, should not be considered limiting—the claims define the protected invention. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this specification and claims. In some cases, well-known circuits, structures, or techniques have not been specifically shown or described so as not to obscure the invention. Similar numbers in two or more figures represent the same or similar elements.

[0020] In this specification, specific details are set forth to describe some embodiments according to this disclosure. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to one skilled in the art that some embodiments may be implemented without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative rather than restrictive. One skilled in the art will recognize that other elements (though not specifically described herein) may be within the scope and spirit of this disclosure. Furthermore, to avoid unnecessary repetition, one or more features shown and described with respect to one embodiment may be incorporated into other embodiments unless otherwise specifically described or if said one or more features would render the embodiment inoperable.

[0021] Furthermore, the terminology used in this specification is not intended to limit the invention. For example, as shown in the figures, spatially relative terms such as “below,” “under,” “below,” “above,” “over,” “near,” “far,” etc., may be used to describe the relationship of one element or feature to another. In addition to the positions and orientations shown in the figures, these spatially relative terms are also intended to cover different positions (i.e., orientations) and orientations (i.e., rotational placements) of elements or their operations. For example, if the contents of one of the figures are flipped, an element or feature described as “below” or “below” other elements or features will be “above” or “over” said other elements or features. Thus, the exemplary term “below” can encompass both above and below positions and orientations. Devices may also be oriented in other ways (rotated 90 degrees or in other orientations), and the spatially relative descriptive terms used herein may be interpreted accordingly. Similarly, descriptions of movement along and around various axes include various specific element positions and orientations. Furthermore, the singular forms “a,” “an,” and “this” attempt to include multiples unless the context explicitly indicates a single one. Furthermore, the terms "comprising," "including," "having," etc., indicate the presence of the listed features, steps, operations, elements, and / or components, but do not exclude the presence of one or more other features, steps, operations, elements, components, and / or sets thereof. Components described as connected may be directly electrically or mechanically connected, or they may be indirectly connected via one or more intermediate components.

[0022] Where feasible, elements described in detail with reference to one embodiment, implementation, or module may be included in other embodiments, implementations, or modules that are not specifically shown or described therein. For example, if an element is described in detail with reference to one embodiment but not with reference to a second embodiment, that element may still be claimed as being included in the second embodiment. Therefore, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects unless specifically described otherwise, unless one or more elements would render one embodiment or implementation inoperable, or unless two or more elements provide conflicting functionality.

[0023] In some cases, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0024] This disclosure describes various devices, elements, and computer-aided devices and parts thereof according to their state in three-dimensional space. As used herein, the term "position" refers to the orientation of an element or part of an element in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x-, y-, and z-coordinates). As used herein, the term "orientation" refers to the rotational placement of an element or part of an element (three rotational degrees of freedom, e.g., roll, pitch, and yaw). As used herein, the term "shape" refers to a set of positions or orientations measured along the element. As used herein, and for devices with repositionable arms, the term "proximal" refers to the direction along its kinematic chain toward the base of the computer-aided device, while "distal" refers to the direction along the kinematic chain away from the base.

[0025] The aspects of this disclosure are described with reference to computer-aided systems and apparatuses, which may include remotely operated, remotely controlled, autonomous, semi-autonomous, robotic, and / or similar systems and apparatuses. Furthermore, aspects of this disclosure are described with reference to embodiments using surgical systems, such as the daVinci® surgical system commercially available from Intuitive Surgical, Inc., Sunnyvale, California. However, those skilled in the art will understand that these examples are not limiting, and the inventive aspects disclosed herein can be embodied and implemented in various ways, including robotic embodiments and, where applicable, non-robotic embodiments. For example, the techniques described with reference to surgical instruments and methods can be used in other settings. Therefore, the instruments, systems, and methods described herein can be used for humans, animals, parts of human or animal anatomy, industrial systems, general-purpose robots, or remote operating systems. As a further example, the instruments, systems, and methods described herein can be used for non-medical purposes, including industrial use, general-purpose robotic use, sensing or manipulating non-tissue artifacts, cosmetic enhancements, human or animal anatomical imaging, data collection from human or animal anatomy, setting up or removing systems, training medical or non-medical personnel, and / or similar situations. Additional example applications include processing on tissue removed from human or animal anatomy (without returning the human or animal anatomy) and processing on human or animal carcasses. Furthermore, these techniques can also be used in medical treatment or diagnostic procedures, with or without surgical aspects.

[0026] System Overview Figure 1 This is a simplified diagram of an example remote operating system 100 according to various embodiments. In some examples, the remote operating system 100 may be a remotely operated medical system, such as a surgical system. As shown, the remote operating system 100 includes a follower device 104 and a guide device in a guide-follower configuration. In the guide-follower configuration, the follower simulates the movement of the guide. Figure 1 The guide device is shown as including an input system 102 in the form of a workstation (also referred to as a "console"). In various embodiments, the input system 102 can be any suitable form and may or may not include a workstation. Figure 1 In the example, follower device 104 is controlled by one or more input devices 106 of input system 102, as described in more detail below. A leader-follower configuration is sometimes referred to as a master-slave configuration, and a system that includes both leader and follower devices is sometimes referred to as a master-slave system, where the leader is the "master" and the follower is the "slave".

[0027] In this example, input system 102 includes one or more input devices that are contacted and manipulated by operator 108. As shown, input system 102 includes input devices 106 used by each of the operator's hands. Input devices 106 are supported by input system 102 and may be mechanically grounded. In some embodiments, an ergonomic support 110 (e.g., a forearm brace) may be provided on which operator 108 rests his or her forearm. In some examples, during a procedure, operator 108 can perform tasks at a work site near follower device 104 by commanding follower device 104 using input devices 106.

[0028] Display unit 112 is another input device included in input system 102. Display unit 112 can display images for operator 108 to observe. Display unit 112 can move with various degrees of freedom to adapt to the observation position of operator 108 and / or provide control functions. In an example of remote operating system 100, the displayed images can describe the work location where operator 108 is performing various tasks by manipulating input device 106 and / or display unit 112. In some examples, input system 102 can receive images displayed by display unit 112 from one or more imaging devices to capture images arranged at the work location. In other examples, the images displayed by display unit 112 can be generated by display unit 112 (or by other connected devices or systems), such as for use in virtual presentation tools, work locations, or for user interface components.

[0029] When using the input system 102, the operator 108 may stand or sit on a chair or other support, positioning his or her eyes to observe the display unit 112, manipulating the input device 106 and / or the display unit 112, and resting his or her forearms on the ergonomic support 110 as needed. In some embodiments, the operator 108 may stand at the input system 102 or assume other postures, and the display unit 112 and other input devices may be adjusted in position (height, depth, etc.) to accommodate the operator 108.

[0030] The remote operating system 100 may also include a follower device 104, which can be commanded by a guide device (such as input system 102). In a medical example, the follower device 104 can be located near an operating table (e.g., a table, bed, or other support) on which the patient may be positioned. In such a case, the work site can be located on or within the operating table, such as on or within a patient, a simulated patient, or a model (not shown). The illustrated follower device 104 includes a plurality of manipulator arms 120, each configured to be coupled to an instrument 122. The manipulator arms 120 are examples of repositionable structures on which instruments 122 (such as manipulators or instruments with imaging devices) can be mounted. The instrument 122 may include, for example, an end effector 126 and a housing configured to be coupled to the manipulator arms 120.

[0031] In various embodiments, one or more of the instruments 122 may include an imaging device for capturing images (e.g., an optical camera, a hyperspectral camera, an ultrasonic sensor, etc.). For example, one or more of the instruments 122 may be an endoscope that includes an imaging device that can provide a captured image of a portion of the working site for display via the display unit 112.

[0032] In some embodiments, in response to operator 108 manipulating the input device, manipulator arm 120 can be controlled to move, hinge, or actuate instrument 122 (such as by translating or rotating the entire instrument 122, hinged or actuating end effector 126, or any instrument joint hinged to the proximal side of end effector 126), thereby allowing operator 108 to perform a task at the work site. For a surgical example, the operator may direct manipulator arm 120 to move instrument 122 to perform surgical procedures at an internal surgical site through a micro-orifice or natural opening.

[0033] As shown, the control system 140 is provided externally to and communicates with the input system 102. In other embodiments, the control system 140 may be provided within the input system 102 and / or the follower device 104. As the operator 108 moves one or more of the input devices 106 and / or the display unit 112, sensed spatial information, including sensed position and / or orientation information, based on the movement of the input devices 106 and / or the display unit 112, is provided to the control system 140. The control system 140 may determine or provide control signals to the follower device 104 to control the movement of the manipulator arm 120 or the instrument 122 based on the received information and user input. In one embodiment, the control system 140 supports one or more wired communication protocols (e.g., Ethernet, USB, and / or similar protocols) and / or one or more wireless communication protocols (e.g., Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, wireless telemetry, and / or similar protocols).

[0034] The control system 140 can be implemented on one or more computing systems. One or more computing systems can be used to control the follower device 104. In addition, one or more computing systems can be used to control components of the input system 102, such as controlling the movement of the display unit 112 in response to input provided through the head of the operator 108.

[0035] As shown, the control system 140 includes a processor 150 and a memory 160 of a storage control module 170. In embodiments, the control system 140 may include one or more processors, non-persistent memory (e.g., volatile memory such as random access memory (RAM), cache memory), persistent memory (e.g., hard disks, optical disc drives such as compact disc (CD) drives or digital versatile disc (DVD) drives, flash memory, etc.), communication interfaces (e.g., Bluetooth interfaces, infrared interfaces, network interfaces, optical interfaces, etc.), and many other elements and functions. Furthermore, the functionality of the control module 170 can be implemented in any technically feasible software and / or hardware.

[0036] Each of the one or more processors in the control system 140 may be an integrated circuit for processing instructions. For example, the one or more processors may be a processor, a central processing unit (CPU), a microprocessor, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), a tensor processing unit (TPU), and / or one or more cores or microcores of the like.

[0037] The communication interface of the control system 140 may include an integrated circuit for connecting the computing system to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN), such as the Internet, a mobile network, or any other type of network) and / or to another device such as another computing system.

[0038] Furthermore, the control system 140 may include one or more output devices, such as a display device, printer, speaker, external memory, or any other output device. Many different types of computing systems can exist, and the aforementioned input(s) and output devices(s) may take other forms.

[0039] Software instructions in the form of computer-readable program code for performing embodiments of the present disclosure may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer-readable medium, such as a CD, DVD, storage device, disk, magnetic tape, flash memory, physical memory, or any other computer-readable storage medium. Specifically, the software instructions may correspond to computer-readable program code configured to perform some embodiments of the present invention when executed by one or more processors.

[0040] continue Figure 1 The control system 140 can be connected to a network or a part of a network. The network may include multiple nodes. The control system 140 can be implemented on a single node or on a group of nodes. For example, the control system 140 can be implemented on a node in a distributed system that is connected to other nodes. For example, the control system 140 can be implemented in a distributed computing system with multiple nodes, wherein different functions and / or components of the control system 140 can reside on different nodes within the distributed computing system. Furthermore, one or more elements of the aforementioned control system 140 can be located remotely and connected to other elements via a network.

[0041] In some embodiments, one or more input devices may be ungrounded (ungrounded input devices are kinematically ungrounded, and in this example, ungrounded input devices are configured to be held by the operator 108's hand without requiring additional physical support from hardware). Such ungrounded input devices can be used in conjunction with the display unit 112. In some embodiments, the operator 108 can use the display unit 112 positioned near the work site, allowing the operator 108 to manually operate instruments at the work site, such as laparoscopic instruments in the surgical example, while observing the images displayed on the display unit 112.

[0042] Some embodiments may include one or more components of a remotely operated medical system, such as the daVinci® surgical system, commercially available from Intuitive Surgical, Inc., Sunnyvale, California, USA.

[0043] Figure 2 This is a perspective view of an exemplary display system 200 according to various embodiments. In some embodiments, the display system 200 is used on a workstation with a remote operating system (e.g., in...). Figure 1 In the workstation shown, or the display system 200 can be used in other systems or as a standalone system, for example to allow operators to observe work areas or other physical locations, or to display virtual environments, etc. Although Figure 2 A specific configuration is shown, but other embodiments may use different configurations.

[0044] The display system 200 includes a base support 202, an arm support 204, and a corresponding... Figure 1 The display unit 206 of the display unit 112. The display unit 206 has multiple degrees of freedom of movement provided by a support linkage device, which includes a base support 202, an arm support 204 connected to the base support 202, and an inclined member 224 (described below) connected to the arm support 204, wherein the display unit 206 is connected to the inclined member 224.

[0045] The base support 202 may be a vertical member mechanically grounded, for example, directly or indirectly connected to the ground, such as by resting on or attaching to a floor. For example, the base support 202 may be mechanically coupled to a support structure 210 connected to the ground. The base support 202 includes a first base portion 212 and a second base portion 214, which are coupled such that the second base portion 214 can translate relative to the first base portion 212 with linear degrees of freedom.

[0046] The arm support 204 may be a horizontal member mechanically connected to the base support 202. The arm support 204 includes a first arm portion 218 and a second arm portion 220. The second arm portion 220 is connected to the first arm portion 218 such that the second arm portion 220 can be linearly translated relative to the first arm portion 218 with a first linear degree of freedom (DOF).

[0047] The display unit 206 can be mechanically coupled to the arm support 204. The display unit 206 can be moved with a second linear DOF provided by the linear translation of the second base portion 214 and the second arm portion 220.

[0048] In some embodiments, the display unit 206 includes a display device capable of displaying digital images, such as one or more displays, projectors, or other display devices. The display unit 206 may include two viewports 223, wherein display device devices are provided behind or included within the viewports. In some embodiments, one or more displays or other display devices may be positioned on the display unit 206 in place of the viewports 223.

[0049] In some embodiments, display unit 206 displays an image of the working site (e.g., the patient's internal anatomy in a medical example) captured by an imaging device such as an endoscope. The working site may alternatively be a virtual representation of the working site. While one or more of these instruments 122 are controlled by an operator via an input device of input system 102, the image may display a captured image or virtual rendering of the instrument 122 of follower device 104.

[0050] In some embodiments, the display unit 206 is rotatably coupled to the arm support 204 by a tilting member 224. In the illustrated example, the tilting member 224 is coupled at its first end to a second arm portion 220 of the arm support 204 by a rotary coupler configured to provide rotational movement of the tilting member 224 and the display unit 206 relative to the second arm portion 220 about a tilt axis 226. In some embodiments, the tilt axis 226 is positioned above the display device in the display unit 206.

[0051] Each of the various degrees of freedom discussed herein can be passive and require manual manipulation, or can be moved by one or more actuators, such as one or more motors, solenoid valves, etc. For example, the rotational movement of the tilting member 224 and the display unit 206 about the tilt axis 226 can be driven by one or more actuators, such as a motor coupled to the tilting member at or near the tilt axis 226.

[0052] Display unit 206 can be rotatably coupled to tilt member 224 and can rotate about yaw axis 230. For example, this could be a lateral or left-right rotation starting from the viewpoint of an operator viewing an image of display unit 206 via viewport 223. In this example, display unit 206 is coupled to tilt member by a rotating mechanism, which could be a track mechanism. For example, in some embodiments, the track mechanism includes a curved track 228 that slidably engages a recessed member 229 coupled to tilt member 224, thereby allowing display unit 206 to rotate about yaw axis 230 by causing the curved track 228 to move through the recess of recessed member 229.

[0053] The display system 200 can therefore provide the display unit 206 with vertical linearity 216, horizontal linearity 222, rotational (tilt) degree of freedom 227, and rotational yaw degree of freedom 231. The coordinated combination of movements of the components of the display system 200 in these degrees of freedom allows the display unit 206 to be positioned in various locations and orientations within its workspace. The movement of the display unit 206 in the tilt, horizontal, and vertical degrees of freedom allows the display unit 206 to remain near or in contact with the operator's head when the operator is providing head input through head movements.

[0054] The degrees of freedom of the display system 200 allow the display unit 206 to pivot in physical space about pivot axes that may be located in different orientations. For example, the display system 200 can provide the display unit 206 in physical space corresponding to the movement of the operator's head when the operator is operating the display system 200. Such movement may include rotation about a defined neck pivot axis that approximately corresponds to the neck axis of the operator's head at the operator's neck. Rotation allows the display unit 206 to be moved according to the operator's head guiding the movement of the display unit 206. In another example, the movement may include rotation about a defined forehead pivot axis that approximately corresponds to the forehead axis extending through the operator's head at the forehead when the display unit 206 is oriented as shown in a central yaw rotation position about the yaw axis 230.

[0055] Display unit 206 may include one or more input devices that allow an operator to provide input to manipulate the orientation and / or position of display unit 206 in space, and / or manipulate other functions or components of display system 200 and / or larger system (e.g., remote operating system).

[0056] Schematic, the display unit 206 includes a head input area 242. In some embodiments, the head input area 242 is positioned on the surface of the display unit 206 facing the operator's head during operation of the display unit 206.

[0057] The head input area 242 can be shaped to form a headrest that can contact the operator's head when the operator is providing head input. More specifically, the head input area 242 can be located in an area above the viewport 223 so that it contacts the operator's forehead while the operator is viewing an image through the viewport 223. The display unit 206 may include one or more head input sensors that sense operator head input to the head input area 242 as a command to cause movement of the imaging device, or otherwise cause an update of the view in the image presented to the operator (e.g., through graphics rendering, digital zoom, or panning). Furthermore, in some embodiments and under some operating conditions, the sensed head movement is used to move the display unit 206 to compensate for the head movement. Therefore, even when the operator performs head movements to control the field of view provided by the imaging device, the position of the operator's head can remain stationary relative to the viewport 223. This ensures proper alignment of the operator's eyes with the viewport.

[0058] In some embodiments, sensing operator head input includes sensing the presence of or contact between the operator's head or a portion of the head (e.g., the forehead) and the head input area 242. One or more head input sensors may include any of various types of sensors, such as resistive sensors, capacitive sensors, force sensors, optical sensors, etc.

[0059] continue Figure 2 The orientation and / or position of the display unit 206 can be changed by the display system 200 based on operator head input to the head input area 242. For example, the sensed operator input is provided to the control system (e.g., control system 140), which controls the actuators of the display system 200 to move the second base portion 214 with linear degree of freedom 216, the second arm portion 220 with linear degree of freedom 222, the tilting member 224 with rotational degree of freedom 227, and / or the display unit 206 with rotational degree of freedom 231, so that the display unit 206 moves as commanded (e.g., according to) the sensed operator head input. The sensed operator head input can also be used to control the display system 200 and / or a larger system (e.g., Figure 1 Other functions of the remote operating system 100. Therefore, in some embodiments, the operator can move his or her head to provide input, thereby controlling the display unit 206 to move by the display system 200 according to the movement of the head, thereby allowing the display unit 206 to follow the movement of the operator's head and change the viewing angle.

[0060] In some embodiments, the image displayed by the display unit 206 and / or other controlled devices is altered and manipulated based on the sensed motion of the display unit 206. In some embodiments of the display system, the display unit 206 may rotate about a yaw axis 230 with a degree of freedom 231, and one or more other degrees of freedom 216, 222, and 227 are omitted from the display system 200. For example, the display unit 206 may be rotated about the yaw axis 230 (e.g., by one or more actuators and / or manually by an operator), and the display unit 206 may be manually positioned higher and / or lower (e.g., by one or more actuators and / or manually by an operator), for example by using a base support 202 or other mechanism, where the horizontal degree of freedom 222 and / or the tilting degree of freedom 227 are omitted.

[0061] Those skilled in the art will realize Figure 2 Only one example of the display system configuration is shown. Alternative configurations supporting movement of the display unit 206 based on operator input are also possible. Any linkage device supporting the desired movement of the display unit 206 can be used instead. Figure 2 The configuration shown.

[0062] While described herein primarily with reference to display unit 206 as part of a grounded mechanical structure (e.g., display system 200), in other embodiments the display unit can be one or more technically feasible display devices. For example, the display unit can be a handheld device held by an operator, such as a tablet or mobile phone. As another example, the display unit can be a head-mounted device (e.g., glasses, goggles, helmet). In such cases, the position and / or orientation of the display unit can be determined using one or more accelerometers, gyroscopes, inertial measurement units, cameras, and / or other sensors, either inside or outside the display unit.

[0063] Imaging device control technology As described, in some embodiments, the head input device in the display unit can include one or more head input sensors that sense operator head input (e.g., head input applied to, or moved or repositioned by, the display unit, such as display unit 206). For example, the head input device in display unit 206 can capture and convert head input provided by head force or movement to command a repositionable structure fitted with an imaging device (e.g., within an endoscope). In the example of an endoscope, the endoscope can capture and provide an image of a portion of the working site, which is displayed as output via a display unit of the input system (e.g., display unit 112 of input system 102).

[0064] Figure 3Various degrees of freedom of the display unit, the hand-operated controller, and the imaging device according to various embodiments are illustrated. As shown in panel A, in some examples, the display unit 310 (corresponding to...) Figure 1 The display unit 112 and Figure 2 The display unit 206 is movable on four DOFs. Figure 3 In the specific system posture shown, operator 302 can cause display unit 310 to translate vertically (up-down on the paper) at DOF 314 and horizontally (left-right on the paper) at DOF 316. In the specific system posture shown, operator 302 can also cause display unit 310 to rotate about the vertical axis (yaw) at DOF 318 and about the horizontal axis (pitch) at DOF 312. For example, as described above... Figure 2 The orientation and / or position of the display unit 206 can be changed based on operator head input to the head input area 242 to manually move the display unit 206 or control the actuators of the display system 200 to move the display unit 206. For example, by causing the second base portion 214 to move with a linear degree of freedom 216, the second arm portion 220 to move with a linear degree of freedom 222, the tilting member 224 to move with a rotational degree of freedom 227, and / or the display unit 206 to move with a rotational degree of freedom 231, the movement provided manually or by the actuator can follow the movement of the operator's head.

[0065] As shown in panel B, when mounted on and operated on the manipulator arm 120, which includes a repositionable structure, the imaging device 320 can be restricted to movement at three degrees of freedom (DOF). In some examples, the imaging device 320 can be introduced into the working area via an inlet / outlet port or cannula. For example, the imaging device 320 can be included in an endoscope, as described above. Figure 1 In some such cases, the imaging device 320 may have a pitch DOF 324, a yaw DOF 326, and an insertion-retraction DOF 322 with respect to a remote center of motion associated with the inlet / outlet port or cannula. For example, the imaging device 320 may be a 0° endoscope, in which case the viewing direction of the imaging device 320 corresponds to the insertion direction of the insertion-retraction DOF 322. As another example, the imaging device may be a 30° endoscope, in which case the viewing direction of the imaging device 320 may deviate by 30° from the insertion direction of the insertion-retraction DOF 322. It should be noted that the endoscope may also have a roll DOF for rotation about the axis of the endoscope, or provide one or more connectors for additional DOFs along the axis of the endoscope. However, the display unit 310 may not have a corresponding DOF for roll or such additional connectors.

[0066] While this description is by way of example of a display unit 310 configured to move at four degrees of freedom (DOF), in other embodiments, the display unit may be configured to move at more than four DOFs. For example, as described above in conjunction with... Figure 2 The display unit may be a handheld device, such as a tablet or mobile phone, or a head-mounted device that the operator can translate and rotate like a rigid body in up to six degrees of freedom. While described herein by reference to an imaging device 320 having three DOFs, in other embodiments, the imaging device 320 may have more than three DOFs. For example, the imaging device may be an endoscope having a distal end coupled to an axis via one or more additional connectors, forming a “wrist” that provides one or more DOFs different from the three DOFs described above with respect to the imaging device 320. While described herein primarily with reference to a display unit 310 without a roll DOF, in other embodiments, such as when the display unit is a handheld device or a head-mounted device, the display unit may have a roll DOF that can be mapped to the roll DOF of the imaging device 320. More generally, in some embodiments, the display unit 310 may have any suitable DOFs that differ in number and / or direction from the DOFs of the imaging device 320.

[0067] In some embodiments, movement of the display unit 310 (which may substantially follow the head movement of the operator 302) is not directly translated into movement commands for the imaging device 320. This can occur in embodiments where the display unit 310 and the imaging device 320 have matching DOF and / or range of motion constraints, or where the display unit 310 and the imaging device 320 have different DOF and / or range of motion constraints. In some examples, certain movements of the display unit 310 (e.g., up and down) cannot be explicitly followed by the imaging device 320 due to DOF or range of motion constraints, and in such cases, attempting to control the movement of the imaging device 320 to directly follow the movement of the display unit 310 may produce unexpected, unintuitive, and / or deviations from the movement of the display unit 310.

[0068] Figure 4The mapping between reference points in the workspace of the display unit and reference points in the workspace of the imaging device, according to various embodiments, is illustrated. As shown in panels 4A-B, in order to map the movement of the display unit 310 to the target movement of the imaging device 320, the control system (e.g., control system 140) maps the position of a reference point 400 offset from the display unit 310 (also referred to herein as "display unit reference point 400") to a target position of a reference point 428 offset from the imaging device 320 (also referred to herein as "imaging device reference point 428"). As shown, the display unit reference point 400 is offset from the display unit 310 in a direction away from the operator 302, in front of the display unit. Similarly, the imaging device reference point 428 is oriented toward the field of view of the imaging device and offset from the imaging device 320 in front of the imaging device 320. The control system converts the position of the display unit reference point 400 into the target position of the imaging device reference point 428, such that the movement of the imaging device reference point 428 toward the target position is similar to the movement of the display unit reference point 400. Therefore, by using the imaging device reference point 428 and the display unit reference point 400, operator input that changes the position of the display unit 310 can be mapped to the command position, speed, or acceleration of the imaging device 320.

[0069] In some examples, the display unit reference point 400 is located at a distance 402 in front of the lens of the display unit 310 (e.g., the lens within the viewport 223 of the display unit 206) in a direction away from the operator 302 (i.e., in front of the operator 302's eyes). In some examples, the display unit reference point 400 may be approximately 30 cm in front of the lens of the display unit 310 in a direction away from the operator 302, which is roughly the distance between the operator 302's eyes and the operator 302's hand. In some examples, such as when the display unit is a handheld device or a head-mounted device, the display unit reference point 400 may be approximately 30 cm in front of the operator's eyes. Additionally, in some examples, including cases where the display unit reference point 400 is approximately 30 cm in front of the operator's eyes, the imaging device reference point 428 may be approximately 10 cm in front of the distal end of the imaging device 320, which is roughly the distance between the working part of the imaging device 320 and one or more remotely operated instruments (e.g., the instrument 122 of the remotely operated follower device 104). In some embodiments, the distance of the display unit reference point 400 in front of the operator 302 and / or the distance of the imaging device reference point 428 in front of the imaging device 320 may not be fixed. For example, the image captured by the imaging device 320 and presented to the operator via the display unit 310 can be modified by optical or digital zoom or panning of the imaging device 320 and / or by optical zoom or panning applied to the image displayed by the display unit 310. In such cases, the imaging device reference point 428 can be moved relative to the imaging device 320 based on optical or digital zoom and panning. For example, optical or digital zoom can cause the imaging device reference point 428 to move along the central axis of the field of view and closer or farther away from the imaging device 320, while optical or digital panning can cause the imaging device reference point 428 to move away from the central axis of the field of view and laterally relative to the imaging device 320.

[0070] As shown, the position of the display unit reference point 400 can be represented in any suitable coordinate system, such as using Cartesian coordinates (x, y, z). d y d , z d The axes 412, 414, and 426 are shown. The position of the image device reference point 428 can also be represented in Cartesian coordinates (x, y, y). i y i , z i ), which are shown as axes 422, 424, and 426. Regardless of the DOF of display unit 310, as combined Figure 3 As described above, movement of display unit 310 at any point of view (DOF) of display unit 310 can be projected to a distance 402 in front of display unit 310 to achieve a certain effect. d yd and z d The aspect determines the display unit reference point 400. Similar mapping can also be used to determine the movement of the imaging device 320 at any DOF of the imaging device 320 and the imaging device reference point 428 in the x-axis. d y d and zd Relationships between movements in coordinate systems (or other coordinate systems).

[0071] In some embodiments, the control system solves for the position of the display unit reference point 400 by using the known joint positions of the forward kinematics and the repositionable structure supporting the display unit 310. For example, the control system itself may calculate the joint positions required to follow the head movement of the operator 302, or obtain these joint positions from another module that calculates the joint positions. Based on the joint positions, the position of the lens of the display unit 310 is solved, and a distance 402 is added in front of the lens position in a direction perpendicular to the lens's viewing plane to determine the position of the display unit reference point 400. In other embodiments, the control system can determine the position of the display unit reference point 400 in any technically feasible manner. For example, when the display unit is a handheld device and / or an ungrounded head-mounted display, the position of the display unit may be determined by data captured by sensors of the display unit and / or the head-mounted display and / or by data captured using one or more external sensor systems. A distance can then be added to the position of the display unit to obtain the position of the display unit reference point.

[0072] In some embodiments, after determining the position of the display unit reference point 400, the control system generates a control command that controls the repositionable structure on which the imaging device 320 is mounted to move to the corresponding position, so that the imaging device reference point 428 substantially follows the position of the display unit reference point 400. The final position of the imaging device reference point 428 is also referred to herein as the "target" position. The target position can be defined using a scaling factor; in this case, a scaling factor is applied to the movement of the display unit reference point 400 to obtain the corresponding target position of the imaging device reference point 428. For example, the display unit reference point 400 moves along the x-axis... d y d and z d Cartesian motion with degrees of freedom can be mapped to a corresponding target position experiencing a scale factor. In some examples, the scale factor can be determined based on operator preference, the type of display unit 310, the type of imaging device 320, the executed program, and / or the like.

[0073] In some embodiments, the repositionable structure on which the imaging device 320 is mounted can be actuated to update the position and / or orientation of the imaging device 320 in the working area, so that the imaging device reference point 428 moves to or toward a target position. In such a case, the control system can use the target position of the imaging device reference point 428 to determine the position of the distal end of the imaging device 320. Subsequently, the reverse kinematics of the imaging device 320 and / or the repositionable structure on which the imaging device 320 is mounted can be used to determine how to actuate the imaging device 320 and / or the connector on which the imaging device 320 is mounted to move the imaging device 320 accordingly.

[0074] In some embodiments, one or more instruments (e.g., one or more of instruments 122) move relative to a reference frame that remains unchanged as the imaging device 320 moves. For example, in the follower mode described above, one or more instruments may move relative to a reference frame based on commands generated using input from a handheld controller. In such cases, different views of the environment enabled by the movement of the imaging device 320 do not affect the reference frame in which the controlled instruments are positioned. The different views of the environment only change the field of view of the imaging device 320 relative to the instruments, which alters the effective viewpoint presented to the instrument operator by the image captured by the imaging device 340.

[0075] Figure 5 The differences between the display unit reference point and the display unit baseline reference point, and between the imaging device reference point and the imaging device baseline reference point, according to various embodiments, are shown. As illustrated, in some examples, the position of the display unit reference point 400 is defined relative to the position of the display unit baseline reference point 502, and the position of the imaging device reference point 428 is defined relative to the position of the imaging device baseline reference point 512. For example, when operator 302 selects to enter a mode that converts the movement of the display unit 310 into a command causing the movement of the imaging device 320 to follow the movement of the display unit 310, the display unit baseline reference point 502 and the imaging device baseline reference point 512 may be points located, respectively, a certain distance away from the operator 302 in front of the display unit 310 and in front of the imaging device 320. The position of the display unit reference point 400 at a subsequent time can then be represented as a difference 504 (also referred to as "difference 504") between the current display unit reference point 400 and the display unit baseline reference point 502, which is denoted herein as... Similarly, the position of the imaging device reference point 428 at a subsequent time can then be represented as the difference 514 (also referred to as "difference 514") between the current imaging device reference point 400 and the imaging device baseline reference point 512, which is denoted here as As described, the position change of the display unit reference point 400 can be mapped to the position change of the imaging device reference point 428 by applying a scaling factor according to Equation 1.

[0076] , Equation 1 In Equation 1, the scaling factor can be, for example, static or variable. As a specific example, the scaling factor can be an optional value: less than 1 (e.g., 1 / 3) for more precise control of the imaging device 320; 1 for normal control of the imaging device 320; and / or greater than 1 (e.g., 1.5) for coarser control of the imaging device 320.

[0077] Furthermore, in some embodiments, the control system is capable of accommodating manual adjustments to the position of the display unit 310 and / or the imaging device 320. For example, operator 300 may ergonomically adjust the position of the display unit 310. In such a case, the position of the display unit 310 moves (e.g., ergonomically for the operator), and the control system does not command the movement of the imaging device 320 to follow the ergonomically adjusted movement. As another example, if operator 300 or someone uses other techniques to move the imaging device 320, the imaging device 320 can be manually adjusted. In such a case, the position of the imaging device 320 is shifted, but the position of the display unit 310 remains unchanged. As described more specifically below, in some examples, when an ergonomic adjustment to change the position of display unit 310 occurs, the control system: (1) updates the display unit baseline reference point 502 based on the position and orientation of display unit 310 upon completion of the ergonomic adjustment, setting the difference 504 of display unit reference point 400 to zero; (2) sets a correction equal to the difference 514 of imaging device reference point 428; and (3) performs a ratcheting technique to reduce corrections over multiple movement cycles. Furthermore, in some examples, when a manual repositioning of imaging device 320 occurs, the control system: (1) updates the imaging device baseline reference point 512 based on the position and orientation of imaging device 320 upon completion of the manual repositioning, setting the difference 514 of imaging device reference point 428 to zero; (2) determines a correction based on the difference 504 of display unit reference point 400 multiplied by -1; and (3) performs a ratcheting technique to reduce corrections over multiple movement cycles.

[0078] Furthermore, in some embodiments, when the imaging device reference point 428 cannot follow the display unit reference point 400, for example due to range of motion (ROM) limitations associated with the imaging device 320 and / or the repositionable structure on which the imaging device 320 is mounted, collisions, and / or similar situations, the control system can determine, according to Equation 2, a tactile feedback proportional to the difference between the realized difference of the imaging device reference point 428 and the target difference of the imaging device reference point 428, the target difference of the imaging device reference point 428 being determined by applying a scaling factor based on the difference 504 of the display unit reference point 400.

[0079] Equation 2 In equation 2, It is force feedback and It is a proportionality constant. The control system can further convert the tactile feedback into a joint force and / or torque of the repositionable structure with the display unit 310 mounted, which prevents the operator from further attempting to increase the difference between the achieved imaging device reference point and the target difference at the imaging device reference point 428, which is determined based on the difference 504 at the display unit reference point 400 by applying a proportionality factor. In some examples, the x-axis of the display unit reference point 400 can be used as a proportionality constant. d y d and / or z d Each of these features individually identifies and / or applies haptic feedback.

[0080] Figure 6 A simplified diagram of a method 600 for controlling an imaging apparatus according to various embodiments is shown. One or more of the processes 602-620 of method 600 may be implemented, at least in part, in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when run by one or more processors (e.g., processor 150 in control system 140), can cause said one or more processors to perform one or more of processes 602-620. In some embodiments, method 600 may be executed by one or more modules, such as control module 170 in control system 140. In some embodiments, method 600 may include additional processes not shown. In some embodiments, one or more of processes 602-620 are executed, at least in part, by one or more units of control system 140.

[0081] As shown, method 600 begins at process 602, where display unit baseline reference point 502 and imaging device baseline reference point 512 are initialized. In some embodiments, display unit baseline reference point 502 and imaging device baseline reference point 512 are initialized to display unit reference point 400 and imaging device reference point 428, wherein when the operator selects to enter a mode that converts movement of display unit 310 into a command causing movement of imaging device 320 to follow movement of display unit 310, display unit reference point 400 and imaging device reference point 428 may be located at a distance from the operator in front of display unit 310 and in front of imaging device 320, respectively. As described, the position of display unit reference point 400 at a subsequent time can be represented as the difference or variation between the current display unit reference point and display unit baseline reference point 502. Similarly, the position of imaging device reference point 428 at a subsequent time can be represented as the difference or variation between the current imaging device reference point and imaging device baseline reference point 512.

[0082] At process 604, the position of the display unit reference point 400 is determined. In some embodiments, the position of the display unit reference point 400 is determined as a distance in front of the display unit 310 in a direction away from the operator. In some examples, the position of the display unit reference point 400 can be determined using forward kinematics based on the joint position of the repositionable structure on which the display unit 310 is mounted, such as when the display unit moves with the operator 300's head. In other examples, the position of the display unit reference point 400 can be determined in any technically feasible manner, such as via one or more sensors of the display unit and / or using one or more external tracking systems.

[0083] At process 606, if display unit 310 has been manually adjusted, method 600 continues to process 608, where display unit baseline reference point 502 is updated and correction is determined. As described, display unit 310 can be manually adjusted, for example, when operator 300 makes ergonomic adjustments to the position of display unit 310. In some examples, display unit baseline reference point 502 is updated by setting the position of display unit reference point 400 when manual adjustment of display unit 310 is completed. Furthermore, a correction is created for the mapping between display unit reference point 400 and imaging device reference point 428, and then gradually reduced to zero via a ratchet technique described below to correct for changes in display unit baseline reference point 502 and avoid position jumps in the imaging device reference point. In some examples, the correction is defined as the difference between the difference 514 of imaging device reference point 428 and the difference 504 of display unit reference point 400, as described above. Figure 5For example, the correction can be determined by solving for the correction in Equation 3. Equation 3 In this scenario, when the display unit baseline reference point 502 is reset, the difference 504 of the display unit reference point 400 becomes zero, and the correction is equal to the difference 514 of the imaging device reference point 428. This correction ensures that the difference 504 of the imaging device reference point 428 remains the same after the operator 300 moves it, so that when the new difference of the imaging device baseline reference point 428 is determined based on the difference of the display unit reference point 400 relative to the new display unit baseline reference point, the operator 300 will not see a discontinuous change to another position. That is, the correction compensates for the difference between the difference 504 of the display unit reference point 400 and the difference 514 of the imaging device reference point 428 caused by manual adjustment of the display unit 310. As described more specifically below, the correction is gradually reduced to zero over multiple movement cycles, after which the difference 514 of the imaging device reference point 428 follows the difference 504 of the display unit reference point 400 according to Equation 1.

[0084] If it is determined that the imaging device 320 has been manually adjusted at process 610, method 600 proceeds to process 612, where the imaging device baseline reference point 512 is updated and a correction is determined. In some examples, the imaging device baseline reference point 512 is updated by setting it to the position of the imaging device reference point 428 when the manual adjustment of the imaging device 320 is completed. Similar to process 608, a correction for the mapping between the display unit reference point 400 and the imaging device reference point 428 is determined at process 610, and this correction is then gradually reduced to zero via a ratchet technique to correct for the changing imaging device baseline reference point. In some examples, the correction is set to multiply the difference 504 of the display unit reference point 400 by -1 (and optionally a scaling factor), because the difference of the imaging device reference point in Equation 3 becomes zero when the imaging device baseline reference point 512 is updated.

[0085] Then proceed to process 612, or if it is not determined at process 610 that the imaging device 320 has been manually adjusted, determine the target position of the imaging device reference point 428 at process 614. Figure 7Procedure 614 is illustrated more specifically in accordance with various embodiments. In some embodiments, processes 702-710 are performed for each of a plurality of axes (e.g., each of the x, y, and z axes), respectively. As shown, at process 702, if a non-zero correction from a previous movement cycle is identified, method 600 proceeds to process 704, where a new (k-th) correction is determined. In some examples, the new (k-th) correction can be determined as: Equation 4 where k and k-1 indicate the k-th and (k-1)-th movement cycles, respectively.

[0086] At process 706, if the magnitude of the new (k-th) correction is greater than the magnitude of the previous (k-1)-th correction, then at process 708, the correction is set to the previous (k-1)-th correction. On the other hand, if the magnitude of the k-th correction is not greater than the magnitude of the previous (k-1)-th correction, then at process 710, the correction is set based on the new (k-th) correction, the previous (k-1)-th correction, and a ratchet factor. In some examples, processes 706-710 can be performed according to Equation 5.

[0087] If , then Else Equation 5 where r is a ratchet factor with 0 < r ≤ 1. In some examples, r can be selected based on one or more of operator preferences, the type of display unit 310, the type of imaging device 320, the program executed, and / or the like.

[0088] Subsequent to processes 708 and 710, or if no correction is identified at process 702, then at process 712, a target position of the imaging device reference point 428 is determined based on the display unit reference point 400 and the correction (if any). In some examples, process 712 can be performed according to Equation 6. Equation 6 In Equation 6, the correction (from Equation 5) is used to reduce the movement of the imaging device reference point 428 based on the ratchet factor until the correction becomes zero after several cycles. The more the movement of the imaging device reference point 428 is reduced, the more noticeable the effect on the operator will be; however, the faster the correction is eliminated, allowing the imaging device reference point 428 to follow the display unit reference point 400 according to Equation 1. In some examples, the ratchet factor r can be set to 1. In such a case, the imaging device reference point 428 (and the image seen by the operator) will not move until the correction is eliminated and the imaging device reference point 428 directly follows the display unit reference point 400. Furthermore, by allowing the correction to become only a small number, it is ensured that the correction converges to zero and the difference 514 of the imaging device reference point 428 converges to the difference 504 of the display unit reference point 400 (multiplied by the scaling factor).

[0089] return Figure 6 After determining the target position of the imaging device reference point 428, method 600 proceeds to process 616. In process 616, if the target position of the imaging device reference point 428 can be achieved, method 600 proceeds to process 618, where the imaging device 320 is actuated based on the target position of the imaging device reference point 428.

[0090] In some cases, if the target position of the imaging device reference point 428 cannot be achieved, method 600 proceeds directly to process 620, where an achievable position is determined based on the target position of the imaging device reference point 428, and the imaging device 320 is actuated to or toward the achievable position. In some examples, tactile feedback is also provided. For example, operator 300 may move the display unit to a position that the imaging device 320 cannot follow due to DOF or range of motion limitations associated with the imaging device 320 and / or the repositionable structure on which the imaging device 320 is mounted, collisions, and / or similar reasons. In such cases, the imaging device 320 can be repositioned to the achievable position; tactile feedback may also be provided. Tactile feedback may drive the operator to move the display unit toward a position corresponding to the achievable target position of the imaging device, and may prevent further movement of the display unit to the target position corresponding to the increase in the difference between the achieved imaging device reference point and the target difference of the imaging device reference point determined based on the difference of the display unit reference point.

[0091] Figure 8 A more specific process 620 according to various embodiments is illustrated. In some embodiments, processes 802-808 are performed separately for each of a plurality of axes (e.g., each of the x, y, and z axes).

[0092] As shown, at process 802, the achievable position of the imaging device 320 is determined. As described, the achievable position can be determined based on the target position as the position of the imaging device 320 that can be achieved while satisfying the DOF or range of motion limitations, collision constraints and / or the like associated with the imaging device 320 and / or the repositionable structure on which the imaging device 320 is mounted.

[0093] At process 804, haptic feedback is determined. In some examples, haptic feedback can be determined according to Equation 2 as a force proportional to the difference between the difference at the implemented imaging device reference point and the target difference at the imaging device reference point determined based on the difference at the display unit reference point.

[0094] At process 806, the haptic feedback is converted into joint force and / or torque. In some examples, the haptic feedback can be converted into joint force and / or torque of the joint of the repositionable structure on which the display unit 310 is mounted via inverse kinematics. In other examples, the haptic feedback can be converted into any technically feasible form (one or more), such as vibrations from a handheld device and / or a head-mounted device.

[0095] At process 808, joint force and / or torque (or other (one or more) forms of tactile feedback) is applied to the repositionable structure on which the display unit 310 is mounted. In some examples, one or more commands may be generated and sent to a controller for the joint in the repositionable structure on which the display unit 310 is mounted in order to provide joint force and / or torque (or other (one or more) forms of tactile feedback).

[0096] return Figure 6 When the imaging device 320 is actuated to or toward a new position through process 618 or 620, the next movement cycle begins by returning to process 604.

[0097] The disclosed embodiments control a repositionable imaging device by using a display unit reference point offset from the display unit when the display unit moves. The disclosed embodiments convert the position of the display unit reference point into a target position for the imaging device reference point. The imaging device reference point is offset from the imaging device. The disclosed embodiments command the repositioning of the imaging device based on the target position of the imaging device reference point. In the context of a remote operating system, the offset of the display unit reference point can be a certain distance in front of the display unit in a direction away from the operator, and can be within the nominal working distance between the display unit and the operator's hand. Also in the context of a remote operating system, the offset of the imaging device reference point can be a second distance in front of the imaging device, which can be within the nominal working distance between the imaging device and one or more instruments being remotely operated. When ergonomic adjustments occur, the baseline reference point is updated based on the position and orientation of the display unit when the ergonomic adjustment is completed; the display unit reference point is calculated as the difference from the baseline reference point. Furthermore, in some cases, the correction is set equal to the difference of the imaging device reference point, and a ratchet technique is executed to reduce the correction. When manual repositioning of the imaging device occurs, the baseline reference point of the imaging device is updated. Furthermore, in some cases, correction is determined based on the difference between the display unit reference point and the baseline reference point, multiplied by -1, and a ratchet technique is executed to reduce the correction. Additionally, in some cases, when the imaging device reference point cannot follow the display unit reference point, haptic feedback can be provided based on the difference between the realized difference between the imaging device reference point and the target difference between the imaging device reference point determined based on the difference between the display unit reference point; for example, the haptic feedback can be proportional to such a difference.

[0098] Advantageously, the disclosed embodiments allow the movement of the repositionable imaging device to resemble the movement of the display unit that commands the movement of the imaging device. This provides more intuitive control for the operator moving the display unit, even when the display unit and imaging device have different degrees of freedom and / or different ranges of motion. Furthermore, the disclosed embodiments can reduce or avoid unintended movements of the imaging device or display unit, or discontinuities in control of the imaging device, when the operator makes ergonomic adjustments to the display unit or manually repositions the imaging device.

[0099] Figures 6-8 One or more of the processes in the method shown can be implemented, in part or in whole, in the form of executable code stored on a non-transitory tangible machine-readable medium. When run by one or more processors (e.g., processor 150 in control system 140), the executable code can cause one or more processors to execute the processes of method 600 and / or Figure 6 , Figure 7 and / or Figure 8One or more processes are included in any of the accompanying drawings. In some embodiments, the processes may be executed by one or more modules, such as control module 170 in control system 140. In some embodiments, method 600 may include additional processes not shown. Processes that may include method 600 and / or Figure 6 , Figure 7 and / or Figure 8 Some common forms of machine-readable media for the process are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tapes, any other physical media with perforated patterns, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or tapes and / or any other media suitable for reading by a processor or computer.

[0100] While illustrative embodiments have been shown and described, extensive modifications, alterations, and substitutions are contemplated in the foregoing disclosure, and in some cases, certain features of the embodiments may be used without using other features. Many variations, alternatives, and modifications will be apparent to those skilled in the art. Therefore, the scope of the invention should be limited only by the appended claims, and it is appropriate that the claims be interpreted broadly and in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A computer-aided device, comprising: A first repositionable structure configured to support an imaging device; as well as A control system connected to the first repositionable structure, wherein the control system is configured to: Determine the position of the second reference point offset from the display unit; The position of the second reference point is converted into a target position offset from the first reference point of the imaging device; A movement command is determined for the first repositionable structure, which moves the imaging device such that the first reference point moves toward the target position; as well as The movement command causes actuation of the first repositionable structure.

2. The computer-aided device according to claim 1, wherein the second reference point is offset from the display unit in front of the display unit in a direction away from the operator, and the first reference point is offset from the imaging device in front of the imaging device in a direction of the field of view of the imaging device.

3. The computer-aided device according to claim 1, wherein the control system is further configured to determine the offset of the first reference point based on the zoom level of the imaging device.

4. The computer-aided device of claim 1, wherein the control system is further configured to determine the offset of the first reference point based on digital zoom or panning applied to the image displayed by the display unit.

5. The computer-aided device of claim 1, further comprising a second repositionable structure configured to support the display unit.

6. The computer-aided device of claim 5, wherein the position of the second reference point is determined based on the positive kinematics of the second repositionable structure.

7. The computer-aided device according to claim 5, wherein the imaging device and the first repositionable structure together have a first plurality of degrees of freedom, wherein the display unit and the second repositionable structure together have a second plurality of degrees of freedom different from the first plurality of degrees of freedom.

8. The computer-aided device according to claim 7, wherein the first plurality of degrees of freedom is different in number from the second plurality of degrees of freedom, or wherein the first plurality of degrees of freedom includes one or more degrees of freedom that do not match the second plurality of degrees of freedom.

9. The computer-aided device according to claim 1, wherein the display unit comprises a device selected from the group consisting of a handheld device and a head-mounted device.

10. The computer-aided apparatus of claim 1, wherein, in order to convert the position of the second reference point into the target position of the first reference point, the control system is configured to: Determine a first difference between the second reference point and the baseline reference point of the display unit; Determine a second difference in the first reference point based on the first difference; and The target position of the first reference point is determined by the second difference and the baseline reference point of the imaging device.

11. The computer-aided device of claim 10, wherein the control system is configured to: The baseline reference point of the display unit is determined based on the position of the second reference point when entering the first mode, wherein in the first mode, movement of the display unit is used to control movement of the imaging device; and The baseline reference point of the imaging device is determined based on the configuration of the imaging device when entering the first mode.

12. The computer-aided apparatus of claim 10, wherein, in order to determine the second difference based on the first difference, the control system is configured to scale the first difference according to a scaling factor.

13. The computer-aided device of claim 12, wherein the scaling factor is determined based on one or more parameters selected from the group consisting of: operator preference, type of display unit, type of imaging device, and program being executed.

14. The computer-aided apparatus of claim 12, wherein, in order to convert the position of the second reference point into the target position of the first reference point, the control system is further configured to: add correction to the second difference.

15. The computer-aided device according to any one of claims 1-14, wherein the control system is further configured to respond to recognizing a manual adjustment to the display unit: The baseline reference point of the display unit is set to the position of the second reference point when the manual adjustment of the display unit is completed; A first correction is set based on the difference between the first reference point and the baseline reference point of the imaging device; and Reduce the first correction over one or more moving cycles.

16. The computer-aided apparatus of claim 15, wherein, in order to reduce the first correction for the first movement cycle, the control system is configured to: A second correction for the first movement cycle is determined based on the difference between the first reference point and the baseline reference point of the imaging device for the first movement cycle, and the difference between the first reference point and the baseline reference point of the display unit from the previous movement cycle; and When the amplitude of the second correction is not greater than the amplitude of the first correction, the first correction is set based on the first correction, the second correction, and the ratchet factor.

17. The computer-aided device according to any one of claims 1-14, wherein the control system is further configured to respond to the recognition of manual adjustment to the imaging device: The baseline reference point of the imaging device is set to the position of the first reference point when the manual adjustment of the imaging device is completed; A first correction is set based on the difference between the second reference point and the baseline reference point of the display unit; and Reduce the first correction over one or more moving cycles.

18. The computer-aided apparatus of claim 17, wherein, in order to reduce the first correction for the first movement cycle, the control system is configured to: A second correction for the first movement cycle is determined based on the difference between the second reference point and the baseline reference point of the imaging device for the first movement cycle, and the difference between the second reference point and the baseline reference point of the display unit from the previous movement cycle; and When the amplitude of the second correction is not greater than the amplitude of the first correction, the second correction is set based on the first correction, the second correction, and the ratchet factor.

19. The computer-aided device according to any one of claims 1-14, wherein the control system is further configured to, in response to determining that the first repositionable structure is unable to carry out the movement command: Haptic feedback is determined based on the difference between the achievable position of the first reference point and the target position of the first reference point; Based on the tactile feedback, the joint force or torque of the connector in the second repositionable structure configured to support the display unit is determined; and Actuation of the joint of the second repositionable structure is caused by a determined joint force or torque.

20. The computer-aided device according to any one of claims 1-14, wherein the position of the second reference point is 30 cm in front of the lens included in the display unit, or wherein the position of the first reference point is 10 cm in front of the distal end of the imaging device.

21. The computer-aided device according to any one of claims 1-14, wherein the computer-aided device is a remotely operated medical device.

22. A method of operating a computer-aided device including a repositionable structure and one or more processors, said one or more processors being communicatively coupled to said repositionable structure, wherein the method is performed on a simulated patient or model, said method comprising: The position of the first reference point offset from the display unit is determined by one or more processors; The position of the first reference point is converted into the target position of the second reference point offset from the imaging device by one or more processors; The repositionable structure is determined by one or more processors, the movement command moving the imaging device such that the first reference point moves toward the target position, wherein the repositionable structure is configured to support the imaging device; as well as The repositionable structure is actuated by one or more processors based on the movement command.

23. The method of claim 22, wherein the second reference point is offset from the display unit in front of the display unit in a direction away from the operator, and the first reference point is offset from the imaging device in front of the imaging device in a direction of the field of view of the imaging device.

24. The method of claim 22, further comprising: The offset of the first reference point is determined based on the zoom level of the imaging device.

25. The method of claim 22, further comprising: The offset of the first reference point is determined based on digital zoom or panning applied to the image displayed by the display unit.

26. The method of claim 22, further comprising: The position of the second reference point is determined based on the positive kinematics of the second repositionable structure configured to support the display unit.

27. The method of claim 22, wherein the imaging device and the repositionable structure together have a first plurality of degrees of freedom, and wherein the display unit and the second repositionable structure supporting the display unit together have a second plurality of degrees of freedom different from the first plurality of degrees of freedom.

28. The method of claim 27, wherein the first plurality of degrees of freedom differs in number from the second plurality of degrees of freedom; or wherein the first plurality of degrees of freedom includes one or more degrees of freedom that do not match the second plurality of degrees of freedom.

29. The method of claim 22, wherein converting the position of the second reference point into the target position of the first reference point comprises: Determine a first difference between the second reference point and the baseline reference point of the display unit; A second difference is determined based on the first difference; as well as The target position of the first reference point is determined by the second difference and the baseline reference point of the imaging device.

30. The method of claim 29, further comprising: The baseline reference point of the display unit is determined based on the position of the second reference point when entering the first mode, wherein in the first mode, the movement of the display unit is used to control the movement of the imaging device; as well as The baseline reference point of the imaging device is determined based on the configuration of the imaging device when entering the first mode.

31. The method of claim 29, wherein determining the second difference based on the first difference comprises scaling the first difference according to a scaling factor.

32. The method of claim 31, further comprising: The scaling factor is determined based on one or more parameters selected from the group consisting of: operator preference, the type of the display unit, the type of the imaging device, and the program being executed.

33. The method of claim 31, wherein converting the position of the second reference point into the target position of the first reference point further comprises: Add correction to the second difference.

34. The method according to any one of claims 22-33, further comprising, in response to recognizing a manual adjustment to the display unit: The baseline reference point of the display unit is set to the position of the second reference point when the manual adjustment of the display unit is completed; A first correction is set based on the difference between the first reference point and the baseline reference point of the imaging device; and Reduce the first correction over one or more moving cycles.

35. The method of claim 34, wherein reducing the first correction for the first movement cycle comprises: A second correction for the first movement cycle is determined based on the difference between the first reference point and the baseline reference point of the imaging device for the first movement cycle, and the difference between the first reference point and the baseline reference point of the display unit from the previous movement cycle; and When the amplitude of the second correction is not greater than the amplitude of the first correction, the first correction is set based on the first correction, the second correction, and the ratchet factor.

36. The method according to any one of claims 22-33, further comprising, in response to recognizing a manual adjustment to the imaging device: The baseline reference point of the imaging device is set to the position of the first reference point when the manual adjustment of the imaging device is completed; A first correction is set based on the difference between the second reference point and the baseline reference point of the display unit; and Reduce the first correction over one or more moving cycles.

37. The method of claim 36, wherein reducing the first correction for the first movement cycle comprises: A second correction for the first movement cycle is determined based on the difference between the second reference point and the baseline reference point of the imaging device for the first movement cycle, and the difference between the second reference point and the baseline reference point of the display unit from the previous movement cycle; and When the amplitude of the second correction is not greater than the amplitude of the first correction, the second correction is set based on the first correction, the second correction, and the ratchet factor.

38. The method according to any one of claims 22-33, further comprising, in response to determining that the repositionable structure is unable to fulfill the movement command: Haptic feedback is determined based on the difference between the achievable position of the first reference point and the target position of the first reference point; Based on the tactile feedback, determine the joint force or torque of the connector in another repositionable structure configured to support the display unit; and Actuation of the joint of the other repositionable structure is caused by a determined joint force or torque.

39. The method according to any one of claims 22-33, wherein the position of the second reference point is 30 cm in front of the lens included in the display unit, or wherein the position of the first reference point is 10 cm in front of the distal end of the imaging device.

40. One or more non-transitory machine-readable media comprising a plurality of machine-readable instructions, which, when executed by one or more processors of a computer-aided device including a repositionable structure, are adapted to cause the one or more processors to perform a method comprising the following steps: The position of the first reference point offset from the display unit is determined by one or more processors; The position of the first reference point is converted into the target position of the second reference point offset from the imaging device by one or more processors; The repositionable structure is determined by one or more processors, the movement command moving the imaging device such that the first reference point moves toward the target position, wherein the repositionable structure is configured to support the imaging device; as well as The repositionable structure is actuated by one or more processors based on the movement command.

41. One or more non-transient machine-readable media according to claim 40, wherein the second reference point is offset from the display unit in front of the display unit in a direction away from the operator, and the first reference point is offset from the imaging device in front of the imaging device in a direction of the field of view of the imaging device.

42. The one or more non-transitory machine-readable media of claim 40, wherein the method further comprises: The offset of the first reference point is determined based on the zoom level of the imaging device.

43. The one or more non-transitory machine-readable media of claim 40, wherein the method further comprises: The offset of the first reference point is determined based on digital zoom or panning applied to the image displayed by the display unit.

44. The one or more non-transitory machine-readable media of claim 40, wherein the method further comprises: The position of the second reference point is determined based on the positive kinematics of the second repositionable structure configured to support the display unit.

45. One or more non-transient machine-readable media according to claim 40, wherein the imaging device and the repositionable structure together have a first plurality of degrees of freedom, and wherein the display unit and the second repositionable structure supporting the display unit together have a second plurality of degrees of freedom different from the first plurality of degrees of freedom.

46. ​​The one or more non-transient machine-readable media of claim 45, wherein the first plurality of degrees of freedom differs in number from the second plurality of degrees of freedom; or wherein the first plurality of degrees of freedom includes one or more degrees of freedom that do not match the second plurality of degrees of freedom.

47. One or more non-transient machine-readable media according to claim 40, wherein converting the position of the second reference point into the target position of the first reference point comprises: Determine a first difference between the second reference point and the baseline reference point of the display unit; A second difference is determined based on the first difference; as well as The target position of the first reference point is determined by the second difference and the baseline reference point of the imaging device.

48. The method further comprises: one or more non-transitory machine-readable media according to claim 47. The baseline reference point of the display unit is determined based on the position of the second reference point when entering the first mode, wherein in the first mode, the movement of the display unit is used to control the movement of the imaging device; as well as The baseline reference point of the imaging device is determined based on the configuration of the imaging device when entering the first mode.

49. One or more non-transitory machine-readable media according to claim 47, wherein determining the second difference based on the first difference comprises: The first difference is scaled according to a scaling factor.

50. The one or more non-transitory machine-readable media of claim 49, wherein the method further comprises: The scaling factor is determined based on one or more parameters selected from the group consisting of: operator preference, the type of the display unit, the type of the imaging device, and the program being executed.

51. The one or more non-transient machine-readable media of claim 47, wherein converting the position of the second reference point into the target position of the first reference point further comprises: Add correction to the second difference.

52. One or more non-transitory machine-readable media according to any one of claims 40-51, wherein the method further comprises, in response to recognizing a manual adjustment to the display unit: The baseline reference point of the display unit is set to the position of the second reference point when the manual adjustment of the display unit is completed; A first correction is set based on the difference between the first reference point and the baseline reference point of the imaging device; and Reduce the first correction over one or more moving cycles.

53. The one or more non-transient machine-readable media of claim 52, wherein reducing the first correction for the first movement cycle comprises: A second correction for the first movement cycle is determined based on the difference between the first reference point and the baseline reference point of the imaging device for the first movement cycle, and the difference between the first reference point and the baseline reference point of the display unit from the previous movement cycle; and When the amplitude of the second correction is not greater than the amplitude of the first correction, the first correction is set based on the first correction, the second correction, and the ratchet factor.

54. One or more non-transient machine-readable media according to any one of claims 40-51, wherein the method further comprises, in response to identifying a manual adjustment to the imaging device: The baseline reference point of the imaging device is set to the position of the first reference point when the manual adjustment of the imaging device is completed; A first correction is set based on the difference between the second reference point and the baseline reference point of the display unit; and Reduce the first correction in one or more moving cycles.

55. One or more non-transient machine-readable media according to claim 54, wherein reducing the first correction for a first movement cycle comprises: A second correction for the first movement cycle is determined based on the difference between the second reference point and the baseline reference point of the imaging device for the first movement cycle, and the difference between the second reference point and the baseline reference point of the display unit from the previous movement cycle; and When the amplitude of the second correction is not greater than the amplitude of the first correction, the second correction is set based on the first correction, the second correction, and the ratchet factor.

56. One or more non-transitory machine-readable media according to any one of claims 40-51, wherein the method further comprises, in response to determining that the repositionable structure is incapable of fulfilling the movement command: Haptic feedback is determined based on the difference between the achievable position of the first reference point and the target position of the first reference point; Based on the tactile feedback, determine the joint force or torque of the connector in another repositionable structure configured to support the display unit; and Actuation of the joint of the other repositionable structure is caused by a determined joint force or torque.

57. One or more non-transient machine-readable media according to any one of claims 40-51, wherein the position of the second reference point is 30 cm in front of the lens included in the display unit, or wherein the position of the first reference point is 10 cm in front of the distal end of the imaging device.

Images