Automatic determination of deployment settings for computer-aided systems

Abstract

A computer-assisted device includes a repositionable structure and a processing system. The processing system is configured to perform the steps of: determining a position of the repositionable structure relative to a patient on the operative support; determining an imaging device orientation of the imaging device relative to the patient; determining a deployment of the repositionable structure based on the imaging device orientation and the position of the repositionable structure relative to the patient; and causing display of one or more instructions for configuring the repositionable structure.

Description

[0001] Related Applications This application claims the benefit of U.S. Provisional Application No. 63 / 494,919, filed April 7, 2023, entitled “Automated Determination of Deployment Settings for a Computer-assisted System,” which is incorporated by reference herein. TECHNICAL FIELD

[0002] The present disclosure relates generally to the operation of computer-assisted systems having repositionable structures, such as articulated arms, and more particularly to the automated determination of deployment settings for such computer-assisted systems. BACKGROUND

[0003] The use of computer-assisted electronic systems is increasing in frequency. This is particularly true in industrial, entertainment, educational, and other environments. As a medical example, today’s medical facilities have a large number of electronic systems in operating rooms, interventional suites, intensive care units, emergency rooms, and the like. Many of these electronic systems can be capable of autonomous or semi-autonomous motion. It is also known for personnel to use one or more input devices located at a user control system to control the motion and / or operation of the electronic systems. As a specific example, minimally invasive robotic tele-surgical systems allow surgeons to operate on patients from a bed-side or remote location. Tele-surgery generally refers to surgery performed using surgical systems in which the surgeon uses some form of remote control, such as a servomechanism, to manipulate the movement of surgical instruments rather than holding and moving them by hand.

[0004] When using a computer-assisted system to perform a task at a work site (e.g., a patient’s internal anatomy in a medical example), the computer-assisted system needs to be positioned within a sterile field and oriented toward the patient so that the repositionable structures of the computer-assisted system can have an appropriate range of motion to control one or more instruments to access the work site. Ideally, when the instruments are used to access a target anatomical structure of the patient that is configured to control one or more instruments, the computer-assisted system is positioned so that some or all of the joints of each repositionable structure are at or near the center of their respective ranges of motion, the patient is within the work site, and the work site is on the sterile side of the computer-assisted system. Typically, the surgeon who will be operating the computer-assisted system can best determine which target anatomical structure within the work site is best suited for a particular procedure.

[0005] Accordingly, improved techniques for implementing deployment of a computer-assisted system are desirable. SUMMARY

[0006] In some embodiments, a computer-assisted system includes a repositionable structure and a processing system. In embodiments, the processing system is configured to perform the steps of: determining a position of the repositionable structure relative to a patient on an operating support; determining an imaging device orientation of an imaging device relative to the patient; determining a deployment of the repositionable structure based on the imaging device orientation and the position of the repositionable structure relative to the patient; and causing display of one or more instructions for configuring the repositionable structure.

[0007] In some embodiments, a method includes: determining a position of a repositionable structure of a computer-assisted system relative to a patient on an operating support; determining an imaging device orientation of an imaging device relative to the patient; determining a deployment of the repositionable structure based on the imaging device orientation and the position of the repositionable structure relative to the patient; and causing display of one or more instructions for configuring the repositionable structure.

[0008] In some embodiments, a non-transitory machine-readable medium includes a plurality of machine-readable instructions that, when executed by one or more processors associated with a computer-assisted device, are adapted to cause the one or more processors to perform the methods disclosed herein.

[0009] The above general description and the following detailed description are exemplary and explanatory in nature and are intended to provide a understanding of the present disclosure, not to limit the scope of the present disclosure. In this regard, additional aspects, features, and advantages of the present disclosure will be apparent from the following detailed description, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a simplified diagram of an example of a computer-assisted system in accordance with some embodiments.

[0011] Figure 2 is a simplified diagram of a follower device with a sensor system in accordance with some embodiments.

[0012] Figure 3 is a simplified diagram of a follower device with a display system in accordance with some embodiments.

[0013] Figure 4 is a simplified diagram of a method for deploying a computer-assisted system in accordance with some embodiments.

[0014] Figure 5 is a simplified diagram of a patient positioned on an operating support in accordance with some embodiments.

[0015] Figure 6 is a simplified diagram of a target anatomical structure and associated working space in accordance with some embodiments.

[0016] Figure 7is a simplified diagram of various positions proximate to an operating support at which a slave device of a computer-assisted system can be positioned, according to some embodiments.

[0017] Figure 8 is a simplified diagram of a manually controlled imaging device being inserted into a work site, according to some embodiments.

[0018] Figure 9 is a simplified diagram of a manually controlled imaging device being directed toward a target anatomical structure, according to some embodiments.

[0019] Figures 10A-10D is a simplified diagram of various deployments selected by a method according to some embodiments. Figure 4

[0020] Figure 11 is a simplified diagram of a method for selecting a deployment of a computer-assisted system, according to some embodiments.

[0021] Figure 12 is a simplified diagram of a method for selecting a deployment of a computer-assisted system, according to some embodiments.

[0022] In the drawings, elements having the same name or reference numeral have the same or similar function. DETAILED DESCRIPTION

[0023] This specification and the accompanying drawings and figures should not be construed as limiting the inventive aspects, embodiments, examples, or modules to the examples set forth herein — the claims define the protected invention. Various mechanical, compositional, structural, electrical, and operational changes can be made without departing from the spirit and scope of this specification and the claims. In some instances, details have not been shown or described in order not to obscure the application in unnecessary details. The same numbers refer to the same or similar elements throughout the drawings.

[0024] In this specification, specific details of some embodiments consistent with the present disclosure are set forth. For the purpose of providing a thorough understanding of embodiments, numerous specific details are set forth. It will be apparent, however, to one skilled in the art, that some embodiments can be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative only and not limiting. Other elements, although not specifically described herein, can be implemented by persons skilled in the art with the benefit of their knowledge in the field of the disclosure. Furthermore, to avoid unnecessary repetition, one or more features associated with one embodiment can be incorporated into other embodiments, unless specifically described otherwise, or unless the one or more features would make the embodiment unworkable.

[0025] ​Moreover, the terminology used in the description herein is not intended to be limiting of the application. For example, spatially relative terms— such as "beneath", "below", "lower", "above", "upper", "proximal", "distal", and the like— can be used herein for ease of describing one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. Spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientations depicted in the figures. For example, if a device in the figures is turned over, elements described as "below" or "beneath" other elements or features would then be oriented "above" or "over" the other elements or features. Thus, the exemplary term "below" can encompass both an orientation of above and below. The device can be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. Likewise, descriptions of movement along and around various axes include various special

[0026] Elements described in relation to one embodiment, example, or module can be included in other embodiments, examples, or modules that do not specifically show or describe them, as long as is practicable. For example, if an element is described in relation to one embodiment without being described in relation to a second embodiment, the element can still be required to be included in the second embodiment. Thus, to avoid unnecessary repetition in the description that follows, one or more elements shown and described in association with one embodiment, example, or application can be incorporated into other embodiments, examples, or aspects, unless specifically described otherwise, unless the one or more elements would make the embodiment or example inoperable, or unless the provision of two or more of the elements conflicts with a function.

[0027] In certain circumstances, well-known methods, procedures, components, and circuits have not been described in detail since it can be apparent to persons skilled in the art how to implement them without additional information.

[0028] The present disclosure describes various elements (such as systems and devices, and portions of systems and devices) by way of example in three-dimensional space. In these examples, the term "position" refers to the location of an element or portion of an element in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). Also in these examples, the term "orientation" refers to the rotational placement of an element or portion of an element (three rotational degrees of freedom, e.g., roll, pitch, and yaw). Other examples can encompass other dimensional spaces, such as two-dimensional spaces. As used herein, the term "pose" refers to the position, orientation, or combination of position and orientation of an element or portion of an element. As used herein, and with respect to an element or portion of an element of a structure or assembly (e.g., a computer-assisted system or repositionable structure, etc.), the term "proximal" in the kinematic series refers to a direction toward a base of the kinematic series, while the term "distal" refers to a direction away from the base along the kinematic series.

[0029] Aspects of the present disclosure are described with reference to electronic systems, computer-assisted devices, and robotic devices, which can include teleoperated, remotely controlled, autonomous, semi-autonomous, manually manipulated, etc. systems and devices. Example computer-assisted systems include those that include a robot or robotic device. Moreover, aspects of the present disclosure are described in terms of embodiments that use a medical system, such as the da Vinci® surgical system commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. However, those of ordinary skill will understand that the inventive aspects disclosed herein can be embodied and implemented in various ways, including robotic and non-robotic embodiments, if applicable. The embodiments described with respect to the da Vinci® surgical system are merely exemplary and should not be viewed as limiting the scope of the inventive aspects disclosed herein. For example, the techniques described with reference to surgical instruments and surgical methods can be used in other contexts. Thus, the instruments, systems, and methods described herein can be used for humans, animals, portions of human or animal anatomy, industrial systems, general purpose robots, or teleoperated systems. As further examples, the instruments, systems, and methods described herein can be used for non-medical purposes, including industrial uses, general purpose robotic uses, sensing or manipulating non-tissue workpieces, cosmetic improvements, imaging of human or animal anatomy, collecting data from human or animal anatomy, setting up or tearing down systems, training medical or non-medical personnel, etc. Additional example applications include use for procedures on tissue removed from human or animal anatomy (with or without return to the human or animal anatomy), and use for procedures on human or animal cadavers. Moreover, these techniques can also be used for medical treatment or diagnostic procedures that include or exclude surgical aspects.

[0030] Example computer-assisted systems Figure 1is a simplified diagram of an example computer-assisted system 100 according to some embodiments. In some examples, the computer-assisted system 100 is a teleoperational system. In medical examples, the computer-assisted system 100 can be a teleoperational medical system, such as a surgical system. As shown, the computer-assisted system 100 includes a slave device 104 that can be teleoperated by control from one or more master devices (also referred to as "master input devices" when designed to accept external input), described in more detail below. A system including a master device and a slave device is referred to as a master-slave system, and is sometimes also referred to as a master-slave system. Figure 1 An input system including a workstation 102 (e.g., a console) is also shown in FIG. 1, and in various embodiments the input system can be in any suitable form and can or can not include a workstation 102.

[0031] In Figure 1 In the example of FIG. 1, the workstation 102 includes one or more master input devices 106 designed to be contacted and manipulated by an operator 108. For example, the workstation 102 can include one or more master input devices 106 for use by the hands, head, or some other body part of the operator 108. In this example, the master input devices 106 are supported by the workstation 102 and can be mechanically grounded. In some embodiments, an ergonomic support 110 (e.g., an arm rest) can be provided on which the operator 108 can rest his or her forearms. In some examples, the operator 108 can command the slave device 104 to perform tasks at a work site in the vicinity of the slave device 104 during a procedure by using the master input devices 106.

[0032] A display unit 112 is also included in the workstation 102. The display unit 112 can display images for viewing by the operator 108. The display unit 112 can be movable in various degrees of freedom to accommodate viewing positions of the operator 108 and / or optionally provide control functions as another master input device. In examples of the computer-assisted system 100, the displayed images can depict a work site at which the operator 108 performs various tasks by manipulating the master input devices 106 and / or the display unit 112. In some examples, the workstation 102 can receive images displayed by the display unit 112 from one or more imaging devices disposed at the work site. In other examples, the images displayed by the display unit 112 can be generated by the display unit 112 (or by a different connected device or system), such as virtual representations of tools, work site, or user interface components.

[0033] When using workstation 102, operator 108 can sit in a chair or other support in front of workstation 102, position his or her eyes in front of display unit 112, manipulate guidance input device 106, and rest his or her forearms on ergonomic support 110 as needed. In some embodiments, operator 108 can stand at the workstation or assume other postures, and the orientation (height, depth, etc.) of display unit 112 and guidance input device 106 can be adjusted to accommodate operator 108.

[0034] In some embodiments, the one or more guidance input devices 106 can be ungrounded (ungrounded guidance input devices have no kinematic ground, e.g., the guidance input device is held by the hand of operator 108 without an additional physical support). Such ungrounded guidance input devices can be used in conjunction with display unit 112. In some embodiments, operator 108 can use display unit 112 positioned near the work site so that operator 108 manually operates an instrument at the work site while viewing images displayed by display unit 112, e.g., a laparoscopic instrument in a surgical example.

[0035] Computer-assisted system 100 can also include a slave device 104, which can be commanded by workstation 102. In medical examples, slave device 104 can be located near an operating table (e.g., a table, bed, or other support) on which a patient can be positioned. In some medical examples, a work site is provided on the operating table, e.g., on or in a patient, a simulated patient, or a model, etc. (not shown). The illustrated slave device 104 includes a plurality of manipulator arms 120, each configured to be coupled to an instrument assembly 122. Instrument assembly 122 can include, for example, an instrument 126. As shown, each instrument assembly 122 is mounted to a distal portion of a respective manipulator arm 120. The distal portion of each manipulator arm 120 also includes a trocar mount 124 configured to have a trocar (not shown) mounted thereto. When a trocar is mounted to the trocar mount, a shaft of instrument 126 passes through the trocar and into a work site, e.g., a surgical site during a surgical procedure. The distal portion of each manipulator arm 120 also includes a trocar mount 124 configured to have a trocar (not shown) mounted thereto. When a trocar is mounted to the trocar mount, a shaft of instrument 126 passes through the trocar and into a work site, e.g., a surgical site during a surgical procedure. A force transmission mechanism 130 of instrument assembly 122 can be connected to an actuation interface assembly 128 of manipulator arm 120, which includes drivers and / or other mechanisms controllable from workstation 102 to transmit forces to force transmission mechanism 130 to actuate instrument 126.

[0036] In various embodiments, one or more of the instruments 126 can include an imaging device (e.g., optical camera, hyperspectral camera, ultrasonic sensor, etc.) for capturing images. For example, one or more of the instruments 126 can be an endoscope assembly that includes an imaging device that can provide captured images of a portion of the work site for display via the display unit 112.

[0037] In some embodiments, the manipulator arm 120 and / or the instrument assembly 122 can be controlled to move and articulate the instrument 126 in response to manipulation of the lead input device 106 by the operator 108, and to "follow" the lead input device 106 in this manner by teleoperation. This enables the operator 108 to perform tasks at the work site using the manipulator arm 120 and / or the instrument assembly 122. The manipulator arm 120 is an example of a repositionable structure (e.g., a slave device 104) that a computer-assisted device can include. In some embodiments, the repositionable structure of a computer-assisted device can include a plurality of links that are rigid members and joints that are movable components that can be actuated to cause relative motion between adjacent links. For a surgical procedure example, the operator 108 can direct the slave manipulator arm 120 to move the instrument 126 to perform a surgical procedure at an internal surgical site through a minimally invasive hole or natural orifice.

[0038] As shown, the control system 140 is provided external to the workstation 102 and in communication with the workstation 102. In other embodiments, the control system 140 can be provided in the workstation 102 or the slave device 104. As the operator 108 moves the lead input device 106, sensed spatial information including sensed position and / or orientation information is provided to the control system 140 based on movement of the lead input device 106. The control system 140 can determine control signals or provide control signals to the slave device 104 to control movement of the manipulator arm 120, the instrument assembly 122, and / or the instrument 126 based on the received information and operator input. In one embodiment, the control system 140 supports one or more wired communication protocols (e.g., Ethernet, USB, etc.) and / or one or more wireless communication protocols (e.g., Bluetooth, IrDA, HomeRF, IEEE 1102.11, DECT, wireless telemetry, etc.).

[0039] The control system 140 can be implemented on one or more computing systems. The one or more computing systems can be used to control the slave device 104. In addition, the one or more computing systems can be used to control components of the workstation 102, such as movement of the display unit 112.

[0040] As shown, the control system 140 includes a processing system 150 and a memory 160 storing a control module 170. In some embodiments, the control system 140 can include one or more processors, non-persistent storage (e.g., volatile memory, such as random access memory (RAM)), cache memory, persistent storage (e.g., a hard disk, an optical drive like a compact disk (CD) drive or digital versatile disk (DVD) drive, flash memory, a floppy disk, a flexible disk, a tape, any other magnetic medium, any other optical medium, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a FLASH-EPROM, any other memory chip or cartridge, punched cards, paper tape, any other physical medium with patterns of holes, and / or the like), a communication interface (e.g., a Bluetooth interface, an infrared interface, a network interface, an optical interface, and / or the like), and many others. The non-persistent storage and the persistent storage are examples of non-transitory, tangible machine-readable media that can include executable code, which, when executed by the one or more processors (e.g., the processing system 150), can cause the one or more processors to perform one or more of the techniques disclosed herein, including the processes of the method 400, the method 1100, and / or the method 1200 described below. Further, in some embodiments, the functionality of the control module 170 can be implemented in any technically feasible software and / or hardware.

[0041] Each of the one or more processors of the processing system 150 can be an integrated circuit for processing instructions. For example, the one or more processors can be one or more cores or micro-cores of a processor, a central processing unit (CPU), a microprocessor, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), a tensor processing unit (TPU), and / or the like. The control system 140 can also include one or more input devices, such as a touchscreen, a keyboard, a mouse, a microphone, a touchpad, an electronic pen, or any other type of input device.

[0042] The communication interface of the control system 140 can include an integrated circuit for connecting a computing system to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) like the Internet, a mobile network, or any other type of network) and / or another device, such as another computing system.

[0043] In addition, the control system 140 can include one or more output devices, such as a display device (e.g., a liquid crystal display (LCD), a plasma display, a touchscreen, an organic LED display (OLED), a projector, or other display device), a printer, a speaker, an external storage, or any other output device. One or more of the output devices can be the same as or different from the input devices. There are many different kinds of computing systems, and the above-mentioned input and output devices can take other forms.

[0044] In some embodiments, the control system 140 can be connected to or be part of a network. The network can include multiple nodes. The control system 140 can be implemented on one node or a group of nodes. For example, the control system 140 can be implemented on one node of a distributed system that is connected to other nodes. As another example, the control system 140 can be implemented on a distributed computing system with multiple nodes, where different functions and / or components of the control system 140 can be located on different nodes within the distributed computing system. In addition, one or more elements of the control system 140 described above can be located at a remote location and connected to other elements through a network.

[0045] Some embodiments can include one or more components of a teleoperated medical system (such as the da Vinci® surgical system commercialized by Intuitive Surgical, Inc. of Sunnyvale, California). Embodiments of the da Vinci® surgical system are merely examples and should not be considered limiting the scope of the features disclosed herein. For example, different types of teleoperated systems with slave devices at the work site, as well as non-teleoperated systems, can utilize the features described herein.

[0046] Figure 2 is a simplified diagram of a slave device with a sensor system according to some embodiments. As shown, imaging devices 202 (imaging devices 202-1 through 202-4) are attached to portions of the slave device 104. Although described herein with respect to imaging devices as a reference example, in some embodiments, the sensor system can include any technically feasible sensor, such as monoscopic and stereoscopic optical systems, ultrasonic systems, depth cameras (e.g., cameras using time-of-flight sensors), LIDAR (light detection and ranging) sensors, etc., mounted on the computer-assisted system and / or elsewhere. For example, one or more sensors can be mounted on the base, the orientation platform 204, and / or one or more manipulator arms 120 of the slave device 104. As another example, one or more sensors can be worn by the operator or mounted to a wall, ceiling, floor, or other equipment, such as a table or cart.

[0047] Illustratively, imaging device 202-1 is attached to the orientation platform 204 of the slave device 104, imaging device 202-2 is attached to the manipulator arm 120-1 of the slave device 104, imaging device 202-3 is attached to the manipulator arm 120-4 of the slave device 104, and imaging device 202-4 is attached to the base 206 of the slave device 104. In embodiments where the slave device 104 is positioned proximate to a patient (e.g., as a patient-side cart), placing imaging devices 202 at strategic locations on the slave device 104 provides advantageous imaging viewpoints proximate to the patient and the area surrounding the work site where a surgical procedure will be performed on the patient.

[0048] like Figure 2 The placement of the imaging device 202 shown on the components of the follower device 104 is illustrative. In other embodiments, any suitable number of additional and / or alternative placements of imaging devices 202 and / or other sensors on the follower device 104, other components of the computer-assisted system 100, and / or other components (not shown) located near the follower device 104 may be used in the sensor system. The imaging device 202 and / or other sensors may be attached to the components of the follower device 104, other components of the computer-assisted system 100, and / or other components located near the follower device 104 in any suitable manner. Additional computer-assisted systems including sensor systems having sensors are described in International Application Publication No. WO 2021 / 097332, filed on November 13, 2020, entitled “Visibility Metrics in Multi-View Medical Activity Recognition Systems and Methods,” which is incorporated herein by reference.

[0049] Figure 3 is a simplified diagram of a follower device with a display system, according to some embodiments. As shown, the user control interface (helm) 304 of the follower device 104 includes display devices 302 (display devices 302-1 and 302-2). Illustratively, the user control interface 304 is attached to a repositionable structure of the follower device 104 on the side opposite the manipulator arm 120. Display device 302 is an example output device of the follower device 104. In some embodiments, the follower device 104 may include one or more output devices in any technologically feasible manner. For example, one or more of the display devices 302 may be a cathode ray tube (CRT), a liquid crystal display (LCD), a light emitting diode (LED), an organic light emitting diode (OLED), a quantum dot light emitting diode (QLED), a plasma display, a touch screen, a projector, and the like.

[0050] Illustratively, the user control interface 304 also includes a handle 306 that the operator can push or pull to reposition the follower device 104 within the environment. In some embodiments, the follower device 104 includes one or more actuators (e.g., one or more motors or servos) that drive wheels (not shown) of the follower device 104 based on input from the operator to assist the operator in repositioning the follower device 104. For example, the force or torque applied by the operator to the handle 306 can be used to determine the direction and speed of the one or more actuators. In some examples, the user control interface 304 can include one or more buttons or other input devices (e.g., a joystick) to provide directional commands for controlling the one or more actuators. In some embodiments, the repositioning of the follower device 104 can be semi-autonomous or fully autonomous. In some other embodiments, the follower device 104 does not include one or more actuators to assist the operator in repositioning the follower device 104.

[0051] like Figure 3 The placement of display device 302 on slave device 104 shown is illustrative. In other embodiments, any suitable number of additional and / or alternative placements of display device 302 on slave device 104, other components of computer-assisted system 100, and / or other components (not shown) located proximate to slave device 104 may be used. For example, one or more display devices may be attached to components of slave device 104, other components of computer-assisted system 100, and / or other components located proximate to slave device 104 in any suitable manner. As a further example, one or more display devices may be included in a handheld device or a head-mounted device.

[0052] Selecting the deployment of a computer-assisted system As mentioned above, when using a follow-up device (such as Figures 1-3When a computer-assisted system (e.g., the robotic device 104 described above) performs a task at a work site (e.g., internal patient anatomy in the medical example), the robotic device needs to be positioned within a sterile field and oriented toward the patient so that the repositionable structure of the robotic device can have the proper range of motion to control one or more instruments to access the work site. In a typical arrangement, a non-surgeon operator controls the position and orientation of the computer-assisted system from a non-sterile side of the computer-assisted system that is separated from the sterile field of the work site and patient by one or more sterile drapes. The non-surgeon operator typically assists the task through a user interface of the computer-assisted system that presents a number of options for selecting a preselected deployment of the computer-assisted system, where each preselected deployment is associated with one or more target anatomical regions. Once the non-surgeon operator selects the desired deployment, the deployment is used to guide the non-surgeon operator in positioning and orienting the computer-assisted system relative to the patient. However, the responsibility for determining the target anatomical structure rests with the surgeon operator who must communicate sufficient information about the target anatomical structure so that the non-surgeon operator can select the corresponding deployment. Importantly, selecting an incorrect deployment can result in a delay in the procedure or issues with the range of motion of the repositionable structure during the procedure.

[0053] In some cases, once the non-surgeon operator selects the desired deployment, the deployment is used to guide the non-surgeon operator in positioning and orienting the computer-assisted system relative to the patient. However, the responsibility for determining the target anatomical structure rests with the surgeon operator who must communicate sufficient information about the target anatomical structure so that the non-surgeon operator can select the corresponding deployment. To effectively do so, there needs to be sufficient communication between the surgeon operator on the sterile side of the computer-assisted system and the non-surgeon operator on the non-sterile side of the computer-assisted system. The effectiveness of this communication depends on the level of experience of the non-surgeon operator in understanding the association between the deployments and the patient anatomical structure, as well as the indication from the surgeon of the target anatomical structure for the procedure. This can result in the selection of an incorrect deployment, which can result in a delay in the procedure or issues with the range of motion of the repositionable structure during the procedure. To prevent the selection of an incorrect deployment, it is helpful to automatically select the appropriate deployment.

[0054] Consistent with some embodiments, an improved method involves a computer- assisted system that uses sensor data to determine an imaging device orientation relative to a patient on an operating support, and then uses the imaging device orientation to automatically select a deployment for guiding a non-surgeon operator. In more detail, by positioning a repositionable structure of the computer-assisted system proximate to a patient, a surgeon inserts an imaging device (e.g., an endoscope) that is not mounted to the computer-assisted system and is manually controlled by the surgeon operator into the patient. The surgeon operator then positions and orients the imaging device so that the imaging device captures images of a target anatomical structure. Once the surgeon operator positions and orients the imaging device toward the target anatomical structure, the computer-assisted system begins a deployment selection process. During the deployment selection process, the computer-assisted system determines a position of the repositionable structure relative to the patient and an orientation of the imaging device. The computer-assisted system uses the position of the repositionable structure and the orientation of the imaging device to determine a deployment, e.g., a working direction, of the repositionable structure of the computer-assisted system. In some embodiments, the working direction of the repositionable structure is selected from a set of predetermined working directions. Moreover, in some embodiments, the working direction of the repositionable structure is selected based on a predetermined association between a particular working direction and a combination of the current position of the repositionable structure and the current orientation of the imaging device. This improved method reduces the likelihood of an incorrect deployment of the computer-assisted system.

[0055] According to various embodiments, a deployment of a computer-assisted system is automatically selected to guide a non-surgeon operator in correctly positioning and orienting a repositionable structure of the computer-assisted system. The deployment is selected based on a position of the repositionable structure relative to a patient and an orientation of an imaging device relative to the patient. The following describes examples of such embodiments. Figures 4-12

[0056] Figure 4 is a simplified diagram of a method 400 for deploying a computer-assisted system according to some embodiments. According to some embodiments, the method 400 can include one or more of processes 402-412, which can be implemented at least in part in the form of executable code stored on a non-transitory, tangible machine-readable medium, which when run on a processing system 150 in a control system 140 of a computer-assisted system 100 (e.g., Figure 1 as described in connection with FIG. 1), can cause the one or more processors to perform one or more of the processes 402-412. The following describes examples of such processes. Figure 4 ​In the described embodiment, method 400 is described with respect to medical applications and computer-assisted system 100. It will be apparent to one of ordinary skill in the art that method 4100 may be performed using any other suitable computer-assisted system. Furthermore, in other embodiments, method 400 may be used in other non-surgical applications where automatic selection of any other suitable computer-assisted system deployment may be beneficial.

[0057] At process 402, a patient is prepared for a particular surgical procedure in a sterile area of ​​an operating environment. For example, in some cases, the patient may be positioned on an operating table or other support that is within a sterile area that is separated from a non-sterile area by one or more drapes or other barriers. In process 402, the patient may be positioned on the operating support in a specified orientation or body position that is associated with the particular surgical procedure. Examples of possible specified orientations or body positions include supine (face and abdomen facing up), prone (face and abdomen facing down), right side decubitus (right side oriented down), and left side decubitus (left side oriented down). Figure 5 An example embodiment of a patient positioned on an operating support is described.

[0058] Figure 5 is a simplified diagram of a patient 502 positioned on an operating support 504 according to some embodiments. As shown, the patient 502 and the operating support 504 are located within a sterile area 506 of an operating environment. Figure 5 In the example shown, the patient 502 is positioned in a left lateral decubitus position, with the left side of the patient 502 oriented downward. Once the patient is positioned, a surgical working site 520 is identified, for example, by a surgeon operator (not shown) of the computer-assisted system 100. In some embodiments, the surgical working site 520 includes a target anatomical structure 522 and / or anatomical structures of the patient 502 to be manipulated by one or more instruments 126 (e.g., Figure 1 The following is combined with Figure 6 Example embodiments of a surgical workspace and target anatomy associated with a patient are described.

[0059] Figure 6 is a simplified diagram of a target anatomical structure 622 and an associated workspace 620 according to some embodiments. Figure 6In the illustrated embodiment, workspace 620 is a surgical workspace for a surgical procedure and includes a target anatomical structure 622. In some embodiments, target anatomical structure 622 corresponds to a portion or region of the anatomy of patient 602 where a midline 623 of workspace 620 intersects a distal edge 624 of workspace 620. As shown, distal edge 624 of workspace 620 is the edge of workspace 620 distal to ports 631-636, i.e., on the side of workspace 620 opposite ports 631-636. In some embodiments, target anatomical structure 622 includes a pathology or other area or organ to be accessed during a surgical procedure.

[0060] Typically, the workspace 620 includes the patient 602 to be manipulated by one or more instruments 126 (e.g., Figure 1 620 (e.g., to perform a surgical procedure). For example, the workspace 620 may correspond to or include a cavity created by insufflating gas into a region of the patient's anatomical structure 602 surrounding a target anatomical structure 622. During a surgical procedure, one or more instruments 126 may be positioned within such a cavity while observing the movement of the instruments 126 via one or more endoscopic cameras also positioned within the workspace 620 and / or the cavity. Typically, the one or more instruments 126 and endoscopic cameras are inserted into the cavity via one or more ports 631-636.

[0061] Each of the ports 631-636 provides access to a cavity associated with the workspace 620 and surrounding the internal anatomy of the patient 602. In some embodiments, each of the ports 631-636 provides such access to a single instrument 126, such as an imaging device (e.g., an endoscope) or a minimally invasive surgical instrument (e.g., a surgical stapler, suction irrigator, clamps, etc.). In some embodiments, one or more of the ports 631-636 provides access to two or more instruments 126. Figure 6 In the embodiment shown, the ports 631-636 are arranged along a line 611 perpendicular to the intended working direction 612 of the computer-assisted system 100. In other embodiments, the ports 631-636 may have Figure 6 Typically, a port for receiving an instrument 126 is not positioned between the target anatomical structure 622 and another port for receiving an instrument 126. In some embodiments, a working direction 612 (sometimes referred to as an "anatomical structure direction") corresponds to the general direction in which the axis of the instrument 126 is oriented when accessing the workspace 620 during a surgical procedure. In some embodiments, the working direction 612 is parallel to the midline 623 of the workspace 620 and is oriented toward the target anatomical structure 622.

[0062] Returning to Figure 4 In process 402, as part of preparing the patient for a particular surgical procedure, one or more ports (e.g., ports 631-636 in FIG. 6B) are placed appropriately for the particular procedure, e.g., an incision is made and a cannula or other conduit for one or more instruments is inserted by a surgeon operator or other sterile operator. As noted above, placement of ports proximate workspace 620 can depend on the location of the target anatomy associated with the particular surgical procedure within the workspace. Figure 6

[0063] In some embodiments, as part of process 402, one or more repositionable structures (e.g., manipulator arms 120) of computer-assisted system 100 are positioned relative to the patient. In some embodiments, the repositionable structures are positioned relative to the patient by positioning follower device 104 proximate an operating support on which the patient is positioned. Example embodiments of follower device 104 positioned proximate an operating support are described below in connection with FIGS. 7A-7C. Figure 7

[0064] Figure 7 are simplified diagrams of various positions proximate operating support 504 at which follower device 104 can be positioned in accordance with some embodiments. As shown, in some embodiments, follower device 104 can be positioned relative to operating support 504 at a head position 711. In some cases, positioning follower device 104 at head position 711 facilitates an ear, nose, and throat (head and neck) surgical procedure. Additionally or alternatively, in some embodiments, follower device 104 can be positioned relative to operating support 504 at a first side position 712 proximate a head end of operating support 504. First side position 712 can be on a patient left side or a patient right side of operating support 504 (first side position 712 is shown on the patient left side in Figure 7 In some cases, positioning follower device 104 at first side position 712 facilitates a chest and / or upper abdominal surgical procedure. Additionally or alternatively, in some embodiments, follower device 104 can be positioned relative to operating support 504 at a second side position 713 proximate a central region of operating support 504. Second side position 713 can be on a patient left side or a patient right side of operating support 504 (second side position 713 is shown on the patient right side in Figure 7 ​​A second side position 713 is shown in FIG. 7B on the patient’s right side). In some cases, positioning the slave device 104 at the second side position 713 facilitates a kidney, heart, and / or lower abdomen surgical procedure. Additionally or alternatively, in some embodiments, the slave device 104 can be positioned at a third side position 714 relative to the operating support 504, the third side position 714 being near a foot end of the operating support 504. The third side position 714 can be on the patient’s left side or the patient’s right side of the operating support 504 Figure 7 A third side position 714 is shown in FIG. 7B on the patient’s left side). In some cases, positioning the slave device 104 at the third side position 714 facilitates a pelvic surgical procedure. Additionally or alternatively, in some embodiments, the slave device 104 can be positioned at a foot position 715 relative to the operating support 504, the foot position 715 being near a foot end of the operating support 504. In some cases, positioning the slave device 104 at the foot position 715 facilitates a pelvic surgical procedure.

[0065] Returning to Figure 4 In some embodiments, as part of the process 402, one or more repositionable structures of the computer-assisted system 100 that are not mounted on a cart or other slave device 104 are positioned relative to the patient. In such embodiments, the repositionable structures can be positioned in the process 402 by moving along a track (e.g., a ceiling-mounted, wall-mounted, floor-mounted track, or a patient support-mounted track). Additionally or alternatively, in such embodiments, the repositionable structures can be positioned relative to the patient in the process 402 by movement of the operating support relative to the repositionable structures. In either case, the surgical operator can directly oversee or perform such positioning of the one or more repositionable structures relative to the patient.

[0066] In the process 404, a manually controlled imaging device is inserted into the work site, e.g., by the surgical operator. Because the process 404 occurs before the final positioning of the repositionable structures relative to the patient and the docking of the instrument 126 with the patient, the insertion, orientation, and positioning of the imaging device is performed manually. Examples of manually controlled imaging devices that are manually inserted into the work site are described below in connection with Figure 8 Examples of manually controlled imaging devices that are manually inserted into the work site are described below in connection with

[0067] Figure 8 is a simplified diagram of a manually controlled imaging device 801 inserted into a work site 520 in accordance with some embodiments. As shown, a patient 502 is positioned on an operating support 504, and a manually controlled imaging device 801 is inserted into a work site 520 that includes a target anatomical structure 522. For example, the manually controlled imaging device 801 can be inserted into the work site 520 via a suitable port 802. In some embodiments, the manually controlled imaging device 801 is inserted into the work site 520 by the surgical operator.Figure 8 In the illustrated embodiment, the manually controlled imaging device 801 is mounted on a shaft 806 that is compatible for use with the port 802 or a sleeve inserted into the port 802. In some embodiments, the imaging device 801 is kinematically decoupled from the computer-assisted system 100, so the orientation and position of the imaging device 801 is not controlled by the control system 140.

[0068] In some embodiments, during the method 400, the manually controlled imaging device 801 is kinematically decoupled from the system (e.g., not mounted to a robotic arm or repositionable structure of the computer-assisted system 100), but can be kinematically coupled to a component of the computer-assisted system 100 (e.g., via a robotic arm or other repositionable structure) prior to the start of the surgical procedure. In some embodiments, the manually controlled imaging device 801 is also the same imaging device that was used prior to the initiation of the method 400 and electronically registered with the computer-assisted system 100.

[0069] Returning to Figure 4 In the process 406, as Figure 9 illustrated, the manually controlled imaging device 801 is manually directed toward the target anatomical structure 522. Figure 9 is a simplified diagram of the manually controlled imaging device 801 being directed toward the target anatomical structure 522, in accordance with some embodiments. In some embodiments, a surgeon operator or other sterile operator disposed within the sterile field 506 performs the process 406, for example, by monitoring images generated by the manually controlled imaging device 801 and moving the shaft 806 accordingly. In some embodiments, the target anatomical structure 522 is automatically detected by the computer-assisted system 100. For example, in some embodiments, the computer-assisted system 100 automatically detects the target anatomical structure 522 based on computer vision analysis of image information received from the manually controlled imaging device 801. In such embodiments, particular anatomical features and / or pathologies can be identified via the computer vision analysis and indicated to the surgeon operator or other sterile operator who is manually directing the manually controlled imaging device 801. In other embodiments, the surgeon operator directs the manually controlled imaging device 801 toward the target anatomical structure 522 based on image information received from the manually controlled imaging device 801.

[0070] The manually controlled imaging device 801 can be any technically feasible imaging device suitable for use within the work site 520. In some embodiments, the manually controlled imaging device 801 comprises an endoscopic imaging device. In such embodiments, the manually controlled imaging device 801 can be mounted on a straight shaft or an angled shaft. In some embodiments, the manually controlled imaging device 801 comprises a monoscopic or stereoscopic imaging device. Additionally or alternatively, in some embodiments, the manually controlled imaging device 801 comprises a light source, such as a visible, infrared, and / or ultraviolet light source. In some embodiments, the manually controlled imaging device 801 comprises an inertial measurement unit (IMU) mounted thereto that generates position and / or orientation information associated with the manually controlled imaging device 801.

[0071] Returning to Figure 4 In process 408, the computer-assisted system 100 receives an input indicating that a deployment selection process is to be initiated. In some embodiments, the input is a user input associated with the manually controlled imaging device 801, such as a button press or other input selection by the sterile operator controlling the manually controlled imaging device 801. Additionally or alternatively, in some embodiments, the input can be a voice command by the sterile operator, a gesture by the sterile operator, and / or a command entered via a user interface. In such embodiments, the input selection can be performed via an input mechanism coupled to the manually controlled imaging device 801 or included in the slave device 104, such as the guidance input device 106. Alternatively, in some embodiments, the input indicating that a deployment selection process is to be initiated is generated by the computer-assisted system 100, such as in response to the computer-assisted system 100 automatically detecting the target anatomical structure 522 based on computer vision analysis of the work site 520.

[0072] In process 410, the computer-assisted system 100 selects a deployment of the repositionable structure, such as the manipulator arm 120, so that a specified surgical procedure to be performed by the repositionable structure controlling an instrument will be well supported. In detail, when the computer-assisted system 100 receives the input indicating that a deployment selection process is to be initiated, the computer-assisted system 100 selects a suitable working orientation for the computer-assisted system 100 based on the position of the repositionable structure relative to the patient and the orientation of the manually controlled imaging device 801. It should be noted that when the computer-assisted system 100 receives the input, the manually controlled imaging device 801 is directed from the port 802 toward the target anatomical structure 522, as shown in FIG. 4B. Figure 9The manually controlled imaging device 801 is oriented in a direction similar to the direction 912 in which the shaft of the instrument 126 will be generally oriented (e.g., in an orientation that is 45 to 60 degrees from the direction 912) when accessing the work site 520 during the surgical procedure. Thus, when the computer-assisted system 100 receives an input indicating that the deployment selection process is to be initiated, the manually controlled imaging device 801 is oriented in a direction similar to the work direction of the target anatomical structure 522 of the patient 502. The following describes an example embodiment of the computer-assisted system 100 that selects a deployment of the repositionable structure based on the orientation of the manually controlled imaging device 801. Figures 10A-10D Example embodiments of the computer-assisted system 100 that have various deployments with different work directions are described. The following describes an example embodiment of the computer-assisted system 100 that selects a deployment of the repositionable structure based on the orientation of the manually controlled imaging device 801. Figure 11 and Figure 12 Example embodiments of the computer-assisted system 100 that select a deployment of the repositionable structure are described.

[0073] In the process 412, the computer-assisted system 100 displays instructions for deploying the repositionable structure in accordance with the selected deployment. In some embodiments, the selected deployment is displayed outside of the sterile field 506, e.g., on the display devices 302-1 and / or 302-2 of the follower device 104. Examples of the displayed instructions include instructions for repositioning of the follower device 104 (e.g., move forward, rotate left, rotate right, move closer to the patient’s head, move closer to the patient’s feet) and instructions for positioning a portion of the repositionable structure of the computer-assisted system 100 (e.g., rotate the orientation platform 204 to face the patient’s feet, rotate the orientation platform 204 to face the patient’s head, rotate the orientation platform 204 to face the patient’s left side, rotate the orientation platform 104 clockwise or counterclockwise to a specified angle relative to the side of the operating support 504, extend the boom supporting the orientation platform 204 a specified distance, move the orientation platform 204 vertically, etc.

[0074] Additionally or alternatively, in some embodiments, the selected deployment is displayed near the computer-assisted system 100 and / or the patient 502 via a wall-mounted, ceiling-mounted, or rack-mounted display device or another rack. Thus, a non-sterile operator can readily determine how to deploy the repositionable structure of the computer-assisted system 100 for the surgical procedure without having to rely on verbal instructions from a sterile operator. Alternatively, in some embodiments, one or more of the base or the repositionable structure can be automatically moved to the selected deployment in the process 412.

[0075] Example deployments of a computer-assisted system Figures 10A-10D is a simplified diagram of example deployments that can be selected for the repositionable structure of the computer-assisted system 100 in accordance with some embodiments. In some embodiments, in addition to the deployments shown in FIG. 21A, the computer-assisted system 100 can also have deployments in which the base 202 is positioned in a location that is not shown in FIG. 21A, e.g., a location that is not in the same room as the patient 502. Figures 10A-10DIn addition to the illustrated configuration, the computer-assisted system 100 may also select other configurations for the repositionable structure of the computer-assisted system 100 .

[0076] Figure 10A The deployment 1013 of the manipulator arm 120 is shown, wherein the follower device 104 is positioned relative to the operating support 504 and the patient 502. Figure 7 1023 . Furthermore, in deployment 1013, the orientation platform 204 is rotated so that the manipulator arm 120 is positioned for operation in the transverse working direction 1023, as shown. According to various embodiments, the computer-assisted system 100 selects deployment 1013 for the orientation platform 204 when it determines that the follower device 104 is positioned at a lateral position consistent with the second lateral position 713 and the manually controlled imaging device (not shown) is oriented in a direction substantially parallel to the transverse working direction 1023.

[0077] Figure 10B The deployment 1012 of the manipulator arm 120 is shown, wherein the follower device 104 is positioned relative to the operating support 504 and the patient 502. Figure 7 1012. In addition, in deployment 1012, the orientation platform 204 is rotated so that the manipulator arm 120 is positioned for operation in a "head-on" working orientation 1022, as shown. According to various embodiments, the computer-assisted system 100 selects deployment 1012 for the orientation platform 204 when it determines that the follower device 104 is positioned at a lateral position consistent with the first lateral position 712 and the manually controlled imaging device (not shown) is oriented in the head-on working orientation 1022.

[0078] Figure 10C The deployment 1014 of the manipulator arm 120 is shown, wherein the follower device 104 is positioned relative to the operating support 504 and the patient 502. Figure 7 1024 . Furthermore, in deployment 1014, the orientation platform 204 is rotated so that the manipulator arm 120 is positioned to operate in a "towards-feet" working orientation 1024, as shown. According to various embodiments, the computer-assisted system 100 selects deployment 1014 for the orientation platform 204 when it determines that the follower device 104 is positioned at a lateral position consistent with the third lateral position 714 and the manually controlled imaging device (not shown) is oriented in the "towards-feet" working orientation 1024.

[0079] Figure 10D The deployment 1015 of the manipulator arm 120 is shown, wherein the follower device 104 is positioned relative to the operating support 504 and the patient 502.Figure 7 the foot position 715. Further, in the deployment 1015, the orientation platform 204 is rotated so that the manipulator arm 120 is positioned to operate in the "toward feet" working direction 1025, as shown. According to various embodiments, the computer-assisted system 100 selects the deployment 1015 for the orientation platform 204 when the computer-assisted system 100 determines that the follow device 104 is positioned at a location coinciding with the foot position 715 and the manually-controlled imaging device (not shown) is oriented in the toward-feet working direction 1025.

[0080] In Figures 10A-10D the embodiment of the manipulator arm deployment shown, the orientation platform 204 is rotated to one of several discrete orientations so that the manipulator arm 120 is positioned to operate in one of the lateral working direction 1023, the "toward head" working direction 1022, or the "toward feet" working direction 1025. In other embodiments, in the deployment of the manipulator arm 120, the orientation platform 204 is aligned and oriented based on the orientation of a manually-controlled imaging device (e.g., the manually-controlled imaging device 801 in Figure 8 some embodiments, in the deployment of the manipulator arm 120, the orientation platform 204 is centered on the location of the manually-controlled imaging device. Alternatively or additionally, in some embodiments, in the deployment of the manipulator arm 120, the orientation platform 204 is aligned with the orientation of the manually-controlled imaging device, e.g., the face vector of the orientation platform 104 is aligned with the orientation of the manually-controlled imaging device. In other embodiments, in the deployment of the manipulator arm 120, the follow device 104 is positioned relative to the operating support based on the orientation and / or location of the manually-controlled imaging device. For example, in some embodiments, in the deployment of the manipulator arm 120, the follow device 104 is horizontally centered at the horizontal location of the manually-controlled imaging device.

[0081] Selecting a deployment of a computer-assisted system according to a first embodiment Figure 11 is a simplified diagram of a method 1100 for selecting a deployment of a computer-assisted system according to some embodiments. According to some embodiments, the method 1100 can include one or more of processes 1102-1106, which can be implemented at least partially in the form of executable code stored on a non-transitory, tangible machine-readable medium, which when run on one or more processors (e.g., the processing system 150 in the control system 140 of the computer-assisted system 100) can cause the one or more processors to perform one or more of the processes 1102-1106. In conjunction with Figure 1 Figure 11 ​In the described embodiment, the method 1100 is described for a medical application and a computer-assisted system 100. It will be apparent to those of ordinary skill in the art that the method 1100 can be performed with any other suitable robotic-assisted system. Moreover, in other embodiments, the method 1100 can be used for other non-surgical applications where it can be beneficial to automatically select a deployment of any other suitable computer-assisted system.

[0082] At a process 1102, the computer-assisted system 100 determines a position of one or more repositionable structures relative to the patient 502. In some embodiments, the repositionable structures include the manipulator arms 120 and / or the follower device 104. In some embodiments, the computer-assisted system 100 uses imaging information received from one or more imaging, ranging, or tracking sensors associated with the computer-assisted system 100 to determine the position of such repositionable structures. For example, in some embodiments, imaging information from the imaging device 202 is used in the process 1102 to determine the position of the repositionable structures of the computer-assisted system 100. In some embodiments, the computer-assisted system 100 performs computer vision analysis to determine whether the follower device 104 is disposed in one of the head position 711, the first side position 712, the second side position 713, the third side position 714, or the foot position 715 relative to the operating support 504. In some embodiments, point cloud techniques, object detection techniques, object segmentation techniques, and / or part segmentation techniques can be employed to identify objects or portions thereof. These techniques can be machine learning based or classic computer vision algorithms. Details of example computer vision techniques that utilize machine learning are described in PCT / US2001 / 059213, entitled “VISIBILITY METRICS IN MULTI-VIEW MEDICAL ACTIVITY RECOGNITION SYSTEMS AND METHODS,” filed November 12, 2021, which is incorporated by reference herein.

[0083] In some embodiments, the one or more imaging, ranging, or tracking sensors include the imaging device 202. In some embodiments, the one or more imaging, ranging, or tracking sensors can be mounted on the computer-assisted system 100, for example on the base, on a portion of a particular repositionable structure, or on the orientation platform 204. Additionally or alternatively, the one or more imaging, ranging, or tracking sensors can be mounted elsewhere in the operating environment, for example worn by a surgeon operator or a non-surgeon operator, or mounted to a surface outside of the computer-assisted system 100, like a wall, ceiling, or floor, and / or mounted on a table, cart, or other equipment.

[0084] At process 1104, the computer-assisted system 100 determines an orientation of the imaging device 801 relative to the patient 502. In such embodiments, the computer- assisted system 100 performs computer vision analysis to determine the orientation of the imaging device 801 relative to the patient 502. In some embodiments, the computer-assisted system 100 receives imaging information from similar imaging, ranging, or tracking sensors associated with the computer-assisted system 100 that generated the imaging information received in process 1102. In some embodiments, the computer-assisted system 100 performs the computer vision analysis of process 1104 using one or more of the same object recognition techniques employed in process 1102. Additionally or alternatively, in some embodiments, the computer-assisted system 100 determines the orientation of the manually controlled imaging device 801 based at least in part on position and / or orientation information received from an IMU mounted on the manually controlled imaging device 801 (e.g., pitch, yaw, and roll of the shaft 806 on which the manually controlled imaging device 801 is mounted). In some embodiments, the computer-assisted system 100 selects the imaging device orientation from a set of discrete predetermined orientations (e.g., “toward patient’s left side,” “toward patient’s right side,” “facing patient’s head,” and “toward patient’s feet”).

[0085] At process 1106, the computer-assisted system 100 determines a deployment of one or more repositionable structures (e.g., manipulator arms 120) of the computer-assisted system 100. In process 1106, the computer-assisted system 100 determines an appropriate deployment based on the position of the one or more repositionable structures determined in process 1102 and the orientation of the manually controlled imaging device 801 determined in process 1104. After completing process 1106, the computer-assisted system 100 can perform process 412, in which the computer-assisted system 100 displays instructions for deploying the repositionable structures according to the deployment selected in process 1106. In some embodiments, the computer-assisted system 100 determines the deployment of the one or more repositionable structures based on the pose of the manually controlled imaging device 801. In such embodiments, the pose can include both the orientation of the manually controlled imaging device 801 and the position of the manually controlled imaging device 801 relative to the computer-assisted system 100.

[0086] In some embodiments, a table lookup is performed in process 1106 using the determined position of the repositionable structures as a first input and using the current orientation of the manually controlled imaging device 801 as a second input. Based on the first input and the second input, the pre-defined lookup table can then indicate an appropriate deployment. Thus, the lookup table indicates a pre-determined association between a particular deployment and a particular combination of the current position of the repositionable structures relative to the patient and the current orientation of the imaging device relative to the patient.

[0087] In some embodiments, the number of possible positions of the repositionable structure is limited to a small number of discrete options, such as "on the left side of the patient," "on the right side of the patient," "at the patient's feet," and "at the patient's head." Likewise, in some embodiments, the number of possible orientations of the manually controlled imaging device 801 is limited to a small number of discrete options, such as "toward the left side of the patient," "toward the right side of the patient," "toward the patient's head," and "toward the patient's feet." In these embodiments, the pre-defined lookup table can include a relatively small number of available output deployments. Examples of such output deployments include a deployment in which, for the current position of the follow-up device 104 relative to the patient 502, the orientation platform 204 is rotated so that the manipulator arm 120 is positioned to operate in an appropriate working direction. In some embodiments, such a working direction can include a direction parallel to the patient's longitudinal axis (e.g., the head-to-feet axis) and toward the patient's head, a direction parallel to the patient's longitudinal axis and toward the patient's feet, a direction perpendicular to the patient's longitudinal axis and toward the patient's left side, and / or a direction perpendicular to the patient's longitudinal axis and toward the patient's right side. In other embodiments, a greater number of possible discrete positions of the repositionable structure and / or possible discrete orientations of the manually controlled imaging device 801 can be employed to determine an appropriate deployment using table lookup. In some embodiments, such a working direction can include a direction parallel to the imaging device orientation and / or a direction at an acute angle to the patient's longitudinal axis.

[0088] Selecting a deployment of a computer-assisted system according to a second embodiment Figure 12 FIG. 1 1 is a simplified diagram of a method 1200 for selecting a deployment of a computer-assisted system according to some embodiments. According to some embodiments, the method 1200 can include one or more of processes 1202-1212, which can be implemented at least partially in the form of executable code stored on a non-transitory, tangible machine-readable medium, which when executed on one or more processors (e.g., a processing system 150 in a control system 140 of a computer-assisted system 100), can cause the one or more processors to perform one or more of the processes 1202-1212. In embodiments described in conjunction with FIGS. 1-10, the method 1200 is described for a medical application and a computer-assisted system 100. It will be apparent to those of ordinary skill in the art that the method 1200 can be performed with any other suitable robotic-assisted system. Moreover, in other embodiments, the method 1200 can be used for other non-surgical applications in which it can be beneficial to automatically select a deployment of any other suitable computer-assisted system. Figure 1 In embodiments described in conjunction with FIGS. 1-10, the method 1200 is described for a medical application and a computer-assisted system 100. It will be apparent to those of ordinary skill in the art that the method 1200 can be performed with any other suitable robotic-assisted system. Moreover, in other embodiments, the method 1200 can be used for other non-surgical applications in which it can be beneficial to automatically select a deployment of any other suitable computer-assisted system. Figure 12 In embodiments described in conjunction with FIGS. 1-10, the method 1200 is described for a medical application and a computer-assisted system 100. It will be apparent to those of ordinary skill in the art that the method 1200 can be performed with any other suitable robotic-assisted system. Moreover, in other embodiments, the method 1200 can be used for other non-surgical applications in which it can be beneficial to automatically select a deployment of any other suitable computer-assisted system.

[0089] At process 1202, the computer-assisted system 100 determines the position of the one or more repositionable structures relative to the patient 502. In some embodiments, the process 1202 is consistent with the process 1102 in Figure 11

[0090] At process 1204, the computer-assisted system 100 determines the position and orientation of the patient 502 relative to the computer-assisted system 100 in a common reference frame. In such embodiments, the computer-assisted system 100 can map the position of the one or more repositionable structures and the position and orientation of the patient 502 to a common reference frame, such as a world or system coordinate frame or a patient-centered coordinate frame. In some embodiments, the computer-assisted system 100 uses imaging information received from one or more of the same imaging, ranging, or tracking sensors associated with the computer-assisted system 100 that were used to determine the position of the repositionable structures in process 1202.

[0091] At process 1206, the computer-assisted system 100 receives image information from the manually controlled imaging device 801. Generally, the image information received in process 1206 can include imaging of the target anatomical structure 522.

[0092] At process 1208, the computer-assisted system 100 determines the position and orientation of the manually controlled imaging device 801. In such embodiments, the computer-assisted system 100 can then map the position and orientation of the manually controlled imaging device 801 to a common reference frame, such as a world or system coordinate frame or a patient-centered coordinate frame. Accordingly, the computer-assisted system 100 can map the position and orientation of the manually controlled imaging device 801 to the common reference frame along with the position of the one or more repositionable structures and the position and orientation of the patient 502.

[0093] ​In some embodiments, the computer-assisted system 100 determines the position and orientation of the manually controlled imaging device 801 based on the image information of the target anatomical structure 522 received in process 1206. In such embodiments, the computer-assisted system 100 performs computer vision analysis of the imaging of the target anatomical structure 522 using one or more of the same object recognition techniques employed in process 1102. Additionally or alternatively, in some embodiments, the computer-assisted system 100 determines the orientation of the manually controlled imaging device 801 based at least in part on position and / or orientation information received from an IMU mounted on the manually controlled imaging device 801 (e.g., the pitch, yaw, and roll of the shaft 806 on which the manually controlled imaging device 801 is mounted). Additionally or alternatively, in some embodiments, the computer-assisted system 100 determines the orientation and position of the manually controlled imaging device 801 based at least in part on the location at which the manually controlled imaging device 801 is inserted into the patient’s anatomy (e.g., the location of the port 802). Additionally or alternatively, in some embodiments, the computer-assisted system 100 determines the orientation and position of the manually controlled imaging device 801 based at least in part on the insertion depth of the shaft 806 as indicated by the same imaging, ranging, or tracking sensors associated with the computer-assisted system 100 used to determine the position of the repositionable structure in process 1202. Additionally or alternatively, in some embodiments, the computer-assisted system 100 determines the orientation and position of the manually controlled imaging device 801 based at least in part on the insertion angle of the shaft 806 as indicated by the same imaging, ranging, or tracking sensors associated with the computer-assisted system 100.

[0094] At process 1210, the computer-assisted system 100 determines the location of the target anatomical structure 522 relative to the position and orientation of the patient 502. In such embodiments, the computer-assisted system 100 determines the particular sector or region of the patient’s 502 anatomy in which the target anatomical structure 522 is disposed. For example, in some embodiments, the patient’s 502 anatomy can be divided into predefined sectors or quadrants, and in process 1210, the computer-assisted system 100 determines in which predefined sector or quadrant the target anatomical structure 522 is disposed. Thus, in these embodiments, the computer-assisted system 100 determines which of a discrete number of available locations of the target anatomical structure 522 the target anatomical structure 502 is located. In some embodiments, the computer-assisted system 100 determines in which predefined sector or quadrant the target anatomical structure 522 is disposed based on various inputs, including the position and orientation of the patient 502 determined in process 1204, the orientation of the manually controlled imaging device 801 determined in process 1208, and / or the image information of the target anatomical structure 522 received in process 1206.

[0095] At process 1212, the computer-assisted system 100 determines a deployment of one or more repositionable structures (e.g., the manipulator arms 120) of the computer-assisted system 100. In process 1212, the computer-assisted system 100 determines an appropriate deployment based on the particular inputs (e.g., the position of the one or more repositionable structures determined in process 1202, the position and orientation of the patient 502 determined in process 1204, the position of the target anatomical structure 522 determined in process 1210, and / or the orientation of the manually controlled imaging device 801 determined in process 1208). In some embodiments, the computer-assisted system 100 selects an appropriate deployment of the one or more repositionable structures from a discrete number of available deployments in process 1212. Thus, in these embodiments, each combination of the particular inputs has a predetermined association with a discrete deployment of the one or more repositionable structures of the computer-assisted system 100. An example of such a deployment includes a deployment in which, for the current position of the follow-up device 104 relative to the patient 502, the platform 204 is rotated such that the manipulator arms 120 are positioned to operate in an appropriate working orientation. After completing process 1212, the computer-assisted system 100 can perform process 412 in which the computer-assisted system 100 displays instructions for deploying the repositionable structures according to the deployment selected in process 1212.

[0096] In the above-described embodiments, as the imaging device is directed toward the target anatomical structure and captures an image of the target anatomical structure, the computer-assisted system 100 uses sensor data to determine an imaging device orientation relative to the patient on the operating support. The computer-assisted system 100 then uses the imaging device orientation to automatically select a deployment of the repositionable structures for guiding the non-surgeon operator. In other embodiments, the computer-assisted system 100 employs a multi-target approach in which multiple target anatomical structures are identified and stored, and the repositionable structures can change from a first deployment to a second deployment based on one or more of the stored target anatomical structures. In some embodiments, each of the multiple target anatomical structures can be disposed in a different sector or quadrant of the patient anatomical structure. In some embodiments, the repositionable structures change from a first deployment to a second deployment during a procedure (e.g., during a change in procedure phase from a first target anatomical structure to a second target anatomical structure) in response to operator instructions and / or in response to a change in a mode of the computer-assisted system 100. Alternatively or additionally, in some embodiments, a single deployment is selected based on the multiple target anatomical structures, such that the computer-assisted system 100 selects a deployment that is appropriate for the procedure associated with two or more of the target anatomical structures.

[0097] While the illustrative embodiments have been illustrated and described, it will be apparent to those of ordinary skill in the art that many changes, substitutions and alterations can be made to the embodiments described without departing from the intended broad aspect of the application, and some features of the present embodiments can be employed without a corresponding use of other features. It will be appreciated that many variations, modifications and changes can be made thereunto in light of the above description without departing from the scope of the present application as understood by one of ordinary skill in the art. It is intended to cover in the appended claims all such changes and modifications that fall within the scope of the application. Accordingly, the scope of the present application should not be limited by the foregoing description, but instead should be defined in accordance with the following claims, and equivalents thereof.

[0098] Any and all combinations of any of the claim elements and / or any of the elements described in this application, in any way, are within the intended scope of the application and protected.

[0099] The description of the various embodiments has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0100] Aspects of the present embodiments can be embodied as a system, method, or computer program product. Accordingly, aspects of the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that can generally be referred to herein as a "module", "system" or "computer". Furthermore, any of the hardware and / or software technology, processes, functions, components, engines, modules or systems described in the present disclosure can be implemented as a circuit or a set of circuits. Additionally, aspects of the present disclosure can take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

[0101] Any combination of one or more computer readable medium(s) can be utilized. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or apparatus or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0102] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to produce a machine, such that the instructions that execute on the computer or other programmable data processing apparatus create means for implementing the functions specified in the flowchart block or blocks. These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture that includes an implementation to

[0103] The flow and block diagrams in the drawings represent architectures, functional and operational aspects of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending on the functionality involved. It will also be noted that each block in the block and / or flow diagrams and combinations of blocks in the block and / or flow diagrams can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0104] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure can be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.

Claims

1. A computer-aided system comprising: Relocatable structures; and processing systems; The processing system is configured to perform the following steps: determining a position of the repositionable structure relative to a patient on an operating support; determining an imaging device orientation relative to the patient; determining a deployment of the repositionable structure based on the imaging device orientation and the position of the repositionable structure relative to the patient; as well as One or more instructions for configuring the repositionable structure are caused to be displayed.

2. The computer-assisted system of claim 1, wherein the imaging device is kinematically decoupled from the computer-assisted system.

3. The computer-assisted system of claim 1, wherein determining the imaging device orientation is performed in response to receiving user input associated with the imaging device.

4. The computer-assisted system of claim 1, wherein determining the imaging device orientation is performed in response to the processing system determining that the imaging device is directed toward a target anatomy of the patient.

5. The computer-assisted system of claim 4, wherein determining that the imaging device is directed toward the target anatomical structure of the patient comprises: receiving image information from the imaging device; as well as Computer vision analysis is performed on the image information.

6. The computer-assisted system of claim 1, wherein determining the imaging device orientation comprises at least one of receiving information from one or more imaging sensors, receiving information from one or more ranging sensors, or receiving information from one or more tracking sensors.

7. The computer-assisted system of claim 6, wherein the one or more imaging sensors, the one or more ranging sensors, or the one or more tracking sensors are mounted on at least one of a component of the computer-assisted system, a surface external to the computer-assisted system, or a non-sterile operator of the computer-assisted system.

8. The computer-assisted system of any one of claims 1 to 7, wherein determining the deployment of the repositionable structure comprises determining a working direction of the repositionable structure based on the imaging device orientation.

9. The computer-assisted system of claim 8, wherein determining the working direction based on the imaging device orientation comprises selecting the working direction from a set of predetermined working directions.

10. The computer-assisted system according to claim 9, wherein the set of predetermined working directions includes a direction toward the left side of the operating support, a direction toward the right side of the operating support, a direction toward the patient's feet, and a direction toward the patient's head.

11. The computer-assisted system of claim 9, wherein selecting the working direction from the set of predetermined working directions comprises performing a table lookup.

12. The computer-assisted system of claim 8, wherein determining the working direction comprises one of: a direction parallel to an orientation of the imaging device; a direction at an acute angle to a longitudinal axis of the patient; a direction parallel to the longitudinal axis of the patient and toward the patient's head; a direction parallel to the longitudinal axis of the patient and toward the patient's foot; A direction perpendicular to the longitudinal axis of the patient and toward the left side of the patient; or a direction perpendicular to the longitudinal axis of the patient and toward the right side of the patient.

13. The computer-aided system according to any one of claims 1 to 7, wherein: The repositionable structure is mounted on a servo device of the computer-aided system, and Determining the position of the repositionable structure relative to the patient includes selecting a position of the follower device from a group of positions, the group of positions consisting of a position near the left side of the operating support, a position near the right side of the operating support, a position near the patient's feet, and a position near the patient's head.

14. The computer-assisted system according to any one of claims 1 to 7, wherein the processing system is configured to perform the step of determining the position of a target anatomical structure of the patient.

15. The computer-assisted system of claim 14, wherein determining the deployment of the repositionable structure comprises determining the deployment based on a predetermined correlation between the position of the target anatomical structure and the deployment. 16 . The computer-assisted system according to claim 15 , wherein the predetermined association between the position of the target anatomical structure and the deployment is included in a plurality of predetermined associations.

17. The computer-assisted system of claim 16, wherein each predetermined association is between a different position of the target anatomical structure and an available deployment of the repositionable structure.

18. The computer-assisted system of claim 14, wherein determining the position of the target anatomical structure of the patient comprises receiving information from an inertial measurement unit coupled to the imaging device.

19. The computer-assisted system of claim 18, wherein determining the location of the target anatomical structure is based on a location at which the imaging device is inserted into the patient's internal anatomy.

20. The computer-assisted system of claim 19, wherein determining the position of the target anatomical structure is further based on an insertion depth of an instrument including the imaging device into the internal anatomy of the patient.

21. The computer-assisted system of claim 19, wherein determining the position of the target anatomical structure is further based on an insertion angle of an instrument including the imaging device into the internal anatomy of the patient.

22. The computer-assisted system of claim 18, wherein determining the position of the target anatomical structure comprises mapping the imaging device position, the imaging device orientation, and the position of the repositionable structure relative to the patient to a common reference frame.

23. The computer-assisted system of claim 14, wherein determining the location of the target anatomical structure of the patient comprises determining a region or organ to be accessed.

24. A computer-assisted system according to any one of claims 1 to 7, wherein the one or more instructions for configuring the repositionable structure include at least one of the following: instructions for positioning a follower device comprising the repositionable structure or instructions for positioning a portion of the repositionable structure of the computer-assisted system.

25. The computer-assisted system of any one of claims 1 to 7, wherein determining the deployment of the repositionable structure comprises determining a location of a target anatomical structure of the patient.

26. The computer-assisted system of claim 25, wherein determining the position of the target anatomical structure of the patient comprises determining a region of the patient where the target anatomical structure is located.

27. The computer-assisted system of any one of claims 1 to 7, wherein determining the deployment of the repositionable structure is further based on the orientation of the imaging device.

28. A method comprising: determining, by the processing system, a position of a repositionable structure of the computer-assisted system relative to a patient on an operating support; determining, by the processing system, an imaging device orientation relative to the patient; determining, by the processing system, a deployment of the repositionable structure based on the imaging device orientation and the position of the repositionable structure relative to the patient; as well as One or more instructions for configuring the repositionable structure are caused to be displayed, by the processing system.

29. The method of claim 28, wherein the imaging device is kinematically decoupled from the computer-assisted system.

30. The method of claim 28, wherein determining the imaging device orientation is performed in response to receiving user input associated with the imaging device.

31. The method of claim 28, wherein determining the imaging device orientation is performed in response to the processing system determining that the imaging device is directed toward a target anatomy of the patient.

32. The method of claim 31 , wherein determining that the imaging device is directed toward the target anatomical structure of the patient comprises: receiving image information from the imaging device; as well as Computer vision analysis is performed on the image information.

33. The method of claim 28, wherein determining the imaging device orientation comprises at least one of receiving information from one or more imaging sensors, receiving information from one or more ranging sensors, or receiving information from one or more tracking sensors.

34. The method of claim 33, wherein the one or more imaging sensors, the one or more ranging sensors, or the one or more tracking sensors are mounted on at least one of a component of the computer-assisted system, a surface external to the computer-assisted system, or a non-sterile operator of the computer-assisted system.

35. The method of claim 28, wherein determining the deployment of the repositionable structure comprises determining a working direction of the repositionable structure based on the imaging device orientation.

36. The method of claim 35, wherein determining the working direction based on the imaging device orientation comprises selecting the working direction from a set of predetermined working directions.

37. The method of claim 36, wherein the set of predetermined working directions includes a direction toward a left side of the operating support, a direction toward a right side of the operating support, a direction toward the patient's feet, and a direction toward the patient's head.

38. The method of claim 36, wherein selecting the operating direction from the set of predetermined operating directions comprises performing a table lookup.

39. The method of claim 35, wherein determining the working direction comprises one of: a direction parallel to an orientation of the imaging device; a direction at an acute angle to a longitudinal axis of the patient; a direction parallel to the longitudinal axis of the patient and toward the patient's head; a direction parallel to the longitudinal axis of the patient and toward the patient's foot; A direction perpendicular to the longitudinal axis of the patient and toward the left side of the patient; or a direction perpendicular to the longitudinal axis of the patient and toward the right side of the patient.

40. The method of claim 28, wherein: The repositionable structure is mounted on a servo device of the computer-aided system, and Determining the position of the repositionable structure relative to the patient includes selecting a position of the follower device from a group of positions, the group of positions consisting of a position near the left side of the operating support, a position near the right side of the operating support, a position near the patient's feet, and a position near the patient's head.

41. The method of claim 28, further comprising determining, by the processing system, a location of a target anatomical structure of the patient.

42. The method of claim 41, wherein determining the deployment of the repositionable structure comprises determining the deployment based on a predetermined correlation between the position of the target anatomical structure and the deployment.

43. The method of claim 42, wherein the predetermined association between the position of the target anatomical structure and the deployment is included in a plurality of predetermined associations.

44. The method of claim 43, wherein each predetermined association is between a different location of the target anatomical structure and an available deployment of the repositionable structure.

45. The method of claim 41, wherein determining the position of the target anatomical structure of the patient comprises receiving information from an inertial measurement unit coupled to the imaging device.

46. ​​The method of claim 45, wherein determining the location of the target anatomical structure is based on a location at which the imaging device is inserted into the patient's internal anatomy.

47. The method of claim 46, wherein determining the position of the target anatomical structure is further based on an insertion depth of an instrument including the imaging device into the internal anatomy of the patient.

48. The method of claim 46, wherein determining the position of the target anatomical structure is further based on an insertion angle of an instrument including the imaging device into the internal anatomy of the patient.

49. The method of claim 45, wherein determining the position of the target anatomical structure comprises mapping the position of the imaging device, the imaging device orientation, and the position of the repositionable structure relative to the patient to a common reference frame.

50. The method of claim 41, wherein determining the location of the target anatomical structure of the patient comprises determining a region or organ to be accessed.

51. A method according to claim 28, wherein the one or more instructions for configuring the repositionable structure include at least one of the following: instructions for positioning a follower device that includes the repositionable structure or instructions for positioning a portion of the repositionable structure of the computer-assisted system.

52. The method of claim 28, wherein determining the deployment of the repositionable structure comprises determining a location of a target anatomical structure of the patient.

53. The method of claim 52, wherein determining the location of the target anatomical structure of the patient comprises determining a region of the patient in which the target anatomical structure is located.

54. The method of claim 28, wherein determining the deployment of the repositionable structure is further based on an orientation of the imaging device.

55. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions adapted, when executed by one or more processors associated with a computer-assisted device, to cause the computer-assisted device to perform the method of any one of claims 28-54.

Images