Method and system for coordinating user assistance

Abstract

A coordination system for coordinating remote assistance of a medical system involves a support agent executed by a computer processor. The support agent is configured to receive a first support request submitted by a user of the medical system and identify a support person to respond to the first support request from a plurality of support persons by applying support person identification logic to a system internal context. The system internal context is obtained from a system awareness engine configured to detect information about the medical system. The support agent is further configured to facilitate contact of the support person.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 62 / 942,696, filed December 2, 2019, pursuant to 35 U.S.SC §119(e). U.S. Provisional Patent Application Serial No. 62 / 942,696 is incorporated herein by reference in its entirety. Technical Field

[0003] This invention generally provides improved robotic and / or medical (including surgical) devices, systems, and methods. Background Technology

[0004] Overview

[0005] Robotic systems can be used to perform tasks at a work site. For example, a robotic system may include a robotic manipulator to manipulate machinery used to perform a task.

[0006] Example robotic systems include industrial and recreational robotic systems. Example robotic systems also include medical robotic systems for procedures such as diagnosis, non-surgical treatment, and surgical treatment. As a specific example, robotic systems include minimally invasive robotic remote surgery systems, where surgeons can perform surgery on patients from a bedside or remote location. Remote surgery generally refers to surgery performed using a surgical system where the surgeon uses some form of remote control (e.g., a servo mechanism) to manipulate the movement of surgical instruments, rather than directly holding and moving the instruments by hand. Robotic medical systems that can be used for remote surgery or other telemedicine procedures may include remotely controllable robotic manipulators. Operators can remotely control the movement of remotely controllable robotic manipulators. Operators can also manually move individual parts of the robotic medical system to a position or orientation within the robotic medical system's operating environment.

[0007] A robotic system according to one or more embodiments can be complex. For example, a robotic system may include multiple links connected by multiple joints. Some or all of these joints may be controllable by a user via a control algorithm capable of coordinating joint movements. Furthermore, some of these joints may be able to operate in different modes depending on the task requirements. The robotic system may also be equipped with instruments. These instruments may have multiple degrees of freedom and different types of instruments may be interchangeable. Operating a robotic system of this complexity may not be easy. Moreover, the procedures performed using the robotic system may be complex and may require careful planning, especially when using the robotic system to perform surgery on a patient. Therefore, providing adequate user support may be beneficial, or even essential. Summary of the Invention

[0008] Generally, in one aspect, one or more embodiments relate to a coordination system for coordinating remote assistance to a medical system, the coordination system comprising: a support agent, executed by a computer processor and configured to: receive a first support request submitted by a user of the medical system, identify support personnel by applying support personnel identification logic to a system internal context / condition in response to the first support request from a plurality of support personnel, wherein the system internal context is obtained from a system awareness engine configured to detect information about the medical system and facilitate contact with support personnel.

[0009] Generally, in one aspect, one or more embodiments relate to a method for operating a coordination system, the method comprising: receiving a first support request submitted by a user of a healthcare system by a support agent; identifying support personnel in response to the first support request from a plurality of support personnel by applying support personnel identification logic to an internal system context, wherein the internal system context is obtained from a system awareness engine configured to detect information about the healthcare system; and facilitating contact between support personnel.

[0010] Generally, in one aspect, one or more embodiments relate to a non-transitory machine-readable medium including a plurality of machine-readable instructions executed by one or more processors associated with a coordination system, the plurality of machine-readable instructions causing the one or more processors to perform a method comprising: receiving a first support request submitted by a user of a healthcare system by a support agent; identifying support personnel in response to the first support request from a plurality of support personnel by applying support personnel identification logic to an internal system context, wherein the internal system context is obtained from a system awareness engine configured to detect information about the healthcare system; and facilitating contact between support personnel.

[0011] Other aspects of the invention will become apparent from the following description and the appended claims. Attached Figure Description

[0012] Figure 1 A block diagram of a system for coordinating user assistance according to one or more embodiments is shown.

[0013] Figure 2A A top view of a computer-aided medical system in a robotic programming scenario according to one or more embodiments is shown.

[0014] Figure 2B The illustration schematically shows one or more embodiments. Figure 2A Various components of the robot program scenario.

[0015] Figure 3 An example of a manipulator arm assembly according to one or more embodiments is shown.

[0016] Figure 4A A flowchart illustrating a method for obtaining the internal context of a system according to one or more embodiments is shown.

[0017] Figure 4B A flowchart illustrating a method for obtaining the external context of a system according to one or more embodiments is shown.

[0018] Figure 5 A flowchart illustrating a method for providing support in response to a support request, according to one or more embodiments, is shown.

[0019] Figure 6A and Figure 6B An example of a set of rules for identifying appropriate support personnel, according to one or more embodiments, is shown. Detailed Implementation

[0020] Specific embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. For consistency, the same elements in the drawings are indicated by the same reference numerals.

[0021] In the following detailed description of embodiments of this disclosure, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.

[0022] Throughout the application, ordinal numbers (e.g., first, second, third, etc.) may be used as adjectives for elements (i.e., any noun in this application). The use of ordinal numbers does not imply or create any particular order of elements, nor does it limit any element to being a single element, unless explicitly stated otherwise, such as through the use of the terms “before,” “after,” “single,” and other such terms. Rather, ordinal numbers are used to distinguish elements. As an example, the first element is distinct from the second element, and the first element may contain more than one element and be ordered after (or before) the second element.

[0023] While some examples described herein relate to surgical procedures or tools, or medical procedures and tools, the disclosed techniques are applicable to both medical and non-medical procedures and tools. For example, the tools, systems, and methods described herein can be used for non-medical purposes, including industrial use, general robotic use, and sensing or manipulating non-tissue artifacts. Other example applications include cosmetic enhancement, imaging of human or animal anatomy, collecting data from human or animal anatomy, setting up or dismantling systems, and training medical or non-medical personnel. Other example applications include procedures for removing tissue from human or animal anatomy (without returning the anatomy), and performing procedures on human or animal cadavers. Furthermore, these techniques can be used in medical or diagnostic procedures, with or without surgical aspects.

[0024] Generally, according to one or more embodiments, embodiments of this disclosure can coordinate remote assistance to support users of robots and / or medical systems. Coordination of remote assistance may include, for example, identifying suitable support personnel from a pool of potentially available support staff and providing the identified support personnel with helpful information. Coordination may further include scheduling responses to support requests, particularly when multiple competing support requests exist.

[0025] In one or more embodiments, the coordination of remote assistance takes into account the context associated with the support request. Contextual information can be collected from the robot or medical system itself (internal system context) and / or by observing the operating environment in which the system operates (external system context). Contextual information can be used to efficiently obtain support for the user who issued the support request. In one or more embodiments, the robot or medical system includes a number of components whose states can be detected and used to determine the operating state of the system. For example, one or more sensors can be used to detect internal system context, such as the state of the system's robotic manipulators or the state of instruments potentially attached to them. Further examples of internal system context that can be detected include whether a part of the system is covered by a curtain, whether a cannula for guiding medical instruments (in a medical example) is connected to the medical system, and whether and what errors or malfunctions have been detected during system operation. This information obtained from the detection can then be used as internal system context information. Examples of external system context information include information acquired by a computer vision system of the operating environment outside the system. For example, image data of the workspace can be used to identify the current stage of a procedure being performed using a medical or robotic system.

[0026] In combination, the contextual information obtained from a robot or medical system can be used for a variety of purposes. Example purposes include, but are not limited to: identifying the appropriate support personnel to contact when a user of the system issues a support request, prioritizing support requests, and compiling information to provide to support personnel. These and additional features, and how they are implemented, will be discussed in the following description.

[0027] Referring now to the accompanying drawings, in which the same reference numerals denote the same parts in several views. Figure 1 A block diagram of a system for coordinating remote assistance from a computer-aided system (110) according to one or more embodiments is schematically shown. During operation, the computer-aided system (110) interacts with a target (196) in the operating environment and is operated by a user (198). The computer-aided system (110) may include a computer-aided medical system, such as a computer-aided diagnostic system or a computer-aided surgical system. The coordination system for coordinating remote assistance for the computer-aided system (110) may include a support agent (150) and may further include a system awareness engine (120) and / or an environment awareness engine. Support personnel (192A-192N) from the support team can be contacted to support the user (198). Each of these elements will be described subsequently.

[0028] In one or more medical embodiments, the computer-aided system (110) is as follows (reference hereinafter). Figure 2A , Figure 2B and Figure 3 The described medical system, or any other type of medical system. Alternatively, in other medical embodiments, the computer-assisted system (110) is a non-surgical medical system (such as a non-invasive diagnostic system). Furthermore, as another example, the computer-assisted system (110) can be a non-medical system (such as an industrial robot).

[0029] In the medical example, the user (198) of the computer-assisted system (110) can be a healthcare professional operating the computer-assisted system (110). For a computer-assisted system (110) that includes a surgical system, the healthcare professional can be a surgeon or a surgical assistant.

[0030] In one medical embodiment, the target may be a removed portion of a human or animal anatomical structure, a corpse, a person, an animal, etc. Figure 1 The target (196) shown as an example is a patient undergoing medical testing or treatment via a procedure executed by a computer-assisted system (110).

[0031] In one or more embodiments, the system awareness engine (120) is software, hardware, and / or a combination of software and hardware configured to determine the internal context (122) of the system. The system awareness engine (120) includes a set of machine-readable instructions (stored on a computer-readable medium) that are executed when performed by a computing device. Figure 4A One or more operations are described in the flowchart. The system awareness engine (120) can be used in the computing device of the computer-aided system (110) (such as the one described in the following reference). Figure 2B The computer-aided system is implemented on the computing system described. The computer-aided system can be a medical system. In general, the system perception engine (120) acquires parameters of the computer-aided system (110). These parameters may include any information that provides insight into the current operational state of the computer-aided system (110). For example, many parameters of a robot manipulation system (see reference below) Figure 2A , Figure 2B and Figure 3 As mentioned above, parameters such as joint position or speed, tool or instrument type, tool or instrument status, and control mode can be collected. Other parameters that can be collected include hardware serial number, firmware version, installed hardware and / or software modules, error logs, etc. The collected parameters can be compiled to form a system internal context (122), which will be referred to below. Figure 4A Let's discuss this in more detail.

[0032] In one or more embodiments, the camera (130) is configured to capture images or image sequences (videos) of the computer-assisted system (110) and / or the operating environment outside the computer-assisted system, such as the space around or near the computer-assisted system. Thus, the camera (130) can provide image data suitable for determining the external context of the system (i.e., context not necessarily derived directly from data about the computer-assisted system (110) itself). The computer-assisted system may be a medical system. For example, the image data can be used to detect the state of a target (e.g., the target (196) is an example of a patient: what is the patient's state if the patient is present (patient is ready and / or not ready for anesthesia, etc.)). The camera (130) can capture two-dimensional or three-dimensional image data. The camera may be mounted on a component of the computer-assisted system (110) or on a wall, ceiling, etc. Alternatively, the camera may be part of an augmented or mixed reality system (198) worn by the user. Head tracking may be integrated to enable the captured image data to be registered with the computer-assisted system (110) and / or the operating environment of the computer-assisted system. As an alternative to or addition to the camera (130), other types of operating environment sensors can be used. For example, one or more motion sensors, laser scanners, ultrasound scanners, etc., can be used. Sensors integrated with other devices can also be used. For example, sensors can be integrated into the operating table to detect the presence of a patient, or one or more sensors can be integrated into an anesthesia cart to allow monitoring of anesthesia status, etc.

[0033] An environment-aware engine (140) is software, hardware, and / or a combination of software and hardware configured to determine the external context (142) of the system. The environment-aware engine (140) includes a set of machine-readable instructions (stored on a computer-readable medium) that are executed when carried out by a computing device. Figure 4B One or more operations are described in the flowchart. The environment-aware engine (140) can be on the computing device of the computer-aided system (110) (such as the one described in the following reference). Figure 2B The described computing system is implemented on the environment perception engine (140). In general, the environment perception engine (140) processes image data acquired from the camera (130) to extract additional information beyond the system parameters of the computer-aided system (110). For example, the environment perception engine (140) can detect the position of the computer-aided system (110) relative to a target (196). In a medical example, the environment perception engine (140) can detect whether the target (196) is already present for the upcoming operation, whether the components of the computer-aided system (110) are inside or outside a sterile area, etc. See below for reference. Figure 4BThe flowchart describes other examples of the system's external context derived by the environment-aware engine (140) (such as digital image processing using image data provided by the camera (130)). The information collected by the environment-aware engine (140) can be used as the system's external context (142).

[0034] The support agent (150) is software, hardware, and / or a combination of software and hardware, configured to handle internal system context (122) and / or external system context (142) to coordinate remote assistance in response to a support request (198) received from a user. The support agent (150) includes a set of machine-readable instructions (stored on a computer-readable medium) that, when executed by a computing device, are executed. Figure 5 One or more operations are described in the flowchart. The support agent (150) can be implemented on the computing device of the computer-aided system (110), such as the one described below. Figure 2B The computing system described. Alternatively, in one embodiment, the support agent (150) is hosted in a cloud environment. Thus, a data network (not shown) can establish a connection between the system awareness engine (120), the environment awareness engine (140), and the support agent (150).

[0035] In one embodiment, the support agent (150) includes support personnel identification logic (152). The support personnel identification logic (152) can use an internal system context (122) and / or an external system context (142) to identify appropriate support personnel from the support team (192A-192N) based on known characteristics of the support personnel in order to respond to support requests. A description of the support personnel identification logic (152) is given below with reference to a flowchart. Furthermore, in Figure 6A and 6B Examples of the logic are provided in the document.

[0036] In one embodiment, the support agent (150) includes conflict resolution logic (154). The conflict resolution logic (154) can prioritize support when multiple competing support requests are received, using the known availability of the system's internal context (122) and / or external context (142) and support personnel (192A-192N). A description of the conflict resolution logic (154) is provided below with reference to the flowchart.

[0037] The support team (190) may include multiple support personnel (192A-192N). Support personnel may have similar or different known characteristics. More specifically, different support personnel may be familiar with different aspects of the computer-aided system (110), different aspects or stages of the procedures being performed by the computer-aided system (110), etc. Some support personnel may be local, while others may be remote. Support personnel may also have different levels or types of training, experience, and / or expertise. Some support personnel may specialize in one or more specific aspects of the computer-aided system (110) or its procedures, while others may be generalists. Some support personnel may have specific licenses or qualifications that qualify them to provide better support on issues than other support personnel. For example, some support personnel may have received specialized training and / or certification. Furthermore, access to some support personnel may be more expensive than to others.

[0038] In one embodiment, it is assumed that the support team (190) includes support personnel (192A-192N) who specialize in providing support for robotic medical systems including computer-assisted systems (110). (See below for further details) Figure 2A , Figure 2B and Figure 3 (Example discussion: robotic medical system). The support team may include personnel such as: field staff, technicians and engineers, robot coordinators, field supervisors, and remote experts performing the procedures. For convenience, the system will be referred to as a medical system in the following discussion, and the same concept can be applied to non-medical computer-aided systems (110).

[0039] A robotic coordinator (not shown) can be located on-site within a healthcare system and can be an employee of the institution owning the system. The robotic coordinator can receive training in the basics of the healthcare system and assist in coordinating its use.

[0040] Field technicians (not shown) may be available near the medical system site and potentially locally at the medical system site. For example, a field technician might be a hospital employee. As another example, a field technician may not be immediately local, but close enough to arrive on-site within hours or days. Field technicians may have limited general training and knowledge and may be able to assist with common, basic technical support issues. Field technicians may have already received training from the medical system manufacturer.

[0041] A site supervisor (not shown) may be able to assist in resolving technical issues that are beyond the scope of knowledge of a typical site technician. Upon request for support, the site supervisor may be able to arrive on-site within a reasonable timeframe and be ready to provide remote support to site technicians or other on-site personnel. The site supervisor may have received training from the medical system manufacturer.

[0042] Remote experts (not shown) can assist with a wide range of challenges. For example, a remote technical expert may be able to assist a healthcare system in solving challenging technical problems. A remote medical expert may be able to assist with challenges in medical procedures, such as complex workflows or other complex issues in an ongoing surgery. Different types of remote technical experts are feasible. For example, some remote experts may assist with broad systems-level knowledge, while others may assist with highly specific problems, such as those related to specific tools, specific procedures being performed by the healthcare system, etc.

[0043] Figure 2A A top view of a computer-aided system (200) is shown in a robotic program scenario. Although in Figure 2A In this context, minimally invasive robotic surgical systems are shown as computer-assisted medical systems (200), but the following description applies to other scenarios and systems, such as non-surgical scenarios or systems, non-medical scenarios or computer-assisted systems, etc.

[0044] In this example, a diagnostic or surgical procedure is performed on a patient (290) lying on an operating table (210). The system may include a user control system (220) for use by an operator (292) (e.g., a clinician such as a surgeon) during the procedure. One or more assistants (294A, 294B, 294C) may also participate in the procedure. The computer-assisted medical system (200) may further include a robotic manipulator system (230) (e.g., a patient-side robotic device) and an assistive system (240). The robotic manipulator system (230) may include at least one manipulator arm (250A, 250B, 250C, 250D), each manipulator arm supporting a detachably coupled tool (260) (also referred to as an instrument (260)). In the illustrated procedure, the tool (260) may enter the patient's (290) body through a natural opening such as the throat or anus or through an incision, while the operator (292) observes the work site (e.g., the surgical site in a surgical setting) through the user control system (220). Images of the work site can be acquired via an imaging device (e.g., an endoscope, optical camera, or ultrasound probe), i.e., a tool (260) for imaging the work site, which can be manipulated by a robotic manipulator system (230) to position and orient the imaging device. An auxiliary system (240) can be used to process the images of the work site for display to the operator (292) via a user control system (220) or another display system located locally or remotely from the procedure. The number of tools (260) used at one time typically depends on the task and space constraints, among other factors. If it is appropriate to change, clean, inspect, or reload one or more tools (260) in use during the procedure, an assistant (294A, 294B, 294C) can remove a tool (260) from the manipulator arm (250A, 250B, 250C, 250D) and replace it with the same tool (260) or another tool (260).

[0045] Figure 2B A schematic diagram (202) of a computer-assisted medical system (200) is provided. The computer-assisted medical system (200) may include one or more computing systems (242). The computing systems may be used to process inputs provided by an operator to a user control system (220). The computing systems may further be used to provide outputs, such as video images, to a display (244). One or more computing systems (242) may further be used to control a robot manipulation system (230).

[0046] The computing system (242) may include one or more computer processors, non-permanent memory (e.g., volatile memory such as random access memory (RAM), cache memory), permanent storage (e.g., hard disk, optical drive (such as optical disc (CD) drive or digital multifunction disk (DVD) drive), flash memory, etc.), communication interfaces (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and many other components and functions.

[0047] The computer processor of the computing system (242) may be an integrated circuit for processing instructions. For example, the computer processor may be one or more cores or microcores of a processor. The computing system (242) may also include one or more input devices, such as a touch screen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device.

[0048] The communication interface of the computing system (242) may include an integrated circuit for connecting the computing system (242) to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) (such as the Internet), a mobile network, or any other type of network) and / or another device, such as another computing system (242).

[0049] Furthermore, the computing system (242) may include one or more output devices, such as display devices (e.g., liquid crystal displays (LCDs), plasma displays, touch screens, organic LED displays (OLEDs), projectors, or other display devices), printers, speakers, external storage, or any other output devices. The one or more output devices may be the same as or different from the input devices(s). Many different types of computing systems exist, and the aforementioned input and output devices(s) may take other forms.

[0050] Software instructions in the form of computer-readable program code for performing embodiments of the present disclosure may be stored, wholly or partially, temporarily or permanently, on a non-transitory computer-readable medium such as a CD, DVD, storage device, floppy disk, magnetic tape, flash memory, physical memory, or any other computer-readable storage medium. Specifically, the software instructions may correspond to computer-readable program code that, when executed by one or more processors, is configured to perform one or more embodiments of the present invention.

[0051] The computing system (242) may be connected to a network or a portion of a network. The network may include multiple nodes. Each node may correspond to a computing system or a group of nodes. As an example, embodiments of this disclosure may be implemented on nodes of a distributed system connected to other nodes. As another example, embodiments of the invention may be implemented on a distributed computing system with multiple nodes, wherein each part of this disclosure may reside on a different node within the distributed computing system. Furthermore, one or more components of the aforementioned computing system may be located in remote locations and connected to other components via a network.

[0052] The robotic manipulation system (230) can use a tool (260) including imaging equipment (e.g., an endoscope or ultrasound probe) to capture images of the work site and output the captured images to the auxiliary system (240). The auxiliary system (240) can process the captured images in various ways before any subsequent display. For example, the auxiliary system (240) can overlay the captured images with a virtual control interface before displaying the combined images to the operator via the user control system (220). The robotic manipulation system (230) can output the captured images for processing outside the auxiliary system (240). One or more separate displays (244) can also be coupled to the computing system (242) and / or the auxiliary system (240) for local and / or remote display of images, such as images of the surgical site or other relevant images.

[0053] An example of a manipulator assembly (300) according to an embodiment of the present disclosure is shown in Figure 3 As shown in the diagram. The manipulator assembly (300) may include a manipulator arm (302) and a tool (320) (also referred to as an instrument (320)). Figure 3 (Only the axis of the tool is shown, not the tool itself). The manipulator arm (302) can correspond to Figure 2B The manipulator arms (250A, 250B, 250C, 250D) are described above. During operation, the manipulator arms (302) typically support the distal instrument or tool (320) and enable movement of the tool (320).

[0054] In minimally invasive settings, the incision location and manipulation tool (320) within the patient's body can be used to keep the kinematically distal center at the incision site, thereby minimizing the size of the incision or the forces applied to the tissue surrounding the incision. More specifically, the elongated shaft of the tool (320) allows the distal end of the end effector and shaft to be inserted distally into the working site via the lumen of a cannula typically inserted through a body wall (such as the abdominal wall). The working site may be inflated. Images of the working site taken by an imaging tool such as an endoscope may include images of the instrument or the distal end of the tool (320) when the tool (320) is positioned within the field of view of the tool operating as an imaging device.

[0055] Since multiple different tools (320) with different end effectors can be sequentially mounted on the manipulator arm (302), or since tools (320) need to be removed and reinstalled during surgery, the distal tool holder facilitates the removal and replacement of mounted instruments or tools.

[0056] For reference Figure 2A Understandably, the manipulator arm (302) is mounted proximally to the base of the robot assembly. Alternatively, the manipulator arm (302) can be mounted to a separate base that can be moved independently, for example, by mounting the manipulator arm (302) to a single manipulator arm trolley, by providing the manipulator arm with mounting clamps that allow the manipulator arm (302) to be mounted directly or indirectly to the operating table at different locations. Figure 2A (as shown in the figure). Typically, the manipulator arm (302) includes multiple manipulator arm segments and associated joints extending between a proximal base and a distal tool holder.

[0057] In such Figure 3 In the illustrated embodiment, the manipulator arm includes multiple joints (such as rotary joints J1, J2, J3, J4, and J5, and prismatic joint J6) and links or manipulator arm segments (304, 306, 308, and 310). The joints of the manipulator arm may or may not have redundant degrees of freedom. A manipulator arm with one or more redundant degrees of freedom has multiple joints such that, for a given position and orientation of a portion of the manipulator arm, the multiple joints can be driven into a series of different configurations. For example, a manipulator arm with one or more redundant degrees of freedom may have multiple joints that, for a given position and orientation of the distal portion of the manipulator arm or the end effector, can be driven into a series of different configurations. For example, Figure 3 The manipulator arm (302) can be manipulated into different configurations, while the distal member (312) supported within the tool holder (310) remains in a specific state and may include a given position or velocity of the end effector. The tool holder (310) may include a cannula (316) through which the tool shaft of the tool (320) extends, and the tool holder (310) may include a bracket ((314), shown as a box-shaped structure), which translates on a spar to which the tool is attached before the tool extends through the cannula (316) to the workpiece.

[0058] Actuation of the tool's (320) degrees of freedom is typically provided by actuators of the manipulator. These actuators may be integrated into the carrier (314). The distal wrist of the tool may allow the end effector of the tool (320) to pivot and / or move linearly about the tool joint axis of one or more joints at the tool wrist. The angle between the end effector gripper elements may be controlled independently of the end effector position and orientation.

[0059] exist Figure 3 In this configuration, when the tool (320) is engaged or mounted on the manipulator arm (302), the shaft extends through the insertion cannula (316). The tool (320) is typically releasably mounted on a tool holder (310) of the manipulator arm (302), which can be driven to translate along a linear guide formed by a prism joint (J6). This may also be referred to as "IO" and provides in / out movement along the insertion axis.

[0060] Although Figure 2A , Figure 2B and Figure 3 Various configurations of the components are shown, but other configurations can be used without departing from the scope of the invention. For example, various components can be combined to create a single component. As another example, the function performed by a single component can be performed by two or more components. Furthermore, while the components are generally described in the context of a surgical scenario, embodiments of this disclosure are applicable to medical scenarios outside of surgery, as well as to other non-medical fields involving robotic manipulation. Moreover, embodiments of this disclosure can relate to different types of robotic systems. For example, while the manipulator arm (302) is rigid, other embodiments may include flexible robotic devices, such as steerable flexible catheters.

[0061] Flowchart of the transition Figure 4A , Figure 4B and Figure 5 A method for coordinating user assistance, according to one or more embodiments, is described. Figure 4A , Figure 4B and Figure 5 One or more steps in the process can be referenced from previous steps. Figure 1 , Figure 2A , Figure 2B and Figure 3 The various components of the described system are used to perform this action. Some of the figures depict specific computer-aided medical systems. However, the methods subsequently described are not limited to specific configurations of medical systems. Rather, these methods are applicable to any type of medical system, or more generally, to any type of robotic system.

[0062] Although the steps in these flowcharts are presented and described sequentially, those skilled in the art will understand that some or all of these steps may be performed in a different order, may be combined or omitted, and some or all of these steps may be performed in parallel. Additional steps may be performed further. Furthermore, these steps may be performed actively or passively. For example, according to one or more embodiments of the invention, some steps may be performed using polling or interrupt-driven methods. For instance, according to one or more embodiments of the invention, the determination step may not require a processor to process instructions unless an interrupt is received to indicate that the condition exists. As another example, the determination step can be performed by executing tests, such as checking a data value to test whether the value is consistent with a test condition according to one or more embodiments of the invention. Therefore, the scope of this disclosure should not be considered limited to... Figure 4A , Figure 4B and Figure 5 The specific arrangement of the steps shown.

[0063] Figure 4A and Figure 4B The document describes how to obtain the internal and external context of the system in various embodiments. Figure 5 It describes how to obtain support in response to support requests, taking into account both internal and / or external system contexts.

[0064] Turning Figure 4A This figure illustrates a flowchart describing a method for obtaining the internal context of a system according to one or more embodiments. It can be repeated. Figure 4A This method ensures that the internal context of the system is up-to-date at all times.

[0065] In step 400, the operational status of the medical system is acquired. Acquiring the operational status may include obtaining parameters from the medical system. These parameters may include any information that provides insight into the current operational status of the medical system. For example, many parameters of a robot manipulation system may be collected (see reference below). Figure 2A , Figure 2B and Figure 3 The parameters collected may include, for example, the kinematics of one or more manipulator arms and / or one or more instruments, the type of instrument mounted on one or more manipulator arms, the kinematics of the instrument, the activated control mode, and the docking status of one or more cannulas. Other parameters that may be collected may include hardware serial numbers, firmware versions, installed hardware and / or software modules, and errors in the medical system stored in the error log.

[0066] In step 402, the parameters collected in step 400 characterize various aspects of the system's internal context. In one or more embodiments, the acquired parameters are further processed to derive additional system internal context. For example:

[0067] (i) Based on the current kinematic configuration and the patient’s position (if known), it can be estimated whether the robotic manipulator is currently in use, whether it is in a setup state before or after use, and / or whether the components of the robotic manipulator are close to the patient.

[0068] (ii) For systems using intubation, it can be estimated whether a procedure is in progress (no intubation docking) or whether the robotic manipulator system is being set up or in progress (intubation docking) based on whether the intubation is docked with the manipulator arm. When intubation docking occurs, some systems can be programmed to assume that the robotic manipulator system is in physical contact with and / or interacts with the patient.

[0069] (iii) Based on the installation status (indicating whether the instrument is attached to the manipulator arm) and / or the insertion position along the insertion axis of the instrument, it can be estimated whether the instrument extends into the patient's anatomy.

[0070] (iv) Based on the current type of instrument being used, the type and / or state of the procedure currently being performed can be estimated. For example, while a scalpel may be used during an ongoing medical procedure, a needle driver may be used as the procedure is nearing its end.

[0071] (v) Based on the current configuration of the instrument (e.g., whether the forceps are open or closed), it can be estimated whether the instrument is currently gripping the tissue or interacting with the tissue in a different way.

[0072] (vi) Based on the fact that the robot manipulation system is in or has been in remote operation mode, it can be estimated that the program is in progress.

[0073] In summary, as illustrated in Examples (i)-(v), the operational state of a robot manipulator can be estimated based on parameters obtained from the robot manipulator itself. A state history can be built over time, reflecting the procedures already performed / being executed by the robot manipulator. Those skilled in the art will understand that the estimation of the system's operational state is system-specific and application-specific, and not necessarily universally valid. For example, if, for a medical scenario, the use of a particular forceps is known to be uniquely associated with a very specific step being performed, it is reasonable to predict that the specific step is being performed when the forceps are actually present. The same assumption may not apply to other medical scenarios. For example, in other medical scenarios, the forceps may be used in different steps, or may not be used at all. Examples (i)-(v) are associated with different risks and may require different responses in the event of a request for support. While not specifically discussed, additional information can be derived by analyzing parameters obtained from the medical system without departing from this disclosure.

[0074] Figure 4B A flowchart illustrating a method for obtaining the external context of a system, according to one or more embodiments, is shown. This can be repeated. Figure 4B This method ensures that the external context of the system is up-to-date at all times. Figure 4B The execution of the method can be independent of Figure 4A The execution of the method.

[0075] In step 450, image data of the medical system and the external operating environment of the medical system (such as the space around or near the medical system) are acquired. The acquired image data can be a single image frame or a series of image frames (e.g., video).

[0076] In step 452, the system's external context is determined based on the image data. Digital image processing can be performed to detect elements present in one or more image frames(s). Figure 2A , Figure 2B and Figure 3 In examples of robotic medical systems, detectable elements include, but are not limited to, the presence of a robotic control system, an operating table, a patient on the operating table, a sterile area, other related components (such as an anesthesia cart), and the position of the robotic control system relative to the operating table.

[0077] Elements isolated by image processing characterize various aspects of the system's external context and provide information about the system's operational context derived from observations made outside the system. For example:

[0078] (i) The status of a planned or ongoing procedure can be estimated based on the presence of a patient on the operating table (e.g., patient position status): when no patient is present, the robotic system is not operational, while when a patient is present, the robotic system is more likely to be used, is being used, or has just been used in a surgical procedure. Additional indications of an upcoming or ongoing procedure include detecting a sterile area after the patient has been prepared (e.g., patient preparation status) and / or detecting an incision (e.g., incision formation status).

[0079] (ii) The presence of components of the robotic system within the sterile area suggests that a procedure may be in progress, and precautions may be necessary due to the potential for interaction between the robotic system and the patient.

[0080] (iii) Additional external context of the system can be obtained by identifying the presence of people, such as whether surgeons or other clinicians, assistants, etc., are near the healthcare system. Furthermore, their level of experience (if accessible to the environment-aware engine, for example, by retrieving information about people from a database) can also be considered part of the external context.

[0081] Typically, the external context of a system can include any information that cannot be directly obtained from the robot manipulator itself. As shown in Examples (i)-(iii), the operational state of the robot manipulator can be estimated based on parameters that cannot be directly obtained from the robot manipulator itself but are collected by monitoring the operating environment outside the robot manipulator. Over time, a state history can be built up, reflecting the procedures that have been / are being executed by the robot manipulator. Thus, the external context of the system broadens the range of available contexts for coordinating remote assistance. While the examples above illustrate image processing-based external contexts, other external contexts can be relied upon as alternatives or additions. For example, sensors built into a patient operating table or anesthesia cart can provide additional external context.

[0082] exist Figure 4A and / or Figure 4B After the method is executed, the internal and / or external system contexts become available and support... Figure 5 The execution of the method.

[0083] Turning Figure 5 , Figure 5 A flowchart illustrating a method for initiating support in response to a support request, according to one or more embodiments, is shown. Once a support request is received from a user of the computer-aided system, the steps in the flowchart can be executed. For convenience, the following example applies to the case where the computer-aided system is a medical system.

[0084] In step 500, a support request is received from the user of the healthcare system. The support request can be provided in various ways and / or may include a range of levels of information. The support request can be made by the user by pressing a physical or virtual button on the user interface. The user interface can be part of the healthcare system. When entered via the user interface, the support request may be accompanied by additional information. For example, the user making the support request can select certain checkboxes to clarify the nature of the support request, and / or the user can enter free text to provide additional details. Alternatively, the support request can be made by the healthcare system itself, for example, when the healthcare system anticipates needing assistance. This may occur, for example, when a specific type of error is detected.

[0085] In step 502, a support person can be selected to respond to the support request. In one or more embodiments, the support person is identified based on at least one of the system internal context, the system external context, and the support request itself.

[0086] In one or more embodiments, a set of rules is used to identify support personnel. The rule set can be selected to provide optimal support based on metrics including minimizing risk, minimizing the time required to obtain a solution, minimizing cost, and / or workload. Therefore, the most suitable support personnel are selected based on urgency, risk, and / or required expertise. In other words, the selection performed in step 502 ensures that the selected support personnel are adequately qualified.

[0087] In one or more embodiments, the set of rules includes "if-then" conditional statements that can be arranged in a tree structure. By traversing the tree using the system's internal context, the system's external context, and / or the support request itself, the support personnel responding to the support request can be identified. See below for reference. Figure 6A and 6B This section provides a discussion of rules in the form of conditional statements.

[0088] The following rules are applicable Figure 2A , Figure 2B and Figure 3 Examples of robotic surgical systems are described below. Additional rules are as follows: Figure 6A and 6B The example shown illustrates this. Different rules can be used in other healthcare systems without departing from this disclosure.

[0089] (i) Support requests during healthcare system setup: The setup phase can be estimated based on either internal or external system context. For example, it can be assumed that the healthcare system is in a setup phase where no patient is present, a patient is present but a sterile area is not detected, and / or docking is not in place. During the setup phase, a robotic coordinator may be a qualified support person to provide support for numerous assistance requests. The robotic coordinator may have sufficient understanding to resolve issues that may arise during healthcare system setup. While a robotic coordinator may not necessarily be qualified to advise on scenarios involving potential interactions between the healthcare system and the patient, its expertise may be sufficient during setup phases where no patient is present.

[0090] (ii) Support requests during ongoing procedures: An ongoing procedure can be estimated based on either internal or external system context. For example, when components of a medical system extend into a sterile area and / or when cannulation is performed, it can be assumed that the medical system is in the midst of an ongoing procedure. Remote technicians or specialists can be contacted as support personnel to assist with support requests. Remote specialists may have expertise in handling potential interactions between the medical system and the patient. Similarly, a remote specialist can be contacted when a procedure is likely to begin soon, such as when an incision is made.

[0091] (iii) Support requests made during ongoing procedures due to specific issues: The ongoing procedure can be estimated based on either internal or external system context. For example, when components of a medical system extend into a sterile area and / or when cannulation occurs, it can be assumed that the medical system is in the midst of an ongoing procedure. Furthermore, specific issues can be detected based on error flags identifying the problem and / or by specifying the specific issue to the user when making the support request. For example, suppose the tissue stapler malfunctions during an ongoing surgery. A remote expert with specific expertise (in this case, a remote expert with background knowledge related to tissue stapler issues) can be contacted as a support person to assist with the support request.

[0092] (iv) After a support request is nearing completion or an ongoing procedure is finished: The user who issued the support request can indicate that an ongoing procedure is about to be completed. For example, suppose one of several manipulator arms fails in the absence of other complications. The user can receive advice on completing the procedure using the remaining functional manipulator arms. Contact the field technician for instructions to examine the medical system later.

[0093] In summary, if a problem could cause the program to halt, contact a remote expert, which may require the use of alternative technologies to execute the program. For other issues, contact on-site technicians or the robot coordinator.

[0094] While some of the described estimates can be performed based on either internal or external system context, the estimation may benefit from the availability of both internal and external system contexts. The availability of both internal and external system contexts can better inform the selection of support personnel in various situations. For example, consider a scenario where the internal system context indicates that one or more robotic arms of a robotic surgical system are covered by a curtain. Furthermore, in this scenario, the external system context indicates that the patient is on the operating table, the patient is sedated, and a first incision has been made to place a port for inserting a cannula. Viewed in isolation, the internal system state does not provide sufficient information to select support personnel. For example, an emergency situation might arise if the curtain-covered robotic arm is expected to be used for a surgery scheduled to begin soon. In contrast, a non-emergency situation might arise if the robotic arm is covered by a curtain for a surgical procedure scheduled for the following day, or if the robotic arm remains covered by a curtain after the surgical procedure has been completed.

[0095] Similarly, viewed in isolation, the external context of the system does not provide sufficient information for selecting support personnel. For example, based solely on the presence of a patient with an incision, it is unclear whether the robotic surgical system will be used to perform surgery on that patient. However, the combination of the internal and external contexts can provide insight into whether the robotic surgical system will be used on the patient and whether the procedure is scheduled to begin soon. This more complete picture can be derived by considering the temporal aspects of the internal and / or external contexts. Specifically, in this case, the currently constructed robotic surgical system has not previously been used in any surgery (on another patient). Therefore, based on the system's history over time, it is assumed that the robotic arm is covered by a curtain to perform surgery on the patient. Furthermore, based on the detection of the incision, it is clear that the procedure is imminent. Therefore, any received support requests require immediate attention and must be handled by support personnel with the ability to spontaneously resolve the issues behind the support requests, with minimal delay.

[0096] A program can be divided into several phases, such as the "program in progress" which is considered high priority and the "preceding program in progress" which is considered lower priority. Furthermore, sub-phases can be considered when interpreting the combination of internal and external system contexts. For example, in the scenario described earlier, a sub-phase "preceding program in progress" is introduced. This sub-phase immediately follows the program in progress. Although the sub-phase is part of the "preceding program in progress" phase, it is assigned high priority as mentioned earlier.

[0097] A similar distinction can be made in another scenario where the internal system context indicates that docking has begun but not yet completed, while the external system context indicates that the cannula has been placed. In this case, the internal and external system contexts are otherwise the same as described in the previous case. Since docking is not yet complete, considering only the internal system context would lead to the conclusion that the robotic surgical system is in the "before the procedure in progress" stage. However, based on the placed cannula, a sub-stage is introduced to accommodate support personnel with expertise in issues within the procedure.

[0098] The described scenario illustrates how combining internal and external system contexts allows for finer-grained identification of procedural phases and sub-phases to ensure contact with the appropriate support personnel. Typically, sub-phase identification relies on temporal aspects, which can be acquired over time by gathering internal and / or external system context. The context acquired over time can establish a history of the state(s) of the healthcare system(s) and / or the operating environment surrounding the healthcare system.

[0099] Even without a time component, the combination of internal and external system contexts can further provide convergence information. For example, consider a scenario where the internal system context indicates that one or more robotic arms of a robotic surgical system are covered by a curtain. Furthermore, suppose that the external system context, based on image data, indicates that the curtain has been pulled back. Therefore, before the procedure to be performed, support personnel (e.g., technicians) might be required to replace the damaged curtain.

[0100] In another scenario, suppose the external system context indicates, based on image data, that an instrument is currently inserted into the patient. Furthermore, the internal system context identifies the instruments held by the arms of the robotic surgical system. Given the combination of the internal and external system contexts, an estimate can be generated of which instruments are likely to be inserted.

[0101] In step 504, a test is performed to determine if a support request conflicts with another support request. Because a support agent may receive support requests from different users operating different healthcare systems, a support request may conflict with another support request if both requests identify the same support person. If a conflict is detected, the method can proceed to step 506. If no conflict is detected, the method can proceed directly to step 508.

[0102] In step 506, the conflict between the conflicting support requests is resolved. In one or more embodiments, the conflicting requests are prioritized. Generally, higher priority is assigned to requests involving higher urgency compared to lower urgency. For example, for a system using a cannula, urgency may be higher when the cannula is docked with the manipulator arm because the procedure is more likely to be in progress, while urgency may be lower when setting up a medical system without a cannula docking because the procedure is less likely to be in progress at that moment. Similarly, urgency may be higher when an instrument is currently interacting with tissue (e.g., forceps gripping a blood vessel) compared to when the instrument is in a retracted position. As these examples illustrate, urgency can be assessed based on the type of operation currently being performed, the state or stage of the operation being performed, and / or some other appropriate context. Thus, conflicts can be resolved by evaluating one or more conditional statements that determine the priority of each conflicting request based on the type of operation, the state or stage of the operation being performed, etc. The conflicting requests can then be sorted, delayed, reassigned, or otherwise processed according to their associated priority. Figure 5In the example shown, a higher-priority request can be processed immediately by executing step 508. Lower-priority requests may be shelved for later processing. As another example, a higher-priority request can be reassigned to a different support staff member who may have more experience than needed to resolve the issue, or a lower-priority request can be reassigned to a different support staff member. In summary, the relative priority of competing requests is used to resolve conflicting requests.

[0103] In step 508, a subset of internal and / or external system contexts to be provided to support personnel is identified. In one embodiment, context that may be related to the problem that triggered the support request is identified to be provided to support personnel, while another context is not identified to be provided to support personnel. For example, consider a support request instructing a user to encounter a problem with a tissue suture device. In this case, error or other status flags and historical usage information associated with the control of the tissue suture device or components of the medical system supporting the tissue suture device are identified for provision. Conversely, image data showing the medical system and the operating environment outside the medical system are not identified for provision. It is assumed that the information to help support personnel is thus compiled or synthesized. Additional optimizations may be further performed when transmitting internal and / or external system data. For example, in some scenarios, data locating the medical system in its current configuration and including the operating environment is transmitted. The data may be sent as a point cloud acquired from 3D camera images, which would require more bandwidth.

[0104] Where a portion of the data involves devices with known models, this data can alternatively be sent as an object ID and key information for modeling the device (e.g., current kinematics (joint positions of the manipulator system, type of installed device, etc.)). The object ID identifies components of the medical system, and the kinematics indicates the location and orientation of key components of the manipulator system, enabling the reconstruction of a 3D representation of the medical system on the support staff's computing device. The latter approach typically requires less bandwidth to provide information to the support staff. Furthermore, when the patient is present, the patient's image data can be edited or simplified; this saves transmission bandwidth and also helps ensure patient privacy. For example, the patient might be replaced with a generic body representation, the patient's face might be obscured, and so on.

[0105] In step 510, contact with support personnel is facilitated. When contacting support personnel, a subset of the system's internal and / or external context can be provided to the contacted support personnel. Various communication media can be used to contact support personnel. For example, a telephone call or video call can be initiated, a chat session can be started, etc. The contact with support personnel can be performed by the support agent described earlier, or alternatively by another component or system. The support agent can facilitate contact with support personnel and can further facilitate providing a subset of the system's internal and / or external context to the contacted support personnel.

[0106] In various embodiments, a system for coordinating user assistance based on feedback or other inputs, according to one or more embodiments, is illustrated. For example, in some embodiments, feedback may be obtained directly from the user or support staff, or indirectly from parameters indicating successful support (e.g., call duration, data accessed, suggestions provided, etc.), after a support request has been processed. This feedback can be used to assess whether the support provided successfully resolved the potential problem. Furthermore, feedback on the efficiency level of the support provided can be obtained. In some embodiments, this feedback is used to update support staff identification logic and / or conflict resolution logic. Additional updates may be made to accommodate new healthcare systems, features, procedures, etc. Updates may be performed manually, for example, by an administrator modifying a decision tree consisting of "if-then" conditional statements. Alternatively, if a classifier-type machine learning algorithm is used to perform support staff identification, these machine learning algorithms can be updated in a supervised manner based on historical data. Historical data may be obtained from previously recorded support calls.

[0107] Turning Figure 6A and 6B This figure illustrates an example of a set of rules for identifying appropriate support personnel, according to one or more embodiments. In examples (600A, 600B), a series of conditional statements are used to implement support personnel identification logic. The conditional statements are designed to ensure that a support personnel qualified to handle support requests is determined. In this example, the conditional statements are arranged in a sequence that substantially reflects the workflow of the procedures performed using the medical system. The support personnel are then identified based on their familiarity with the medical system, the steps currently being performed using the medical system, the components currently involved, and / or other aspects, etc.

[0108] In this example, the first test verifies whether the medical system is covered by a curtain. The curtain cover can be detected as an internal system context by sensors in the medical system that detect the attachment of the curtain cover components to the system, or as an external system context by sensors that detect curtains near the medical system (e.g., a camera can detect the presence of a curtain cover on any manipulator arm of the medical system). If a system not yet covered by a curtain does not exist, but the system has data to be used (e.g., through scheduling information, previous user data input from the system, etc.), the medical system can be determined to be in the setup process. At this stage, no patient is present, and no medical procedure is about to begin. Therefore, support personnel A are selected; support personnel A can be a technician trained to handle problems with the medical system when no medical procedure is in progress.

[0109] The second test verifies that all subsystems of the healthcare system are connected. This may include verifying network connectivity between the various components of the healthcare system to ensure they can communicate correctly. If it is found that not all subsystems are connected, support personnel A can be selected. Support personnel A may be able to establish a connection.

[0110] The third test verifies whether the medical system is in a sterile area. For example, the presence of a medical system may require different considerations for various reasons: maintaining sterility within a sterile area may be essential. Furthermore, when the medical system is in a sterile area, a procedure is likely about to begin. The patient has been sedated, and an incision may have already been made. Therefore, it is often desirable to find a solution that allows the procedure to proceed rather than be aborted. When the medical system is not in the sterile area, support staff member A can be selected.

[0111] After completing the first, second, and / or third tests, additional tests may be performed to determine whether the problem requires an expert rather than a generalist. For example, a problem related to a specific device may require expertise in other parts of the device or medical system. If an expert is needed, support person A1 can be selected instead of support person A.

[0112] The fourth test verifies whether the cannula is docked with the manipulator arm of the medical system. When the cannula is docked, the procedure is considered in progress. Similarly, the procedure is considered in progress when an incision has been made or an instrument may have been inserted via the cannula. When the cannula is not yet docked, Support Personnel B can be selected. Compared to Support Personnel A, Support Personnel B needs to be familiar with procedures involving patient presence and the sterility of medical system components. By selecting the knowledgeable Support Personnel B instead of Support Personnel A, the likelihood of successfully resolving potential problems should increase, avoiding procedures that would have to be aborted. Additional tests can be performed to determine if the problem requires an expert rather than a generalist. If an expert is required, Support Personnel B1 can be selected instead of Support Personnel B.

[0113] The fifth test verifies whether, or recently, the device supported by the medical system has been manipulated locally or remotely. In various embodiments, such use or such remote manipulation is an indication that a procedure is in progress. The procedure may not have been performed before the remote manipulation, but the device may have already been inserted into the patient. Therefore, if no remote manipulation is detected, support person C can be selected. Support person C can be trained to handle situations involving devices inserted into the patient. Additional tests may be performed to determine if the problem requires a specialist rather than a generalist. If a specialist is required, support person C1 can be selected instead of support person C.

[0114] The sixth test verifies whether the procedure has been completed. This can be determined, for example, based on the fact that all cannulas have been removed and the medical system has been removed from the sterile area. Support person D can be selected while the procedure is still in progress. Support person D may have been trained to provide assistance during the procedure. Therefore, unlike support persons A, B, and C, who are qualified based on their familiarity with the medical system, support person D is selected based on their familiarity with the procedure being performed. Furthermore, support person D is usually also familiar with the medical system. Additional tests can be performed to determine if the problem requires a specialist rather than a generalist. If a specialist is needed, support person D1 can be selected instead of support person D. When the procedure is completed, support person N can be selected. Support person N may have general background knowledge in assessing the reported problem. The problem may be addressed later. Additional tests can be performed to determine if the problem requires a specialist rather than a generalist. If a specialist is needed, support person N1 can be selected instead of support person N.

[0115] Those skilled in the art will understand that the configuration supporting person identification logic depends on many factors. Therefore, this disclosure is not limited to... Figure 6A and 6B The example shown.

[0116] Although the invention has been described with respect to a limited number of embodiments, those skilled in the art who benefit from this disclosure will understand that other embodiments can be devised without departing from the scope of the invention disclosed herein. Therefore, the scope of the invention should be limited only by the appended claims.

Claims

1. A coordination system for coordinating remote assistance of a medical system configured to perform medical procedures on a patient, the medical system including a manipulator arm having a plurality of joints that enable control over the kinematic configuration of the manipulator arm, the coordination system comprising: System awareness engine, which is executed by a computer processor and configured to: Parameter data is obtained from sensors on the manipulator arm, the parameter data including multiple positions or velocities of the plurality of joints; The current kinematic configuration of the manipulator arm is determined based on the multiple positions or velocities, and The internal context of the medical system is determined, including the operational state of the medical system based on the current kinematics and the patient's position. Support agents, which are executed by a computer processor and configured to: Receive the first support request submitted by a user of the medical system. Based on the operational state within the system's internal context, support personnel identification logic is used to identify the support personnel who responded to the first support request from among multiple support personnel. Facilitate communication with the support staff.

2. The coordination system according to claim 1, in, The first support request is for technical support of the medical system, and The support personnel are the technical support personnel of the medical system.

3. The coordination system according to claim 1, wherein, The support agent is further configured as follows: Before facilitating contact with the support personnel, identify a subset of the system's internal context to be provided to the support personnel, and After facilitating contact with the support personnel, facilitate providing the support personnel with the subset of the system's internal context.

4. The coordination system according to claim 1, wherein: The manipulator arm is configured to support the instrument on the manipulator arm, and wherein, The kinematic configuration refers to the kinematic configuration of the manipulator arm and the device.

5. The coordination system according to claim 1, wherein: The internal context of the system also includes the mounting status of the instrument on the manipulator arm.

6. The coordination system according to claim 1, wherein: The internal context of the system includes the type of instrument used by the manipulator arm.

7. The coordination system according to claim 1, wherein: The manipulator arm is configured to dock a cannula, which can be inserted into the patient's body wall or orifice. The cannula includes an inner lumen configured to receive an instrument, and wherein... The internal context of the system includes the docking status of the cannula.

8. The coordination system according to claim 1, wherein, The medical system includes a robotic manipulation system, and the internal context of the system also includes the error state of the robotic manipulation system.

9. The coordination system according to any one of claims 1-8, in, The support agent is further configured to identify the support personnel by further applying the support personnel identification logic to the external context of the system, and The external context of the system is obtained from an environment perception engine, which is configured to detect information about the operating environment outside the medical system.

10. The coordination system according to claim 9, wherein, The system external context includes at least one state selected from the group consisting of: Patient position and condition Patient readiness status The state of incision formation, and The state of the sterile area in the medical system.

11. The coordination system according to claim 9, further comprising: The environment perception engine is configured to: Receive image data from the operating environment outside the medical system, and The external context of the system is determined based on the image data.

12. The coordination system according to any one of claims 1 to 8, wherein, The support staff identification logic includes multiple rules arranged to qualify support staff based on the stage of a procedure performed using the medical system.

13. The coordination system according to claim 12, wherein, The rules are further arranged to derive convergence information from a combination of the system's internal and external contexts.

14. The coordination system according to claim 12, in, The multiple rules are further configured to qualify support personnel based on sub-phases of the phase, and The selected support personnel are based on the sub-phase.

15. The coordination system according to claim 14, wherein, The sub-stages of the aforementioned stage are identified using temporal information obtained from the history of states selected from a group of the following: The medical system, and The environment outside the medical system.

16. The coordination system according to any one of claims 1 to 8, wherein, The support staff identification logic includes rules that qualify support staff based on their familiarity with the healthcare system.

17. The coordination system according to any one of claims 1 to 8, wherein, The support staff identification logic includes rules that qualify support staff based on their familiarity with the procedures performed using the medical system.

18. The coordination system according to any one of claims 1 to 8, wherein, The support staff identification logic includes rules for qualifying support staff based on expertise in one aspect of the healthcare system.

19. The coordination system according to any one of claims 1 to 8, wherein, The support agent is further configured as follows: Receive a second support request; It was determined that there was a conflict between the second support request and the first support request, which simultaneously required the support personnel; The conflict is resolved using conflict resolution logic in the following manner: Identify the relative priority between the first support request and the second support request; as well as Based on the relative priority, at least one of the first support request and the second support request may be delayed or reallocated.

20. The coordination system according to claim 19, wherein, The conflict resolution logic identifies the relative priority based on the type of operation being performed by the medical system.

21. The coordination system according to claim 19, wherein, The conflict resolution logic identifies the relative priority based on the state or stage of the operation being performed by the medical system.

22. A method of operating a coordination system for coordinating remote assistance in a medical system, wherein the medical system is configured to perform medical procedures on a patient, the medical system including a manipulator arm having a plurality of joints, the plurality of joints enabling control over the kinematic configuration of the manipulator arm, the method comprising: The system perception engine obtains parameter data from the sensors of the manipulator arm, the parameter data including multiple positions or velocities of the plurality of joints; The current kinematic configuration of the manipulator arm is determined based on the multiple positions or velocities, and Determine the internal system context of the medical system, which includes the operational state of the medical system determined based on the current kinematic structure and the patient's position; The support agent receives the first support request submitted by the user of the healthcare system. Based on the operational state within the system's internal context, support personnel identification logic identifies the support personnel who responded to the first support request from among multiple support personnel. Facilitate communication with the support staff.

23. The method of claim 22, further comprising: Before facilitating contact with the support personnel, identify a subset of the system's internal context to be provided to the support personnel, and After facilitating contact with the support personnel, facilitate providing the support personnel with the subset of the system's internal context.

24. The method according to claim 22, wherein, The system internal context also includes: The installation status of the instruments in the medical system.

25. The method according to claim 22, wherein, The system internal context also includes: The type of device connected to the medical system.

26. The method according to claim 22, wherein, The system internal context also includes: The docking status of the cannula relative to the medical system.

27. The method according to claim 22, wherein, The system internal context also includes: The error state of the robot control system of the medical system.

28. The method according to any one of claims 22 to 27, further comprising: The supporting agent obtains the external context of the system from the environment-aware engine, which is configured to detect information about the operating environment outside the medical system. as well as The support personnel are identified by further applying the support personnel identification logic to the external context of the system.

29. The method of claim 28, further comprising: The environmental perception engine receives image data of the operating environment outside the medical system; as well as Based on the image data, the external context of the system is determined.

30. The method according to any one of claims 22 to 27, wherein, The support staff identification logic includes multiple rules arranged to qualify support staff based on the stage of a procedure performed using the medical system.

31. The method according to any one of claims 22 to 27, further comprising: The second support request is received by the support agent; It was determined that there was a conflict between the second support request and the first support request, which simultaneously required the support personnel; The conflict is resolved using conflict resolution logic in the following manner: Identify the relative priority between the first support request and the second support request; as well as Based on the relative priority, at least one of the first support request and the second support request may be delayed or reallocated.

32. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions executable by one or more processors associated with a coordination system, wherein the coordination system is configured to coordinate remote assistance of a medical system configured to perform a medical procedure on a patient, the medical system including a manipulator arm having a plurality of joints enabling control of the kinematic configuration of the manipulator arm, the plurality of machine-readable instructions causing the one or more processors to perform a method comprising the following steps: The system perception engine obtains parameter data from the sensors of the manipulator arm, the parameter data including multiple positions or velocities of the plurality of joints; The current kinematic configuration of the manipulator arm is determined based on the multiple positions or velocities, and Determine the internal system context of the medical system, which includes the operational state of the medical system determined based on the current kinematic structure and the patient's position; The support agent receives the first support request submitted by the user of the healthcare system. Based on the operational state within the system's internal context, the support personnel identification logic identifies the support personnel who responded to the first support request from among multiple support personnel. Facilitate communication with the support staff.

33. The non-transitory machine-readable medium according to claim 32, wherein, The method further includes: Before facilitating contact with the support personnel, identify a subset of the system's internal context to be provided to them, and After facilitating contact with the support personnel, the system facilitates providing the support personnel with the subset of the internal context of the system.

34. The non-transitory machine-readable medium according to claim 32, wherein, The system internal context also includes: The installation status of the medical devices in the medical system; or The type of device connected to the medical system; or The docking status of the cannula relative to the medical system.

35. The non-transitory machine-readable medium according to any one of claims 32 to 34, wherein, The method further includes: The supporting agent obtains the system's external context from an environment-aware engine, which is configured to detect information about the operating environment outside the medical system; and The support personnel are identified by further applying the support personnel identification logic to the external context of the system.

36. The non-transitory machine-readable medium according to any one of claims 32 to 34, wherein, The method further includes: The second support request is received by the support agent; It was determined that there was a conflict between the second support request and the first support request, which simultaneously required the support personnel; The conflict is resolved using conflict resolution logic in the following manner: Identify the relative priority between the first support request and the second support request; and Based on the relative priority, at least one of the first support request and the second support request may be delayed or reallocated.

Images