Changing operating mode of instrument based on gesture detection

Abstract

Systems and methods for changing an operating mode of an instrument include a computer-aided device configured to obtain visual data associated with the instrument; obtaining kinematic data associated with at least one of the instrument or the structure supporting the instrument; obtaining event data associated with at least one of the structure or the instrument; discriminating a gesture performed via the instrument based on the visual data, the kinematic data, and the event data; and in response to recognizing the gesture, causing the computer-aided device to change from the first operating mode to the second operating mode.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 391,418, filed on July 22, 2022, and entitled “Changing an operating mode of an instrument based on gesture detection,” the subject matter of which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the operation of instruments associated with computer-assisted devices, and more particularly, to techniques for changing an operating mode of instruments associated with computer-assisted devices based on gesture detection. Background Art

[0004] More and more devices are being replaced by computer-assisted electronic devices. This is especially true in industrial, entertainment, educational, and other settings. As a medical example, today's hospitals include large arrays of electronic devices found in operating rooms, interventional rooms, intensive care units, emergency rooms, and the like. For example, glass and mercury thermometers are being replaced by electronic thermometers, IV drip lines now include electronic monitors and flow regulators, and traditional handheld surgical and other medical instruments are being replaced by computer-assisted medical devices.

[0005] These computer-assisted devices can be used to perform operations and / or procedures on materials, such as tissue of a patient. For many computer-assisted devices, an operator, such as a surgeon and / or other medical personnel, can typically manipulate an input device using one or more controls on an operator console. As the operator manipulates various controls at the operator console, commands are relayed from the operator console to the computer-assisted device located in the workspace, where they are used to position and / or actuate one or more end effectors and / or devices supported by the computer-assisted device ( For example , via a tool that can be repositioned by the arm). In this manner, the operator can perform one or more procedures on the material in the workspace using the end effector and / or the tool.

[0006] Each of the one or more end effectors and / or tools may perform multiple functions, have multiple modes of operation, and / or operate according to one or more adjustable parameters. In a medical example, a tissue sealing instrument may operate at different energy modes depending on the needs of the operator and the task at hand. During a procedure, an operator of the computer-assisted device may change the instrument's operating functions, modes, and / or parameters.

[0007] Current methods of facilitating an operator to change the function, mode, and / or parameter of an instrument associated with a computer-assisted device include the use of additional physical inputs and / or graphical user interfaces. For example, a computer-assisted device may include a foot pedal that may be assigned the ability to change the instrument's operating mode. As another example, an operator may select a mode from a graphical user interface on a display. However, these methods make it difficult to add instruments and modes to a computer-assisted device. If an instrument with multiple modes is to be added to a computer-assisted device, in order to facilitate the mode change of the instrument, the computer-assisted device may be modified to include an additional physical input (or a graphical user interface) in the graphical user interface. For example , additional buttons, foot pedals, etc.; adding voice input capability where it did not previously exist) and / or additional options. Any of these options would require modification of the hardware and / or software of the computer-assisted device. These additional inputs and / or options would increase the learning curve for the operator, who would need to adjust to the new inputs and / or options.

[0008] Furthermore, current methods of changing modes, etc., may interrupt the operator's workflow. Specifically, current methods require the operator to operate an input device or user interface that is not part of the procedure workflow, but rather is directed to the input device or user interface to change the ability of the instrument to operate. The operator's attention is diverted from the workflow to the input device or user interface, reducing the operator's situational awareness of the procedure workflow.

[0009] Therefore, it is desirable to improve methods and systems for modifying the operation of instruments associated with computer-assisted devices. In some examples, it may be desirable to provide gesture-based changes in the operating functions, modes, and / or parameters of an instrument to help ensure that the instrument can successfully perform a desired procedure. Summary of the invention

[0010] In accordance with some embodiments, a computer-assisted device includes a structure configured to support an instrument, a memory storing an application, and a processing system. When executing the application, the processing system is configured to obtain kinematic data associated with at least one of the structure or the instrument based on at least the kinematic data, identify a gesture performed via the instrument; and in response to identifying the gesture, cause the computer-assisted device to change from a first operating mode to a second operating mode.

[0011] Consistent with some embodiments, a method includes obtaining kinematic data associated with at least one of an instrument or a structure supporting the instrument; identifying a gesture performed via the instrument based at least on the kinematic data; and causing a change from a first operating mode to a second operating mode in response to identifying the gesture.

[0012] In accordance with some embodiments, a computer-assisted device includes a structure configured to support an instrument, a memory storing an application, and a processing system. When executing the application, the processing system is configured to obtain visual data associated with the instrument; obtain kinematic data associated with at least one of the structure or the instrument; obtain event data associated with at least one of the structure or the instrument; based on the visual data, the kinematic data, and the event data, identify a gesture performed via the instrument; and in response to identifying the gesture, cause the computer-assisted device to change from a first operating mode to a second operating mode.

[0013] Consistent with some embodiments, a method includes obtaining visual data associated with an instrument; obtaining kinematic data associated with at least one of the instrument or a structure supporting the instrument; obtaining event data associated with at least one of the structure or the instrument; based on the visual data, kinematic data, and event data, identifying a gesture performed via the instrument; and in response to identifying the gesture, causing a change from a first operating mode to a second operating mode.

[0014] Consistent with some embodiments, one or more non-transitory machine-readable media include a plurality of machine-readable instructions that, when executed by a processor system associated with a computer-assisted system, are adapted to cause the processor system to perform any of the methods described herein.

[0015] At least one advantage and technical improvement of the disclosed technology relative to the prior art is that, using the disclosed technology, the mode or function of an instrument can be changed without significantly deviating from the ongoing procedure. Therefore, the operator can maintain a high level of situational awareness of the ongoing procedure. Another advantage and technical improvement is that new instruments with multiple modes and / or functions can be added to the computer-assisted device without requiring significant operator-oriented modifications to the computer-assisted device or the user interface. Therefore, the computer-assisted device can be expanded to transparently include new instruments and without requiring a significant learning curve for the operator. These technical advantages provide one or more technical advances relative to the prior art methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order that the manner in which the above-described features of various embodiments may be understood in detail, a more particular description of the inventive concepts briefly summarized above may be obtained by reference to various embodiments, some of which are illustrated in the accompanying drawings. It should be noted, however, that the drawings illustrate only typical embodiments of the inventive concepts and are therefore not to be considered limiting of the scope in any way, and that there are other equally effective embodiments.

[0017] Figure 1 is a simplified diagram of an example that includes a computer-assisted system in accordance with various embodiments.

[0018] Figure 2In more detail, the various embodiments are shown Figure 1 control module.

[0019] Figure 3 In more detail, the various embodiments are shown Figure 2 The mode change module.

[0020] FIG. 4A to FIG. 4C According to some embodiments Figure 3 An example machine learning implementation of a mode change module for .

[0021] Figure 5 is a table showing example active and passive gestures according to some embodiments.

[0022] Figure 6 is a flow chart of method steps for modifying instrument operation according to some embodiments. DETAILED DESCRIPTION

[0023] The present description and drawings illustrate inventive aspects, embodiments, embodiments or modules and should not be considered limiting - the claims define the protected invention. Various mechanical, compositional, structural, electrical and operational changes may be made without departing from the spirit and scope of the present description and claims. In some instances, well-known circuits, structures or techniques are not shown or described in detail so as not to obscure the present invention. Like numbers in two or more figures represent the same or similar elements.

[0024] In this description, the specific details of some embodiments consistent with the present disclosure are described. In order to provide a thorough understanding of the embodiments, many specific details are described. However, it is obvious to those skilled in the art that some embodiments can be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative and not restrictive. Those skilled in the art will recognize that other elements within the scope and spirit of the present disclosure although not specifically described here. In addition, in order to avoid unnecessary repetition, one or more features shown and described in association with an embodiment may be incorporated into other embodiments, unless specifically described otherwise, or if one or more features will make the embodiments inoperative.

[0025] In addition, the terms in this description are not intended to limit the present invention. For example, spatial relative terms, such as "below", "below", "lower", "above", "upper", "near side", "distal side", etc., can be used to describe the relationship between an element or feature as shown in the figure and another element or feature. These spatial relative terms are intended to include different positions (i.e., places) and orientations (i.e., rotation placement) of elements or their operations in addition to the positions and orientations shown in the figures. For example, if the content of one of the accompanying drawings is flipped, the elements described as "below" or "below" in other elements or features will be "above" or "above" in other elements or features. Therefore, the exemplary term "below" can include both the position and orientation of the upper and lower parts. The device can be oriented in other ways (rotated 90 degrees or in other orientations), and the spatial relative descriptors used herein are therefore explained. Similarly, the description of the movement along and around various axes includes various special element positions and orientations. In addition, unless the context indicates otherwise, the singular forms "one", "one piece" and "the" are also intended to include plural forms. Furthermore, the terms "comprises," "comprising," "includes," etc. specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be coupled directly, electrically or mechanically, or they may be coupled indirectly via one or more intermediate components.

[0026] Elements described in detail with reference to one embodiment, implementation, or module may be included in other embodiments, embodiments, or modules where they are not specifically shown or described in any practical circumstances. For example, if an element is described in detail with reference to one embodiment, and the element is not described with reference to a second embodiment, the element may still be claimed to be included in the second embodiment. Therefore, in order to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation, or application may be combined with other embodiments, implementations, or aspects unless otherwise specifically described, unless one or more elements will render an embodiment or implementation inoperative, or unless two or more of the elements provide conflicting functions.

[0027] In some instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0028] The present disclosure describes various elements (such as systems and devices, and parts of systems and devices) in three-dimensional space. As used herein, the term "position" refers to the position of an element or a part of an element in three-dimensional space ( For example, three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an element or portion of an element (three rotational degrees of freedom - e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the multi-degree-of-freedom (DOF) spatial position and / or orientation of a coordinate system of interest attached to a rigid body. In general, a pose may include pose variables for each of the DOFs in the pose. For example, a full 6-DOF pose would include pose variables corresponding to 3 positional DOFs ( For example , x, y and z) and 3 orientation DOF ( For example , roll, pitch, and yaw). A 3-DOF position-only pose will only include the pose variables for the 3 position DOFs. Similarly, a 3-DOF orientation-only pose will only include the pose variables for the 3 rotation DOFs. For example , one, two, four or five) positions are also possible. As used herein, the term "shape" refers to a position or orientation measured along an element. As used herein, and with respect to an element or a portion of an element, For example Device( For example , computer-assisted system or repositionable arm), the term "proximal" refers to the direction along its kinematic chain toward the base of the system or device of the repositionable arm, and the term "distal" refers to the direction along the kinematic chain away from the base.

[0029] Various aspects of the present disclosure are described with reference to computer-assisted systems, which may include systems and devices that are teleoperated, remotely controlled, autonomous, semi-autonomous, manually operated, and the like. Example computer-assisted systems include those that include a robot or robotic device. In addition, aspects of the present disclosure are described in terms of embodiments using a medical system, such as the one commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. However, those skilled in the art will appreciate that the inventive aspects disclosed herein may be embodied and implemented in a variety of ways, including robotic and (if applicable) non-robotic embodiments. The embodiments described in the surgical system are exemplary only and should not be considered to limit the scope of the inventive aspects disclosed herein. For example, the techniques described with reference to surgical instruments and surgical methods can be used in other contexts. Therefore, the instruments, systems and methods described herein can be used for humans, animals, parts of human or animal anatomical structures, industrial systems, general robots or teleoperation systems. As a further example, the instruments, systems and methods described herein can be used for non-medical purposes, including industrial uses, general robot uses, sensing or manipulating non-tissue artifacts, cosmetic improvements, imaging of human or animal anatomical structures, collecting data from human or animal anatomical structures, setting up or removing systems, training medical or non-medical personnel, etc. Additional example applications include procedures for tissues removed from human or animal anatomical structures (with or without returning to human or animal anatomical structures) and procedures for human or animal corpses. In addition, these technologies can also be used for medical or diagnostic procedures that include or do not include surgical aspects.

[0030] System Overview

[0031] Figure 1 is a simplified diagram of an example computer-assisted system 100 according to various embodiments. In some examples, the computer-assisted system 100 is a teleoperated system. In a medical example, the computer-assisted system 100 can be a teleoperated medical system, such as a surgical system. As shown, the computer-assisted system 100 includes a follower device 104, which can be teleoperated by being controlled by one or more pilot devices (also called "pilot input devices" when designed to accept external inputs), described in more detail below. A system including a pilot device and a follower device is called a pilot-follower system, and is sometimes also referred to as a master-slave system. Figure 1 Also shown is a workstation 102 ( For example , console), and in various embodiments, the input system can be of any suitable form and may or may not include the workstation 102.

[0032] exist Figure 1 In the example of FIG. 1 , the workstation 102 includes one or more pilot input devices 106 that are designed to be contacted and manipulated by an operator 108. For example, the workstation 102 may include one or more pilot input devices 106 for use with the hands, head, or some other body part of the operator 108. The pilot input devices 106 in this example are supported by the workstation 102 and may be mechanically grounded. In some embodiments, an ergonomic support 110 ( For example, forearm rest), on which the operator 108 can rest his or her forearm. In some examples, the operator 108 can instruct the follower device 104 by using the leader input device 106 to perform a task at a work site near the follower device 104 during a procedure.

[0033] A display unit 112 is also included in the workstation 102. The display unit 112 can display images for viewing by the operator 108. The display unit 112 can move in different degrees of freedom to accommodate the viewing position of the operator 108, and / or optionally provide control functions as another pilot input device. In the example of the computer-assisted system 100, the displayed image can depict the work site where the operator 108 performs various tasks by manipulating the pilot input device 106 and / or the display unit 112. In some examples, the image displayed by the display unit 112 can be received by the workstation 102 from one or more imaging devices arranged at the work site. In other examples, the image displayed by the display unit 112 can be generated by the display unit 112 (or by a different connected device or system), such as a virtual representation of a tool, work site, or user interface component.

[0034] When using the workstation 102, the operator 108 may sit in a chair or other support in front of the workstation 102, position his or her eyes in front of the display unit 112, manipulate the pilot input device 106, and desirably rest his or her forearms on the ergonomic support 110. In some embodiments, the operator 108 may stand at the workstation or assume other postures, and the position (height, depth, etc.) of the display unit 112 and the pilot input device 106 may be adjusted to accommodate the operator 108.

[0035] In some embodiments, one or more pilot input devices 106 may be ungrounded (an ungrounded pilot input device has no kinematic grounding, such as a pilot input device held by the hand of operator 108, without an attached physical support). Such ungrounded pilot input devices may be used in conjunction with display unit 112. In some embodiments, operator 108 may use display unit 112 positioned near a work site so that operator 108 manually operates instruments at the work site, such as laparoscopic instruments in surgical examples, while viewing images displayed by display unit 112.

[0036] The computer-assisted system 100 also includes a follower device 104 that can be commanded by the workstation 102. In a medical example, the follower device 104 can be located on an operating table ( For example In some medical examples, the workstation is located on an operating table. For example, on or within a patient, simulated patient or model, etc. (not shown). The follower device 104 shown includes a plurality of manipulator arms 120, each manipulator arm 120 configured to be coupled to an instrument assembly 122. The instrument assembly 122 may include, for example, an instrument 126. In various embodiments, examples of instruments 126 include, but are not limited to, sealing instruments, cutting instruments, sealing and cutting instruments, radiofrequency energy delivery instruments, ultrasonic energy delivery instruments, suturing instruments ( For example Suture needles), needle instruments ( For example biopsy needle) or grasping or grasping instrument ( For example 102, actuating the manipulator arm 120, and / or the like. As shown, each instrument assembly 122 is mounted to the distal portion of the corresponding manipulator arm 120. The distal portion of each manipulator arm 120 further includes a cannula mount 124, which is configured to have a cannula (not shown) mounted thereto. When the cannula is mounted to the cannula mount, the shaft of the instrument 126 passes through the cannula and enters a working site, such as a surgical site during a surgical procedure. The force transmission mechanism 130 of the instrument assembly 122 can be connected to an actuation interface assembly 128 of the manipulator arm 120, which includes a drive and / or other mechanism that can be controlled from the workstation 102 to transmit force to the force transmission mechanism 130 to actuate the instrument 126.

[0037] In various embodiments, one or more of the instruments 126 may include an imaging device ( example like , optical camera, hyperspectral camera, ultrasound sensor, endoscope, etc.). For example, one or more of the instruments 126 may be an endoscopic assembly including an imaging device that may provide a captured image of a portion of the working site to be displayed via the display unit 112.

[0038] In some embodiments, the manipulator arm 120 and / or the instrument assembly 122 can be controlled to move and articulate the instrument 126 in response to the operator 108's manipulation of the pilot input device 106 and in this manner "follow" the pilot input device 106 via teleoperation. This enables the operator 108 to perform tasks at the work site using the manipulator arm 120 and / or the instrument assembly 122. The manipulator arm 120 is a computer-assisted device ( For example , an example of a repositionable structure that the follower device 104 may include. In some embodiments, the repositionable structure of the computer-assisted device may include multiple links as rigid members and joints as movable components that can be actuated to cause relative movement between adjacent links. For the surgical example, the operator 108 can guide the follower manipulator arm 120 to move the instrument 126 to perform a surgical procedure at an internal surgical site through a minimally invasive hole or a natural orifice.

[0039] As shown, the control system 140 is disposed outside the workstation 102 and communicates with the workstation 102. In other embodiments, the control system 140 may be disposed in the workstation 102 or the follower device 104. When the operator 108 moves the leader input device 106, sensed spatial information including sensed position and / or orientation information is provided to the control system 140 based on the movement of the leader input device 106. The control system 140 may determine a control signal or provide the control signal to the follower device 104 to control the movement of the manipulator arm 120, the instrument assembly 122, and / or the instrument 126 based on the received information and the operator input. In one embodiment, the control system 140 supports one or more wired communication protocols ( For example , Ethernet, USB, etc.) and / or one or more wireless communication protocols ( For example , Bluetooth, IrDA, HomeRF, IEEE 1102.11, DECT, wireless telemetry, etc.).

[0040] The control system 140 may be implemented on one or more computing systems. The one or more computing systems may be used to control the follower device 104. Additionally, the one or more computing systems may be used to control components of the workstation 102, such as movement of the display unit 112.

[0041] As shown, the control system 140 includes a processor system 150 and a memory 160 storing a control module 170. In some embodiments, the processor system 150 may include one or more processors, a non-persistent storage device ( For example , volatile memory, such as random access memory (RAM), cache memory), persistent storage ( For example , hard disk, optical drive such as a compact disc (CD) drive or a digital versatile disc (DVD) drive, flash memory, floppy disk, flexible disk, magnetic tape, any other magnetic medium, any other optical medium, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), flash EPROM, any other memory chip or cassette, punch card, paper tape, any other physical medium having a pattern of holes, etc.), communication interface ( For example Bluetooth interface, infrared interface, network interface, optical interface, etc.) and many other elements and functions. Non-persistent storage devices and persistent storage devices are examples of non-transitory, tangible machine-readable media that may include executable code that is executed by one or more processors ( For example, processor system 150) when running, one or more processors can execute one or more of the techniques disclosed herein, including the process of method 600 described below. In addition, in some embodiments, the functions of control module 170 can be implemented in any technically feasible software and / or hardware.

[0042] Each of the one or more processors of the processor system 150 can be an integrated circuit for processing instructions. For example, the one or more processors can be one or more cores or micro-cores of a processor, a central processing unit (CPU), a microprocessor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), a tensor processing unit (TPU), etc. The control system 140 can also include one or more input devices, such as a touch screen, a keyboard, a mouse, a microphone, a touch pad, a touch pad, an electronic pen, or any other type of input device. In some embodiments, the one or more input devices are also used to help control the instrument 126.

[0043] The communication interface of the control system 140 may include a computer system for connecting to a network (not shown) ( For example , a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or any other type of network) and / or an integrated circuit of another device (such as another computing system).

[0044] In addition, the control system 140 may include one or more output devices, such as a display device ( For example , a liquid crystal display (LCD), a plasma display, a touch screen, an organic LED display (OLED), a projector or other display device), a printer, a speaker, an external storage device, or any other output device. One or more of the output devices may be the same as or different from the input device. There are many different types of computing systems, and the input and output devices described above may take other forms.

[0045] In some embodiments, the control system 140 can be connected to a network or as part of a network. The network can include multiple nodes. The control system 140 can be implemented on a node or a group of nodes. For example, the control system 140 can be implemented on a node of a distributed system connected to other nodes. By another example, the control system 140 can be implemented on a distributed computing system with multiple nodes, wherein different functions and / or components of the control system 140 can be located on different nodes within the distributed computing system. In addition, one or more elements of the above-mentioned control system 140 can be located at a remote location and connected to other elements via a network.

[0046] Some embodiments may include teleoperating one or more components of a medical system, such as the one commercialized by Intuitive Surgical, Inc. of Sunnyvale, California, USA. Surgical systems. About da The embodiments of the surgical system are merely examples and should not be considered to limit the scope of the features disclosed herein.For example, different types of teleoperation systems as well as non-teleoperation systems having a follower device at the working site can utilize the features described herein.

[0047] In some embodiments, the control system 142 may register ( For example , record) system states and / or events occurring in the computer-aided system 100. As used herein, a system state refers to any of the following: the computer-aided system 100 and / or any of its components ( For example , instrument 126, manipulator arm 120, imaging device), any changes to the status of the computer-assisted system 100 and / or any of its components, the identity and current mode / function of the instrument 126 currently in use, and / or the current parameters under which the computer-assisted system 100 and / or its components are operating ( For example , grip strength level, sealing energy level). As used herein, an event refers to any of the following: any interaction between the computer-assisted system 100 and the work site ( For example , the action of the instrument 126 on the target in the work area, whether or not the instrument 126 is in contact with the object in the work area), the operator ( For example , any action taken by operator 108 on computer-assisted system 100 and / or any of its components ( For example , operator 108 input to computer-assisted system 100), any output of computer-assisted system 100 and / or any component thereof ( For example , transmission between workstation 102, control system 140 and follower device 104). For simplicity and brevity of the present disclosure, both system states and events are collectively referred to as events. In some embodiments, control module 170 generates an event log, registers the event in the event log, and stores the event log in a computer-readable storage medium ( For example , memory 160).

[0048] The instrument 126 includes a proximal end and a distal end. In some embodiments, the instrument 126 may have a flexible body. In some embodiments, the instrument 126 includes, for example, an imaging device ( For example, image capture probes), biopsy instruments, laser ablation fibers, and / or other medical surgical, diagnostic, or therapeutic tools. More generally, the instrument 126 may include end effectors and / or tools for performing tasks. In some embodiments, the tools included in the instrument 126 include end effectors having a single working member (such as a scalpel, a blunt blade, an optical fiber, an electrode, etc.). Other end effectors may include, for example, forceps, graspers, scissors, clip applicators, etc. Other end effectors may also include electrically activated end effectors, such as electrosurgical electrodes, transducers, sensors, etc.

[0049] In some embodiments, instrument 126 may include a For example , container). The sealing instrument can operate according to any technically feasible sealing method or technology, including, for example, bipolar sealing, monopolar sealing, or sequential or simultaneous sealing and cutting. In addition, the sealing instrument can operate at any technically feasible energy level required to perform the operation. In some embodiments, the energy level parameters of the instrument 126 can be configured or otherwise set by the operator 108.

[0050] In some embodiments, the instrument 126 may include a cutting instrument for cutting tissue. More generally, the instrument 126 may include an instrument that can be operated by the operator 108 to perform any appropriate action in a procedure. In a medical context, such actions include, but are not limited to, sealing, cutting, clamping, stapling, applying clips, flushing, suturing, etc.

[0051] In some embodiments, the instrument 126 may include an instrument that can perform different actions according to different modes and / or perform actions according to different methods and / or parameters. For example, a sealing instrument may operate according to a bipolar mode for bipolar sealing or a monopolar mode for monopolar sealing. As another example, a sealing and cutting instrument may include a first mode for sealing and cutting and a second mode for sealing only.

[0052] Figure 2 In more detail, the various embodiments are shown Figure 1 As shown in the figure, the control module 170 includes but is not limited to a visualization module 204, a kinematics module 206, an event recording module 208 and a mode change module 210. The follower device 104 includes the following Figure 1 In addition to the manipulator arm 120, instrument assembly 122, and instrument 126 described above, the workstation 102 further includes an imaging device 202. The workstation 102 includes a precursor input device 106 and a display unit 112, as described above with reference to Figure 1 described.

[0053] The kinematics module 206 receives information about joint positions and / or joint velocities from the pilot input device 106. In some embodiments, the joint positions may be sampled at the control system processing rate. The kinematics module 206 processes the joint positions and velocities and converts them into information related to the pilot input device 106 ( For example The kinematics module 206a transforms the position and velocity of a reference coordinate system associated with the imaging device 202 (the joint space of the leader input device 106) into a corresponding position and velocity of a reference coordinate system associated with the follower device 104. In some embodiments, the reference coordinate system associated with the follower device 104 is a coordinate system associated with the imaging device 202 or a coordinate system associated with an instrument 126 in the field of view of the imaging device 202. In some examples, the kinematics module 206a performs the transformation in any technically feasible manner (e.g., using one or more kinematic models, homogeneous transformations, etc.). In some embodiments, the reference coordinate system of the imaging device 202 may be a reference coordinate system of the operator's 108 eyes. In some embodiments, the kinematics module 206 ensures that the movement of the instrument 126 in the reference coordinate system of the imaging device 202 corresponds to the movement of the leader input device 106 in the reference coordinate system of the operator's 108 eyes. In some embodiments, the kinematics module 206 receives information about the joint positions and / or velocities associated with the follower device 104 ( For example , joint positions and / or velocities of joints in the manipulator arm 120 , the position and / or orientation of the instrument 126 ). The kinematics module 206 provides kinematics data 214 including these positions and / or velocities associated with the follower device 104 to the mode change module 210 .

[0054] In various embodiments, the follower device 104 includes one or more imaging devices 202. The imaging device 202 may be an instrument 126 ( For example , endoscope). Additionally or alternatively, the follower device 104 may include an imaging device 202 positioned to capture a view of the instrument 126 ( For example , a view of any portion of the distal end and nearby working parts of the instrument 126). In some embodiments, the imaging device 202 is a monoscopic or stereoscopic camera, a still or video camera, an endoscope, a hyperspectral device, an infrared or ultrasonic device, an ultrasound device, a fluorescent device, etc. The images captured by one or more imaging devices 202 can be processed by the visualization module 204 for display on the display unit 112. The imaging device 202 can be monospectral or multispectral, for example, capturing image data in one or more of the visible, infrared, and / or ultraviolet spectrums. The visualization module 204 provides visual data 212 including these images captured by the imaging device 202 to the mode change module 210.

[0055] The event recording module 208 records events in the computer-assisted system 100. The event recording module 208 monitors the computer-assisted system 100 and its components ( For example , obtain event information from the workstation 102 and / or the follower device 104, monitor transmissions within the computer-assisted system 100, monitor one or more operating parameters associated with the follower device 104), identify events based on the monitoring, and record the events in an event log, which may be stored in the memory 160. The event recording module 208 provides event data 216 of the event from the event log to the mode change module 210.

[0056] The instrument 126 may perform different actions according to different modes or functions, and / or perform actions according to different methods and / or parameters. In various embodiments, the mode change module 210 may implement a change in the mode, function, method, and / or parameter of the instrument 126 based on the gestures performed by the operator 108 using the instrument 126 during the procedure. The mode change module 210 may obtain data as input from the kinematics module 206, the visualization module 204, and / or the event recording module 208. For example , respectively, visual data 212, kinematic data 214, and / or event data 216). The mode change module 210 processes the acquired input data, identifies a gesture in the motion of the instrument 126 based on the processing, and issues a control signal based on the identified gesture to implement a change in the operating mode of the instrument 126. Further details about the mode change module 210 are described below.

[0057] Figure 3 In more detail, the Figure 2 The mode change module 210 of the embodiment of the present invention is a mode change module 210. The mode change module 210 includes one or more modules configured to process the visual data 212, the kinematic data 214 and / or the event data 216 to identify the gestures performed using the instrument 126 during the procedure. As shown, the mode change module 210 includes, but is not limited to, a visual data module 302, a kinematic data module 304, a kinematic / visual / event (KVE) fusion module 306, an instrument trajectory module 308, a program state module 310, and a gesture recognition module 312. The mode change module 210 can access a gesture database 314 stored in the memory 160. The mode change module 210 can output, but is not limited to, a control signal 316.

[0058] In various embodiments, the mode change module 210 uses machine learning-based techniques to discern gestures performed using the instrument 126 during a procedure. The mode change module 210 analyzes recent data associated with the instrument 126 and the computer-assisted system 100 to determine the trajectory of the instrument 126 and the state of the current procedure in which the trajectory occurs. Based on the trajectory of the instrument 126 and the state of the procedure, the mode change module 210 discerns whether the operator 108 has performed a gesture using the instrument 126. After discerning the gesture performed using the instrument 126, the mode change module 210 may recognize a change in the mode, function, method, and / or parameter of the instrument 126 based on the discerned gesture (hereinafter collectively referred to as a "mode change" for brevity), and issue a signal, command, etc. ( example like , control signal 316) to cause a mode change. The mode change may include, but is not limited to, changing the activation mode or functionality of the instrument 126 from one mode or function to another mode or function, changing the value of an operating parameter associated with the instrument 126 ( example like , amount of grip force, sealing energy level), etc.

[0059] The mode change module 210 receives as input visual data 212, kinematic data 214, and / or event data 216. The visual data 212 includes the image data obtained by the imaging device 202 ( For example , stereo or monoscopic endoscope) captured images ( For example , still images, videos). For example, the images may include a perspective view of the imaging device 202 from an imaging device pointing in the same direction as the instrument 126 ( For example , captured by an endoscope integrated with the instrument 126) and / or images captured from a third person's view relative to the instrument 126. More generally, the visual data 212 includes images capturing the position and orientation of the instrument 126 over time, from which the movement of the instrument 126 ( For example , movement of the distal portion of instrument 126). In some embodiments, the movement of the distal portion of instrument 126 may be performed at a specific frequency ( For example , capturing images at a frame rate of the imaging device 202, and may be captured at a capture frequency ( example like , the sampling rate is the same as the frame rate of the imaging device 202) or different frequencies ( For example , the sampling rate may be higher or lower than the frame rate) to sample images from the captured image set. In some embodiments, the image sampling rate is a predefined rate ( For example In some embodiments, mode change module 210 may obtain visual data 212 directly from imaging device 202 or indirectly via visualization module 204.

[0060] The kinematic data 214 includes information indicating the instrument 126 (e.g., particularly the distal end of the instrument 126) and / or supporting the instrument 126 (e.g., For example , the manipulator arm 120). In some embodiments, the kinematic data 214 may be sampled at the same sampling frequency as the images are sampled from the visual data 212. In some embodiments, the mode change module 210 may obtain the kinematic data 214 from the kinematic module 206.

[0061] The event data 216 includes data indicating events recorded by the event recording module 208, for example, in an event log associated with the computer-assisted system 100 as a whole and / or any of its components. Non-limiting examples of events recorded in the event data 216 include the instrument 126 contacting an object in the work site, the instrument 126 removing that contact, activating or deactivating the imaging device 202, the operator 108 activating an input device ( For example , pedals, levers, buttons, voice input, a graphical user interface on the display unit 112), the current state of the computer-aided system 100, activating an energy delivery mode of the instrument 126, opening or closing the jaws on the end effector on the instrument 126, the operator 108 selecting a mode, etc.

[0062] In some embodiments, the kinematic data 214 and / or event data 216 include inputs from one or more input devices. For example, a touchpad may be used as an input device to control the instrument 126, the input device having a tap and / or other gesture for recording and sequencing the input. For example, the touchpad may be placed in a gesture mode where a first gesture is the first command of a sequence and a second gesture is the second command of the same sequence. For example, a first tap on the touchpad may command a needle to be thrown, and a second tap on the touchpad may command a needle to be reloaded. In some examples, the touchpad includes one or more force and / or pressure sensors that provide analog / variable inputs. In some examples, the touchpad includes one or more touch, force, pressure, and / or presence sensors that provide binary on / off inputs.

[0063] In some embodiments, visual data 212, kinematic data 214, and / or event data 216 are obtained from corresponding raw rates ( For example , image capture rate, kinematic data capture rate, event recording rate, etc.) and subsampled to a predefined rate ( For example , 10Hz).

[0064] The visual data module 302 samples images from the visual data 212 and processes the sampled images ( For example, the last 20 samples) to generate an output. In some embodiments, the processing of the visual data 212 by the visual data module 302 includes generating a data representation of the sampled images and processing the data representation of the sampled images to analyze the temporal relationship between the images. In some examples, the visual data module 302 outputs the intermediate output ( For example , vectors), for processing by other modules within the mode change module 210. In some embodiments, the visual data module 302 uses one or more machine learning techniques ( For example , neural network and model) to generate data representation, analyze data representation and output intermediate output. FIG. 4A to FIG. 4C An example thereof is described.

[0065] In some embodiments, the visual data 212 data includes left and right 3D images from a stereoscopic imaging device, 3D images from multiple imaging devices configured to capture images from different perspectives, and / or 3D depth or intensity maps from an imaging device. In some embodiments, the visual data module 302 can generate a stereoscopic image frame ( For example , left and / or right 3D images) to process the stereoscopic image frames to generate a data representation of the stereoscopic image frames. In some embodiments, the visual data module 302 also generates a data representation of a region of interest (eg, a region of interest around the distal end of the instrument 126) captured in the sampled image. For example , 1D vector). More generally, for a given image, the visual data module 302 generates one or more data representations for analysis, wherein each of the multiple data representations for a given image can point to a different aspect of the given image ( For example , left or right 3D image, a region of interest surrounding the distal end of the instrument 126 captured in the image).

[0066] The kinematic data module 304 samples the kinematic data 214 and processes the sampled kinematic data 214. In some embodiments, the kinematic data module 304 processes the kinematic data 214 including generating a data representation of the kinematic data and analyzing the data representation of the kinematic data to generate an intermediate output. The kinematic data module 304 outputs the intermediate output ( example like , vectors), for processing by other modules within the mode change module 210. In some embodiments, the kinematic data module 304 generates data representations, analyzes the data representations, and uses one or more machine learning techniques ( For example , neural network and model) output intermediate output, the following is combined FIG. 4A to FIG. 4C An example thereof is described.

[0067] In some embodiments, the kinematics data module 304 generates kinematic data ( For example , angle, position, velocity) are concatenated into a 1D vector to generate a data representation for a given sampling time point.

[0068] The instrument trajectory module 308 processes and analyzes the intermediate output generated by the visual data module 302 and / or the kinematic data module 304 to determine the trajectory of the instrument 126 ( For example , the trajectory or path of the distal end of the instrument 126). That is, the instrument trajectory module 308 uses data representing an image of the instrument 126 and / or data representing the position, orientation, velocity, etc. of joints, etc. associated with the follower device 104 to determine the instrument trajectory. Any suitable data representation ( For example , vector, matrix, etc.) to represent the determined trajectory. In some embodiments, the determined trajectory is associated with a determined confidence level. In some embodiments, the instrument trajectory module 308 can determine multiple candidate trajectories with different confidence levels.

[0069] The KVE fusion module 306 acquires and processes the visual data 212 and the kinematic data 214 in a manner similar to the visual data module 302 and the kinematic data module 304, respectively. That is, the KVE fusion module 306 samples the image data and the kinematic data, generates data representations of the sampled image data and the kinematic data, and analyzes the data representations to generate one or more intermediate outputs. In some embodiments, the KVE fusion module 306 samples the data at the same or different rate than the visual data module 302 and / or the kinematic data module 304.

[0070] The KVE fusion module 306 samples an image from the visual data 212 and generates a sampled image ( For example , the last 32 samples) data representation ( For example , 1D vector representation). The KVE fusion module 306 then combines the data representations of the sampled images into a larger data representation ( For example , larger 1D vectors), and analyze the larger data representation to generate intermediate outputs.

[0071] The KVE fusion module 306 samples the kinematic data 214 and generates a data representation of the sampled kinematic data 214 ( For example , 1D vector representation). The KVE fusion module 306 then analyzes the kinematic data representation using one or more techniques to generate one or more intermediate outputs.

[0072] The KVE fusion module 306 also samples the event data 216 and uses one or more techniques ( For example, classification model) processes the sampled event data to generate a data representation of the sampled event data 216 ( For example , 1D vector representation). The KVE fusion module 306 then analyzes the event data representation using one or more techniques to generate one or more intermediate outputs.

[0073] In some embodiments, the intermediate output generated by the KVE fusion module 306 includes the state of the current program determined based on the visual data 212, the kinematic data 214, or the event data 216 using one or more techniques. In some examples, the intermediate output includes a sequence of program states ( For example , in a timeline). That is, the intermediate output can be a time-ordered sequence of program states.

[0074] The program state module 310 receives the intermediate output generated by the KVE fusion module 306 and uses any suitable technique ( For example , weighted or non-weighted voting) processes the intermediate output to determine a program state that is simultaneous with the instrument trajectory. The program state is a determination of the program state when the instrument trajectory occurs. In some embodiments, the determined program state is a sequence of program states. An example of a sequence of program states can be that the operator 108 lifts the identified instrument 126, then moves the instrument 126 to a surface in a work site, and then sweeps the instrument 126 across the surface. In some embodiments, the determined program state is associated with a determined confidence level. In some embodiments, the KVE fusion module 306 determines multiple candidate program states with different confidence levels, and the program state module 310 selects the candidate program state with the highest confidence level as the program state 474. Additionally or alternatively, in some embodiments, the program state module 310 combines candidate program states to determine an aggregated or combined program state 474.

[0075] Gesture recognition module 312 receives the determined trajectory of instrument 126 from instrument trajectory module 308 and receives the determined program state from program state module 310. Gesture recognition module 312 analyzes the determined trajectory of instrument 126 and the determined program state to recognize a gesture. In some embodiments, gesture recognition includes determining one or more gestures in gesture database 314 that match the determined instrument trajectory and the determined program state, or determining that no gesture in gesture database 314 occurred. In some embodiments, the determination of a gesture or no gesture ( For example , a match in gesture database 314) is associated with a determined confidence level. For example, gesture recognition module 312 can determine a plurality of candidate gesture matches and / or no gestures in gesture database 314, each with a confidence level. In some embodiments, gesture database 314 is a database stored in memory 160, or more generally any suitable data repository or structure ( For example , table), and can be constructed in any suitable manner. The gesture database 314 includes a database of gestures, corresponding instrument trajectories and program states, and corresponding control signals.

[0076] Based on the determined gesture or no gesture, the gesture recognition module 312 generates a corresponding control signal 316 or takes no action, respectively. For example , "No gesture" is the determined candidate with the highest confidence level among the candidates that meets the minimum threshold, then the gesture recognition module 312 will ignore the trajectory and program state. That is, the mode change module 210 does not take any action for the determined instrument trajectory and program state ( For example , no mode change occurs). Mode change module 210 continues to sample and process visual data 212, kinematic data 214, and / or event data 216 to determine updated instrument trajectories and program states, and recognize gestures based on the updated instrument trajectories and program states. In some embodiments, mode change module 210 continuously or periodically generates and updates instrument trajectories and program states by sampling and processing visual data 212, kinematic data 214, and / or event data 216 using a rolling time window or rolling window of samples.

[0077] If the gesture recognition module 312 determines that the gesture ( For example , the gesture from the gesture database 314 is a candidate whose confidence level meets the minimum threshold and is the candidate with the highest confidence level among the candidates), then the gesture recognition module 312 generates and transmits the corresponding control signal 316. The gesture recognition module 312 retrieves the definition or specification of the corresponding control signal 316 from the gesture database 314, generates a signal, and outputs the signal for transmission to the follower device 104. The control signal definition or specification identifies the associated mode change and specifies the control signal that commands the follower device 104 to implement the mode change. The follower device 104 changes the mode of the instrument 126 based on the received control signal 316 in response to the received control signal 316. The mode change module 210 can continue to sample and process the visual data 212, the kinematic data 214, and / or the event data 216 to determine an updated instrument trajectory and program state, and recognize gestures based on the updated instrument trajectory and program state. In some embodiments, mode change module 210 continuously or periodically generates and updates instrument trajectories and program states by sampling and processing visual data 212, kinematic data 214, and / or event data 216 using a rolling time window or rolling window of samples.

[0078] In some embodiments, the mode change module 210 and / or the control module 170 prompts the operator 108 for confirmation and / or additional information associated with the mode change before transmitting the control signal 316 to the follower device 104. For example, the mode change module 210 and / or the control module 170 may prompt the operator 108 to confirm the mode change ( For example , via voice prompts, via prompts displayed in a user interface on the display unit 112). The mode change module 210 and / or the control module 170 may also prompt the operator 108 with additional information associated with the mode change ( For example , parameter values). The operator 108 may use any input method suitable for the computer-assisted system 100 ( For example , input buttons, foot pedals, touch screen input, voice input, performing gestures, performing postures with instrument 126) provide input in response to prompts.

[0079] In some embodiments, instrument trajectory module 308 and / or program state module 310 are combined with gesture recognition module 312. That is, gesture recognition module 312 performs the functions of instrument trajectory module 308 and / or program state module 310 described above as well as the functions of gesture recognition module 312 described above.

[0080] In operation, during a procedure, the mode change module 210 samples the visual data 212, the kinematic data 214, and / or the event data 216 and processes the sampled data to discern a gesture being performed using the instrument 126 or to determine that a gesture has not occurred. If the mode change module 210 determines that a gesture has not occurred, the mode change module 210 will take no action regarding changing the mode of the instrument and then sample further data to make an updated determination. If the mode change module 210 discerns a gesture, the mode change module 210 will generate and output a control signal 316 associated with the discerned gesture to effectuate a mode change.

[0081] FIG. 4A to FIG. 4C An example machine learning implementation of the mode change module 210 according to some embodiments is shown. As described above, the mode change module 210 uses one or more machine learning-based techniques. Examples of machine learning techniques that the mode change module 210 can implement include, but are not limited to, neural networks, convolutional neural networks (CNNs), long short-term memory (LSTM), temporal convolutional networks (TCNs), random forests (RFs), support vector machines (SVMs), etc. and associated models. Although FIG. 4A to FIG. 4C A specific machine learning implementation of the mode change module 210 is shown, but it should be understood that other machine learning implementations or combinations thereof are possible.

[0082] Figure 4AThe visual data module 302, the kinematic data module 304, and the instrument trajectory module 308 are further shown in detail. As shown, the visual data module 302 uses a CNN-1 404 and an LSTM-1 408 to process the visual data 212. The CNN-1 404 receives as input a sampled image frame 402 from the visual data 212. The CNN-1 404 processes the image frame 402 to identify the instrument 126 within the image frame. The CNN-1 404 outputs a 1D vector representation 406 of the image frame 402. The 1D vector representation 406 represents the identification of the instrument 126 and its location in the image captured by the imaging device 202 within the time window. In some embodiments, the CNN-1 404 is a Visual Geometry Group (VGG) convolutional neural network ( For example , VGG-16 with 16 convolutional layers).

[0083] LSTM-1 408 receives as input a 1D vector representation 406. LSTM-1 408 processes the 1D vector representation 406 of the image frame 402 with a retention or memory of the previous image frame. Thus, LSTM-1 408 can track the position and corresponding movement of the instrument 126 or part of the instrument 126 within the image frame 402. LSTM-1 408 outputs a vector 410 representing the temporal relationship between the image frames 402.

[0084] As shown, the kinematic data module 304 processes the kinematic data 214 using an LSTM. The kinematic data module 304 cascades 414 the sampled kinematic values ​​412 from the kinematic data 214 to a 1D vector representation 416. In some embodiments, the kinematic values ​​412 are normalized to a predefined scale before being cascaded. LSTM-2 418 receives the 1D vector representation 416 as input. LSTM-2 418 processes the 1D vector representation 416 of the kinematic values ​​412 and the retention or memory of previous kinematic values. LSTM-2 418 outputs a vector 420. In some embodiments, LSTM-2 418 is an attention-based LSTM.

[0085] The instrument trajectory module 308 cascades the vector 410 and the vector 420 into a feature tensor 424. The LSTM-3 426 implemented within the instrument trajectory module 308 receives the feature tensor 424 as input. In some embodiments, the cascade module 422 cascades the vector output by the visual data module 302 and the vector output by the kinematic data module 304 for a given sampling time point into a single feature tensor 424 for the sampling time point. The LSTM-3 426 processes the feature tensor 424 with a retention or memory of the previous feature tensor to determine the instrument trajectory 428. In some embodiments, the LSTM-3 426 also determines a confidence level associated with the instrument trajectory 428. In some embodiments, the LSTM-3 426 determines a plurality of candidate instrument trajectories 428 with corresponding confidence levels. In some embodiments, the LSTM-3 426 is an attention-based LSTM.

[0086] Figure 4B The KVE fusion module 306 is further shown in detail. The KVE fusion module 306 may implement one or more CNNs, one or more TCNs, one or more LSTMs, one or more RFs, and one or more SVMs. The CNN-2 434 receives the sampled image frame 402 as input. The CNN-2 434 processes the image frame 402 to identify the instrument 126 or part of the instrument 126 within the image frame 402, and outputs a 1D vector representation 436. In some embodiments, the CNN-2 434 is a VGG ( For example VGG 16) convolutional neural network. KVE fusion module 306 includes a cascade module 438, which cascades 1D vector representation 436 into vector 440. TCN-1 442 receives vector 440 as input. TCN-1 442 analyzes vector 440 (which corresponds to image frame 402 over time) using causal convolution to generate a first candidate program state 444-1 as output. Candidate program state 444-1 is TCN-1 442's determination of the program state based on vector 440 ( For example , prediction, classification). In some embodiments, TCN-1 442 also outputs a confidence level associated with candidate program state 444-1.

[0087] The KVE fusion module 306 normalizes 446 the sampled kinematic values ​​412 into normalized kinematic values ​​448, which in some embodiments may be represented as a 1D vector. Each of the TCN-2 450 and the LSTM-4 452 receives the normalized kinematic values ​​448 as input. The TCN-2 450 analyzes the normalized kinematic values ​​448 using causal convolution, thereby outputting a candidate program state 444-2. The candidate program state 444-2 is the determination of the program state by the TCN-2 450 based on the normalized kinematic values ​​448 ( For example, prediction). LSTM-4 452 processes normalized kinematic values ​​448 using retention or memory of previous kinematic values, thereby outputting candidate program state 444-3. Candidate program state 444-3 is the determination of the program state by LSTM-4 452 based on kinematic normalized values ​​448 ( For example , prediction, classification). In some embodiments, TCN-2 450 and LSTM-4 452 also output confidence levels associated with candidate program states 444-2 and 444-3, respectively.

[0088] Each of the classification models RF-1 460, RF-2 462, and SVM 464 receives as input the sampled event data 458 from the event data 216. RF-1 460 and RF-2 462 process the sampled event data 458 to classify the sampled event data 458 into candidate program states. Both RF-1 460 and RF-2 462 analyze the sampled event data 458 via a set of randomized decision trees, wherein the difference between RF-1 460 and RF-2 462 is a different number of trees ( For example , 400 trees and 500 trees, respectively). SVM 464 also analyzes sampled event data 458 to classify sampled event data 458 into candidate program states. RF-1 460, RF-2 462, and SVM 464 output candidate program states 444-4, 444-5, and 444-6, respectively.

[0089] Figure 4C The program state module 310 and the gesture recognition module 312 are shown in further detail. Figure 4C In the example, the program state module 310 receives the candidate program states 444-1 to 444-6 as input. The program state module 310 uses the weighted voting technique 472 to analyze the candidate program states 444-1 to 444-6 to determine ( For example , select candidate program state 444, combine candidate program state 444) program state 474. In some embodiments, weighted voting technique 472 applies weighted voting to the confidence level of candidate program state 444, that is, voting is performed based on the confidence level of candidate program state 444.

[0090] Gesture recognition module 312 receives program state 474 and instrument trajectory 428 as input. LSTM-5 476 implemented in gesture recognition module 312 analyzes program state 474 and instrument trajectory 428 together with the retention or memory of previous program state and instrument movement to determine whether program state 474 and instrument trajectory 428 match a gesture in gesture database 314. In some embodiments, LSTM-5 476 classifies program state 474 and instrument trajectory 428 as one or more matching gestures in gesture database 314 with corresponding confidence levels. LSTM-5 476 can also make a no-gesture determination ( For example , if the match in the gesture database 314 does not meet the minimum confidence level threshold). The LSTM-5 476 outputs a gesture / no gesture determination 478 indicating a matching gesture or no gesture determination.

[0091] The control signal module 480 receives the pose / no pose determination 478. If the pose / no pose determination 478 indicates no pose, the control signal module 480 may indicate that no mode change occurs. If the pose / no pose determination 478 includes a matching pose from the pose database 314, the control signal module 4870 retrieves the control signal specification associated with the matching pose from the pose database 314 and generates a corresponding control signal 316.

[0092] In some embodiments, the machine learning networks and models in the module 210 are changed in a sequential training mode ( example like , neural network, etc.). For example, first train the network in the KVM fusion module 306 ( For example CNN-2 434, TCN-1 442, TCN-2 450, LSTM-4 452, RF-1 460, RF-2 462, and SVM 464). These networks are trained to determine based on a training data set corresponding to visual data 212, kinematic data 214, and / or event data 216 ( For example , classification, prediction) program states. Those networks are frozen after training. Then, the other machine learning networks ( For example , CNN-1404, LSTM-1 408, LSTM-2 418, LSTM-3 426, LSTM-5 476) combined with a frozen machine learning network, is also trained using a training data set corresponding to visual data 212, kinematic data 214 and / or event data 216.

[0093] In some embodiments, training is achieved by minimizing the classification cross entropy between the predictions and the ground truth data. The training data is acquired by pre-forming gestures along normal instrument motion during the procedure, using a recording application to collect synchronized images, kinematics, and / or events. The ground truth data can be manually annotated by a human to indicate states, gestures, etc. corresponding to the ground truth data.

[0094] In some embodiments, the gestures are predefined. That is, a set of gestures ( For example , corresponding instrument trajectory and program state), and the mode change module 210 is trained to recognize predefined gestures. In some embodiments, the gesture is defined after the implementation method ( For example , user defined), and the mode change module 210 is retrained to recognize gestures defined after the implementation as well as predefined gestures.

[0095] In some embodiments, the gestures defined for recognition by the mode change module 210 are preferably gestures that are not susceptible to false positives (low false positive rate), false negatives (low false negative rate), and confusion with other gestures (low probability of being misclassified as a different gesture). In addition, in some embodiments, gestures that can be discerned with low latency are preferred.

[0096] In some embodiments, the gestures defined for the mode change module 210 to discern include active and passive gestures. As used herein, an active gesture is a gesture that includes an instrument motion that is not part of a task associated with a program. That is, the instrument motion does not flow naturally within the task and / or does not deviate significantly from the task. In some embodiments, an active gesture includes an instrument motion that is different from the instrument movement that occurs during the execution of a program performed using an instrument. An active gesture can be predefined before an implementation, or can be defined after an implementation.

[0097] As used herein, a passive gesture is a gesture that includes instrument motion that may be part of a task associated with a procedure. That is, the instrument motion flows naturally within the task and / or is at most a minor or negligible deviation from the task. In some embodiments, a passive gesture includes instrument motion that occurs during the performance of a task as part of a procedure performed using an instrument. Passive gestures may also be predefined prior to implementation or defined after implementation. In some embodiments, whether a gesture for a mode / function change should be defined as an active gesture or a passive gesture may depend on the mode / function change and the degree of disruption to the flow of the gesture during the associated procedure. Figure 5 Examples of active and passive gestures are described. Definitions of active and passive gestures and corresponding mode / function changes and events are stored in the gesture database 314.

[0098] In some embodiments, gesture database 314 includes various types of mode / functional changes for various types of instruments. In some examples, gesture database 314 includes mode pairs, where the same instrument motion can switch between a pair of modes, or more generally between a set of two or more modes, depending on the state of the program. Alternatively, different instrument motions correspond to corresponding modes in a set of modes. That is, one instrument motion is associated with one mode, and a different instrument motion is associated with another mode. Other gestures included in gesture database 314 are gestures that signal to control system 140 that the operator wishes to adjust the computer-assisted system 100 ( For example , particularly in the follower device 104). Table 1 below shows examples of mode pairings and parameter or behavior adjustment changes that can be mapped to gestures. However, it should be understood that the following mode pairs and adjustable parameters and adjustments are merely exemplary, and more or less changeable modes, functions, parameters, and behaviors are possible.

[0099] Table 1:

[0100]

[0101] Figure 5 5 is a table 500 showing example active and passive gestures according to some embodiments. In table 500, gestures 502, 504, and 506 are active gestures, and gestures 508 and 510 are passive gestures.

[0102] Gesture 502 includes the operator drawing a square using the instrument 126. That is, the operator operates the pilot device, which causes the instrument 126 to draw a square using the distal end of the instrument 126 during the procedure. The mode change module 210 recognizes the trajectory of the instrument 126 as drawing a square and maps the associated gesture to a mode change. Similarly, gesture 504 includes drawing a triangle and gesture 506 includes drawing the letter Z. Other examples of active gestures include drawing other geometric shapes ( For example , rectangle), letters and / or numbers.

[0103] Gesture 508 includes the operator closing the jaws of the instrument 126 including the clamp and then swinging the instrument 126 in a plane perpendicular to the current view of the imaging device. Gesture 510 includes the operator closing the jaws of the instrument 126 including the clamp and then rotating the instrument 126 along the wrist. Other examples of passive gestures include reaching and / or grasping a needle during a procedure that is expected to include the use of a needle.

[0104] Specific examples of active and passive gesture recognition will now be described for illustrative purposes. The examples described below all relate to medical contexts. It should be understood that these specific examples are exemplary only and are not intended to be limiting.

[0105] In the first example, operator 108 ( For example , surgeon) is controlling an instrument that includes both a simultaneous seal and cut mode and a sequential seal then cut mode. Operator 108 attempts to seal and sever hypovascular fat or connective tissue ( example like , greater omentum, mesentery, etc.) to reach the target anatomy. The operator 108 wants to move quickly and efficiently through the tissue with low risk of bleeding, so the procedure uses a simultaneous sealing and cutting mode. The operator 108 then comes to tissue with high vascularity or large vital blood vessels ( For example , the inferior mesenteric artery during a colectomy procedure), the operator 108 may wish to move cautiously because the risk of bleeding is high and the anatomy is very critical. The operator 108 may wish to perform a double seal or use a banding technique on the large vessel before final cutting. The operator 108 may perform an active posture ( For example , drawing a square with the instrument) to switch the mode to a sequential seal then cut mode, allowing the operator 108 to more precisely and carefully control that portion of the procedure without having to open or switch to a different instrument or otherwise significantly interrupt the surgical flow. When the operator 108 moves back to less critical tissue with lower vascularity or to a portion of the procedure that the operator 108 wants to move quickly, the operator 108 can switch the mode back to a simultaneous seal and cut mode, such as by using an active gesture for mode switching to more sequential cut then seal, or a different active gesture. The operator 108 can switch back and forth between simultaneous and sequential seal and cut modes at multiple instances during a single procedure based on factors such as tissue type, vascularity, criticality of the anatomy, difficulty of the task, etc.

[0106] In the second example, operator 108 ( For example , surgeon) completed the suturing task during the procedure ( For example , closing a gastrostomy enterotomy defect during gastric bypass surgery, installing a patch during a hernia procedure, etc.). When the instrument grasps the needle, the mode switch is triggered to increase the grip force, making actuation of the needle easier and more efficient. The presence of the needle in the instrument jaws acts as a passive posture to activate the increase in grip force. When the needle is no longer grasped by the instrument jaws, the grip force can be restored to standard levels; the jaws release the needle as a passive posture to restore the grip force to standard levels. This allows a change in grip force to optimize the surgical task without having to swap instruments.

[0107] Figure 6 is a flow chart of method steps for modifying instrument operation according to some embodiments. Figure 14, but those skilled in the art will appreciate that any system configured to perform the method steps in any order falls within the scope of the various embodiments. In some embodiments, one or more of steps 602-614 of method 600 may be implemented at least in part in the form of executable code stored on one or more non-transitory tangible machine-readable media, when executed by one or more processors ( For example , one or more processors of the control system 140) is executed, the executable code causes the one or more processors to perform one or more of steps 602-614. In some embodiments, portions of the method 600 are performed by the control module 170 and / or the mode change module 210.

[0108] As shown, method 600 begins at step 602, where visual data associated with an instrument of a computer-aided device is obtained. In some examples, the visual data includes one or more images of an instrument such as instrument 126 captured by an imaging device such as imaging device 202 located in a workspace. In some examples, the one or more images include single-view images, stereo images, and / or a combination of the two. In some examples, the one or more images include images showing the position and / or orientation of the instrument over time. In some examples, one or more images are captured from the perspective of the instrument, including an area of ​​interest surrounding the instrument, etc. In some examples, the one or more images are processed by a visualization module such as visualization module 204 to generate visual data such as visual data 212. In some examples, the visual data includes information related to the temporal relationship between the one or more images.

[0109] At step 604, kinematic data associated with at least one of the instrument or the computer-assisted device is obtained. The kinematic data includes information indicating a position, orientation, velocity, speed, pose, shape, etc. associated with the instrument, such as information about one or more joints, links, arms, etc. of a structure supporting the instrument. In some examples, the kinematic data includes one or more intermediate values ​​generated by a kinematic module, such as the kinematic module 206. In some examples, the kinematic data is synchronized with the visual data obtained during step 602.

[0110] At step 606, event data associated with at least one of the instrument or the computer-assisted device is obtained. The event data includes information indicating an event associated with the computer-assisted system 100 and / or the instrument 126 ( For example , the state of the instrument 126, events occurring at the computer-assisted system 100, etc.). In some embodiments, the event data includes events recorded by a recording module such as the event recording module 208. In some embodiments, the event data is synchronized with the visual data and / or kinematic data obtained in steps 602 and 604, respectively.

[0111] At step 608, based on the obtained visual, kinematic and / or event data, the state of the program executed via the computer-assisted device and the movement of the instrument associated with the program are determined. The visual data and / or kinematic data may be processed to determine the movement trajectory of the instrument ( For example , instrument trajectory 428). In an example, the visual data and / or kinematic data are separately processed by one or more machine learning techniques to generate a data representation of the data. The data representations are combined, and the combination is analyzed by one or more additional machine learning techniques to determine the instrument movement trajectory. Similarly, the visual data, kinematic data, and / or event data are also processed to determine the state of the current program being executed. In an example, the visual data, kinematic data, and event data and / or their data representations are separately processed by multiple machine learning techniques to generate corresponding candidate program states, and the program state is determined from multiple candidate program states. In some embodiments, one or more machine learning techniques and / or one or more additional machine learning techniques are combined with the above-mentioned FIG. 4A to FIG. 4C Those discussed are consistent.

[0112] In step 610, the instrument posture is detected based on the state of the program and the movement of the instrument. The program state and instrument movement trajectory determined in step 608 are processed ( For example , by the gesture recognition module 312) to recognize gestures performed or not performed with the instrument. In the example, the program state and the instrument movement trajectory are processed by one or more machine learning techniques to make the program state and the instrument trajectory consistent with the gesture database ( For example , one or more candidate postures and / or no posture determinations in the posture database 314). In some examples, one or more candidate postures and / or no posture determinations are also determined with corresponding confidence levels. A candidate posture is selected from the one or more candidate postures and / or no posture determinations based on the confidence level.

[0113] At step 612, a change in instrument operating mode from a first mode to a second mode is determined based on the detected instrument gesture.If a candidate gesture is selected at step 610, a corresponding mode change and associated control signal specifications are identified and / or retrieved from a gesture database.

[0114] In step 614, the mode change module 210 causes the instrument to change from the first mode to the second mode. For example , one or more control signals ( For example , control signal 316), and provide one or more control signals to affect the mode change.

[0115] In summary, a computer-assisted system for an instrument can change the mode or function of the instrument based on a gesture performed during a procedure. A gesture includes an instrument action performed by an operator at certain states in a procedure. The computer-assisted system can acquire visual data, kinematic data, and / or event data associated with the instrument and / or the computer-assisted system. The computer-assisted system can process the visual data, kinematic data, and / or event data to predict procedure states and instrument trajectories using machine learning techniques. The computer-assisted system can determine whether a gesture has occurred based on the prediction of the procedure state and instrument trajectory. The computer-assisted system identifies a mode or function change corresponding to the identified gesture and implements the mode or function change ( example like , transmitting a control signal to cause the change).

[0116] At least one advantage and technical improvement of the disclosed technology relative to the prior art is that, using the disclosed technology, the mode or function of an instrument can be changed without significantly deviating from the ongoing procedure. Therefore, the operator can maintain a high level of situational awareness of the ongoing procedure. Another advantage and technical improvement is that new instruments with multiple modes and / or functions can be added to the computer-assisted device without requiring significant operator-oriented modifications to the computer-assisted device or the user interface. Therefore, the computer-assisted device can be expanded to transparently include new instruments and does not require a significant learning curve for the operator. These technical advantages provide one or more technical advances relative to the prior art methods.

[0117] Any and all combinations of any of the claim elements recited in any of the claims and / or any elements described in this application, in any manner, are within the intended scope of this disclosure and protection.

[0118] The description of the various embodiments is presented for purposes of illustration, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0119] Aspects of the present embodiment may be embodied as a system, method, or computer program product. Therefore, aspects of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or an embodiment of a combination of software and hardware aspects, which may generally be referred to as a "module", "system", or "computer" in this article. In addition, any hardware and / or software technology, process, function, component, engine, module, or system described in the present disclosure may be implemented as a circuit or a group of circuits. In addition, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable media, on which a computer-readable program code is embodied.

[0120] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or apparatus, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media would include the following: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, device or apparatus.

[0121] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiments of the present disclosure. It should be understood that each frame of the flowchart and / or block diagram and the combination of frames in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine. When executed by the processor of a computer or other programmable data processing device, the instruction enables the function / action specified in the flowchart and / or block diagram box to be implemented. Such processors can be, but are not limited to, general-purpose processors, special-purpose processors, application-specific processors or field programmable gate arrays.

[0122] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, segment or portion of a code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions recorded in the box may appear outside the order recorded in the figure. For example, two boxes shown in succession can actually be executed substantially concurrently, or the boxes can sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each box of the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can be implemented by a combination of a dedicated hardware-based system or dedicated hardware and computer instructions that performs a specified function or action.

[0123] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, and the scope of the disclosure is determined by the claims that follow.

Claims

1. A computer-assisted device, comprising: A structure configured to support an instrument; Memory for storing applications; as well as A processing system, which when executing the application is configured to: obtaining kinematic data associated with at least one of the structure or the instrument; identifying a gesture performed via the apparatus based at least on the kinematic data; as well as In response to recognizing the gesture, the computer-assisted device is caused to change from a first operating mode to a second operating mode.

2. The apparatus of claim 1, wherein the processing system is further configured to: obtaining visual data associated with the instrument; and The gesture is further identified based on the visual data.

3. An apparatus according to claim 2, wherein the visual data comprises at least one of: one or more images capturing a perspective view of the instrument, one or more images capturing the instrument, or one or more images capturing an area of ​​interest near the instrument.

4. The apparatus of claim 1, wherein the kinematic data comprises at least one of a position, angle, orientation, velocity, or speed of at least one of the structure or the instrument.

5. The apparatus of claim 1, wherein the kinematic data is associated with at least a link, a joint, or an arm of the structure.

6. The apparatus of claim 1, wherein the processing system is further configured to: obtaining event data associated with at least one of the structure or the instrument; and The gesture is further identified based on the event data.

7. An apparatus according to claim 6, wherein the event data includes at least one of: a state of the instrument, a state of the structure, a state of the computer-assisted device, an input to the computer-assisted device, an output of the computer-assisted device, an identification of the instrument, or contact between the instrument and an object.

8. The apparatus of claim 1, wherein the gesture performed via the instrument comprises movement of the instrument.

9. The apparatus of claim 1, wherein the gesture performed via the instrument comprises the gesture performed during a first state in a procedure.

10. The apparatus of claim 1, wherein recognizing the gesture performed via the instrument comprises determining a trajectory of the instrument and a state of a program during which the trajectory of the instrument occurs.

11. The apparatus of claim 10, wherein recognizing the gesture performed via the instrument comprises matching the trajectory of the instrument and the state of the program to a first gesture in a gesture database.

12. The apparatus of claim 1, wherein recognizing the gesture performed via the instrument comprises recognizing the gesture via one or more machine learning networks.

13. An apparatus according to any one of claims 1 to 12, wherein the gesture comprises movement of the instrument other than movement of the instrument that occurs during performance of a procedure performed using the instrument.

14. An apparatus according to any one of claims 1 to 12, wherein the gesture comprises movement of the instrument occurring during performance of a task as part of a procedure performed using the instrument.

15. The apparatus according to any one of claims 1 to 12, wherein the first operating mode and the second operating mode are associated with operation of the instrument.

16. The apparatus of any one of claims 1 to 12, wherein the processing system, when executing the application, is further configured to prompt an operator to confirm the change from the first operating mode to the second operating mode.

17. The apparatus of any one of claims 1 to 12, wherein the processing system, when executing the application, is further configured to prompt an operator to input additional information associated with the change from the first operating mode to the second operating mode.

18. The apparatus of any one of claims 1 to 12, wherein the change from the first operating mode to the second operating mode comprises changing a first value of a parameter associated with the instrument to a second value.

19. The apparatus of any one of claims 1 to 12, wherein the change from the first operating mode to the second operating mode comprises changing a first value of a parameter associated with operation of the instrument to a second value.

20. The apparatus of any one of claims 1 to 12, wherein the change from the first mode of operation to the second mode of operation comprises changing an action performed by the instrument.

21. The apparatus according to any one of claims 1 to 12, wherein the processing system, when executing the application, is further configured to: An instrument trajectory associated with the instrument is determined.

22. The apparatus according to any one of claims 1 to 12, wherein the processing system, when executing the application, is further configured to: A plurality of candidate states for a procedure in which the instrument is used are determined.

23. The apparatus of claim 22, wherein the processing system, when executing the application, is further configured to determine the state of the program based on the plurality of candidate states.

24. The apparatus of any one of claims 1 to 12, wherein the processing system, when executing the application, is further configured to determine a plurality of candidate gestures and a confidence level for each of the plurality of candidate gestures. 25 . The apparatus of claim 24 , wherein the processing system, when executing the application, is further configured to select the first candidate gesture included in the plurality of candidate gestures based on a confidence level of the first candidate gesture.

26. The apparatus according to any one of claims 1 to 12, wherein the processing system, when executing the application, is further configured to: obtaining one or more of second visual data associated with the instrument, second kinematic data associated with at least one of the structure or the instrument, or second event data associated with at least one of the structure or the instrument; determining, based on the one or more of the second visual data, the second kinematic data, or the second event data, that a gesture has not been performed via the instrument; and Responsive to determining that a gesture has not been performed, the second mode of operation is maintained.

27. The apparatus according to any one of claims 1 to 12, wherein the processing system, when executing the application, is further configured to: recognizing a second gesture performed via the apparatus; and In response to recognizing the second gesture, causing the computer-assisted device to change from the second operating mode to the first operating mode.

28. The apparatus of claim 27, wherein the posture and the second posture are the same posture.

29. A method comprising: obtaining, by a processor system, kinematic data associated with at least one of an instrument or a structure of a computer-assisted device supporting the instrument; identifying, by the processor system, a gesture performed via the apparatus based at least on the kinematic data; as well as A change from a first operating mode to a second operating mode is caused by the processor system in response to recognizing the gesture.

30. The method of claim 29, further comprising: obtaining, by the processor system, visual data associated with the instrument; as well as The gesture is further discerned by the processor system based on the visual data.

31. The method of claim 30, wherein the visual data comprises at least one of: one or more images capturing a perspective view of the instrument, one or more images capturing the instrument, or one or more images capturing an area of ​​interest near the instrument.

32. The method of claim 29, wherein the kinematic data comprises at least one of a position, angle, orientation, velocity, or speed of at least one of the structure or the instrument.

33. The method of claim 29, wherein the kinematic data is associated with at least a link, a joint, or an arm of the structure.

34. The method of claim 29, wherein the processing system is further configured to: obtaining, by the processor system, event data associated with at least one of the structure or the instrument; and The gesture is recognized, by the processor system, further based on the event data.

35. A method according to claim 34, wherein the event data includes at least one of the following: a state of the instrument, a state of the structure, a state of the computer-assisted device, an input to the computer-assisted device, an output of the computer-assisted device, an identification of the instrument, or contact between the instrument and an object.

36. The method of claim 29, wherein the gesture performed via the instrument comprises movement of the instrument.

37. The method of claim 29, wherein the gesture performed via the instrument comprises the gesture performed during a first state in a procedure.

38. The method of claim 29, wherein discerning the gesture performed via the instrument comprises determining a trajectory of the instrument and a state of a program during which the trajectory of the instrument occurs.

39. The method of claim 38, wherein identifying the gesture performed via the instrument comprises matching the trajectory of the instrument and the state of the program to a first gesture in a gesture database.

40. The method of claim 29, wherein recognizing the gesture performed via the instrument comprises recognizing the gesture via one or more machine learning techniques.

41. The method of claim 29, wherein the gesture comprises movement of the instrument other than movement of the instrument that occurs during performance of a procedure performed using the instrument.

42. The method of claim 29, wherein the gesture comprises movement of the instrument occurring during performance of a task as part of a procedure performed using the instrument.

43. The method of claim 29, wherein the first operating mode and the second operating mode are associated with operation of the instrument.

44. The method of claim 29, further comprising prompting, by the processor system, an operator to confirm the change from the first operating mode to the second operating mode.

45. The method of claim 29, further comprising prompting, by the processor system, an operator to enter additional information associated with the change from the first operating mode to the second operating mode.

46. ​​The method of claim 29, wherein the change from the first operating mode to the second operating mode comprises changing a first value of a parameter associated with operation of the instrument to a second value.

47. The method of claim 29, wherein the change from the first operating mode to the second operating mode comprises changing a first value of a parameter associated with operation of the instrument to a second value.

48. The method of claim 29, wherein the change from the first operating mode to the second operating mode comprises changing an action performed by the instrument.

49. The method of claim 29, wherein identifying the gesture performed via the instrument comprises: An instrument trajectory associated with the instrument is determined.

50. The method of claim 29, wherein identifying the gesture performed via the instrument comprises: A plurality of candidate states for a procedure in which the instrument is used are determined.

51. The method of claim 50, wherein discerning the gesture performed via the instrument further comprises determining a state of the program based on the plurality of candidate states.

52. The method of claim 29, wherein discerning the gesture performed via the instrument comprises determining a plurality of candidate gestures and a confidence level for each of the plurality of candidate gestures.

53. The method of claim 52, wherein discerning the gesture performed via the instrument comprises selecting the first candidate gesture included in the plurality of candidate gestures based on a confidence level of the first candidate gesture.

54. The method of claim 29, further comprising: obtaining, by the processor system, one or more of second visual data associated with the instrument, second kinematic data associated with at least one of the structure or the instrument, or second event data associated with at least one of the structure or the instrument; determining, by the processor system based on the one or more of the second visual data, the second kinematic data, or the second event data, that a gesture has not been performed via the instrument; and In response to determining, by the processor system, that a gesture has not been performed, maintaining the second operating mode.

55. The method of claim 29, further comprising: recognizing, by the processor system, a second gesture performed via the apparatus; as well as A change from the second operating mode to the first operating mode is caused by the processor system in response to recognizing the second gesture.

56. The method of claim 55, wherein the posture and the second posture are the same posture.

57. One or more non-transitory machine-readable media comprising a plurality of machine-readable instructions which, when executed by a processor system associated with a computer-assisted system, are adapted to cause the processor system to perform the method of any one of claims 29 to 56.

Images