User interface and model architecture for knot tying via robotic medical systems

Abstract

A user interface and model architecture for knot tying via robotic medical systems is provided. A system receives a video stream of a medical procedure performed on an anatomical structure via a robotic medical system, identify, based at least in part on the video stream, a type of knot to tie with a suture on the anatomical structure, generate, based on the type of knot, one or more actions configured to cause one or more instruments of the robotic medical system to tie the type of knot with the suture on the anatomical structure, and overlay, on a user interface that displays the video stream of the medical procedure, one or more visual indications according to the one or more actions to guide the one or more instruments to tie the type of knot with the suture on the anatomical structure.

Description

USER INTERFACE AND MODEL ARCHITECTURE FOR KNOT TYING VIA ROBOTIC MEDICAL SYSTEMSCROSS-REFERENCES TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63 / 635,300, filed April 17, 2024, which is hereby incorporated by reference herein in its entirety.TECHNICAL FIELD

[0002] The present implementations relate generally to user interfaces, including but not limited to a user interface and model architecture to improve performance of actions executed by a robotic medical system.INTRODUCTION

[0003] Robotic medical systems are increasingly used to perform increasingly complex procedures across a wider array of scenarios. However, it can be technically challenging to control the robotic medical systems to effectively, accurately, and efficiently perform such procedures in the increasingly wide variety of scenarios, without introducing errors, inefficiencies, or excessive energy consumption.SUMMARY

[0004] Aspects of technical solutions of this disclosure are directed to improved control and user interface for a robotic medical system configured to allow a user (e.g., surgeon) of the robotic medical system to perform complex medical procedures, such as one or more knottying operations (e.g., suturing). The user interface can provide one or more outputs indicative of assistance to the user in multiple use modes. For example, the user interface can provide output in a simulation mode associated with a virtual reality (VR) interface, to provide simulated surgeon training of knot tying. For example, the user interface can provide output in a live mode associated with an augmented reality (AR) or mixed reality (MR) interface, to provide assistance to surgeon for knot tying during a given medical procedure. The user interface can be coupled with one or more machine learning models to identify a medical task, and to identify a knot sequence for the medical task. Thus, in at least one aspect of this disclosure, a technical solution for a user interface and model architecture for assisted knot tying is provided.

[0005] A system according to this disclosure can, for example: (i) identify a correct knot to be performed for a given task of a medical procedure; (ii) identify one or more movements or positions of formation of the correct knot for the given task; (iii) provide visual prompts in a stepwise order to assist the user in forming the correct knot; (iv) provide visual indications at various steps to indicate deviation from a knot forming process; or (v) provide force feedback to indicate deviation from a correct force in a knot forming process.

[0006] To do so, aspects of this technical solution can identify a task or phase of a workflow of a medical procedure according to one or more data sets corresponding to the medical procedure. For example, a system according to this disclosure can determine a task or phase based on a machine learning model configured to identify visual features indicative of a given task or a given phase of a given medical procedure. The system can correlate a given knot to a given identified task or given identified phase, and can provide one or more visual indications corresponding to instructions to perform the tying of the knot via one or more instruments (e.g., arms) of a robotic system configured to conduct the medical procedure (e.g., surgical robotic system). For example, a user interface can be presented at a head-up display for a user of the robotic system, and can include one or more stepwise visual instructions identifying at least the correct handedness, wrapping properties, and pulling directions, for a given knot. The system can store a state of a previous knot or knots, and can provide instructions based on previous knot states to present instructions directed to a secure knot or sequence of knots.

[0007] At least one aspect is directed to a system. The system can include one or more processors, coupled with memory. The system can receive a video stream of a medical procedure performed on an anatomical structure via a robotic medical system. The system can identify, based at least in part on the video stream, a type of knot to tie with a suture on the anatomical structure. The system can generate, based on the type of knot, one or more actions configured to cause one or more instruments of the robotic medical system to tie the type of knot with the suture on the anatomical structure. The system can overlay, on a user interface that displays the video stream of the medical procedure, one or more visual indications according to the one or more actions to guide the one or more instruments to tie the type of knot with the suture on the anatomical structure.

[0008] At least one aspect is directed to a method. The method can include receiving a video stream of a medical procedure performed on an anatomical structure via a robotic medical system. The method can include identifying, based at least in part on the video stream, a typeof knot to tie with a suture on the anatomical structure. The method can include generating, based on the type of knot, one or more actions configured to cause one or more instruments of the robotic medical system to tie the type of knot with the suture on the anatomical structure. The method can include overlaying, on a user interface that displays the video stream of the medical procedure, one or more visual indications according to the one or more actions to guide the one or more instruments to tie the type of knot with the suture on the anatomical structure.

[0009] At least one aspect is directed to a non-transitory computer readable medium can include one or more instructions stored thereon and executable by a processor. The processor can receive a video stream of a medical procedure performed on an anatomical structure via a robotic medical system. The processor can identify, based at least in part on the video stream, a type of knot to tie with a suture on the anatomical structure. The processor can generate, based on the type of knot, one or more actions configured to cause one or more instruments of the robotic medical system to tie the type of knot with the suture on the anatomical structure. The processor can overlay, on a user interface that displays the video stream of the medical procedure, one or more visual indications according to the one or more actions to guide the one or more instruments to tie the type of knot with the suture on the anatomical structure.BRIEF DESCRIPTION OF THE FIGURES

[0010] These and other aspects and features of the present implementations are depicted by way of example in the figures discussed herein. Present implementations can be directed to, but are not limited to, examples depicted in the figures discussed herein. Thus, this disclosure is not limited to any figure or portion thereof depicted or referenced herein, or any aspect described herein with respect to any figures depicted or referenced herein.

[0011] FIG. 1 depicts an example system, according to this disclosure.

[0012] FIG. 2 depicts an example computer architecture, according to this disclosure.

[0013] FIG. 3 A depicts an example first knot tying presentation, according to this disclosure.

[0014] FIG. 3B depicts an example second knot tying presentation, according to this disclosure.

[0015] FIG. 3C depicts an example third knot tying presentation, according to this disclosure.

[0016] FIG. 3D depicts an example fourth knot tying presentation, according to this disclosure.

[0017] FIG. 3E depicts an example fifth knot tying presentation, according to this disclosure.

[0018] FIG. 3F depicts an example sixth knot tying presentation, according to this disclosure.

[0019] FIG. 4A depicts an example seventh knot tying presentation, according to this disclosure.

[0020] FIG. 4B depicts an example eighth knot tying presentation, according to this disclosure.

[0021] FIG. 4C depicts an example ninth knot tying presentation, according to this disclosure.

[0022] FIG. 4D depicts an example tenth knot tying presentation, according to this disclosure.

[0023] FIG. 4E depicts an example eleventh knot tying presentation, according to this disclosure.

[0024] FIG. 5A depicts an example first knot tying sequence presentation, according to this disclosure.

[0025] FIG. 5B depicts an example second knot tying sequence presentation, according to this disclosure.

[0026] FIG. 5C depicts an example third knot tying sequence presentation, according to this disclosure.

[0027] FIG. 6A depicts an example first simulated knot tying presentation, according to this disclosure.

[0028] FIG. 6B depicts an example second simulated knot tying presentation, according to this disclosure.

[0029] FIG. 7 depicts an example method of assisted knot tying, according to this disclosure.

[0030] FIG. 8 depicts an example method of assisted knot tying, according to this disclosure.DETAILED DESCRIPTION

[0031] Aspects of this technical solution are described herein with reference to the figures, which are illustrative examples of this technical solution. The figures and examples below are not meant to limit the scope of this technical solution to the present implementations or to asingle implementation, and other implementations in accordance with present implementations are possible, for example, by way of interchange of some or all of the described or illustrated elements. Where certain elements of the present implementations can be partially or fully implemented using known components, only those portions of such known components that are necessary for an understanding of the present implementations are described, and detailed descriptions of other portions of such known components are omitted to not obscure the present implementations. Terms in the specification and claims are to be ascribed no uncommon or special meaning unless explicitly set forth herein. Further, this technical solution and the present implementations encompass present and future known equivalents to the known components referred to herein by way of description, illustration, or example.

[0032] In various aspects, a system according to this disclosure can provide one or more user interfaces directed to knot tying, including but not limited to surgical knot tying in medical environments with a surgical robotic system. A system according to this disclosure can, for example, identify a correct knot to be performed for a given task of a medical procedure. A system according to this disclosure can, for example, identify one or more movements or positions of formation of the correct knot for the given task. A system according to this disclosure can, for example, provide visual prompts in a stepwise order to assist the user in forming the correct knot. A system according to this disclosure can, for example, provide visual indications at various steps to indicate deviation from a knot forming process. A system according to this disclosure can, for example, provide force feedback to indicate deviation from a correct force in a knot forming process. For example, the system can identify a task or phase of a workflow of a medical procedure according to one or more data sets corresponding to the medical procedure. For example, a system according to this disclosure can determine a task or phase based on a machine learning model configured to identify visual features indicative of a given task or a given phase of a given medical procedure. The system can correlate a given knot to a given identified task or given identified phase, and can provide one or more visual indications corresponding to instructions to perform the tying of the knot via one or more instruments (e.g., arms) of a robotic system configured to conduct the medical procedure (e.g., surgical robotic system). For example, a user interface can be presented at a head-up display for a user of the robotic system, and can include one or more stepwise visual instructions identifying at least the correct handedness, wrapping properties, and pulling directions, for a given knot. The system can store a state of a previous knot or knots, and can provideinstructions based on previous knot states to present instructions directed to a secure knot or sequence of knots.

[0033] FIG. 1 depicts an example system, according to this disclosure. As illustrated by way of example in FIG. 1, a system 100 can include at least a network 101, a data processing system 102, a client system 103, and a robotic system 104.

[0034] The network 101 can include any type or form of network. The geographical scope of the network 101 can vary widely and the network 101 can include a body area network (BAN), a personal area network (PAN), a local-area network (LAN), e.g., Intranet, a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The topology of the network101 can be of any form and can include, e.g., any of the following: point-to-point, bus, star, ring, mesh, or tree. The network 101 can include an overlay network which is virtual and sits on top of one or more layers of other networks 101. The network 101 can be of any such network topology as known to those ordinarily skilled in the art capable of supporting the operations described herein. The network 101 can utilize different techniques and layers or stacks of protocols, including, e.g., the Ethernet protocol, the Internet protocol suite (TCP / IP), the ATM (Asynchronous Transfer Mode) technique, the SONET (Synchronous Optical Networking) protocol, or the SD (Synchronous Digital Hierarchy) protocol. The TCP / IP Internet protocol suite can include application layer, transport layer, Internet layer (including, e.g., IPv6), or the link layer. The network 101 can include a type of a broadcast network, a telecommunications network, a data communication network, or a computer network.

[0035] The data processing system 102 can include a physical computer system operatively coupled or coupleable with one or more components of the system 100, either directly or indirectly through an intermediate computing device or system. The data processing system102 can include a virtual computing system, an operating system, and a communication bus to effect communication and processing. The data processing system 102 can include a system processor 110, an interface controller 112, an environment processor 120, a robot state processor 130, a presentation controller 140, and a system memory 150.

[0036] The system processor 110 can execute one or more instructions associated with the system 100. The system processor 110 can include an electronic processor, an integrated circuit, or the like including one or more of digital logic, analog logic, digital sensors, analog sensors, communication buses, volatile memory, nonvolatile memory, and the like. The systemprocessor 110 can include, but is not limited to, at least one microcontroller unit (MCU), microprocessor unit (MPU), central processing unit (CPU), graphics processing unit (GPU), physics processing unit (PPU), embedded controller (EC), or the like. The system processor 110 can include a memory operable to store or storing one or more instructions for operating components of the system processor 110 and operating components operably coupled to the system processor 110. For example, the one or more instructions can include one or more of firmware, software, hardware, operating systems, embedded operating systems. The system processor 110 or the system 100 generally can include one or more communication bus controller to effect communication between the system processor 110 and the other elements of the system 100.

[0037] The interface controller 112 can link the data processing system 102 with one or more of the network 101 and the client system 103, by one or more communication interfaces. A communication interface can include, for example, an application programming interface (“API”) compatible with a particular component of the data processing system 102, or the client system 103. The communication interface can provide a particular communication protocol compatible with a particular component of the data processing system 102 and a particular component of the client system 103. The interface controller 112 can be compatible with particular content objects and can be compatible with particular content delivery systems corresponding to particular content objects, structures of data, types of data, or any combination thereof. For example, the interface controller 112 can be compatible with transmission of text data or binary data structured according to one or more metrics or data of the system memory 150.

[0038] The environment processor 120 can identify one or more characteristics of a medical environment. For example, the environment processor 120 can include a vision architecture. The environment processor 120 can include one or more components or functionalities depicted in FIG. 2, including, for example, an image feature processor 212, suture state processor 214, or patient state processor 216. The environment processor 120 can generate one or more output metrics. For example, the environment processor 120 can include one or more logical or electronic devices including but not limited to integrated circuits, logic gates, flip flops, gate arrays, programmable gate arrays, and the like. One or more electrical, electronic, or like devices, or components associated with the environment processor 120 can also be associated with, integrated with, integrable with, replaced by, supplemented by, complementedby, or the like, the system processor 110 or any component thereof. The output metrics can be indicative of a state of a medical procedure of the medical environment, or any component, person, or object thereof, or any combination thereof. For example, the output metrics can include an activity detection metric that indicates an action being performed by one or more persons in the medical environment. For example, the output metrics can include a reconstruction output that indicates a structure of at least a portion of the medical environment. For example, the reconstruction output can include a three-dimensional model of at least a portion of the medical environment during the medical procedure. For example, the output metrics can include an object detection metric that indicates a state of one or more objects in the medical environment. For example, the environment processor 120 can first identify one or more objects, and can subsequently identify corresponding states for one or more of the identified objects, via one or more of the object detection metrics.

[0039] The robot state processor 130 can process one or more metrics indicative of one or more components of the robotic system 104 with respect to one or more given medical procedures or medical procedures of a given type. The robot state processor 220 can include one or more components or functionalities depicted in FIG. 2, including, for example, an instrument metric processor 222, knot execution processor 224, or knot sequence processor 226. The metrics can correspond to one or more quantitative values of force, displacement, rotation, for example, applied by an instrument or an actuator associated with the instrument at a given time or over a given time period for a given medical procedure. For example, the metrics can be indicative of a forceps instrument closing at a at a given time during a medical procedure. The metrics can correspond to one or more quantitative values applied to an instrument or an actuator associated with the instrument at a given time or over a given time period for a given medical procedure. For example, the metrics can be indicative of force at a forceps instrument hindered from moving freely through contact with tissue, at a given time during a medical procedure.

[0040] The presentation controller 140 can provide instructions to cause the user interface 160 to generate output or receive input according to one or more types of input and output. The presentation controller 230 can include one or more components or functionalities depicted in FIG. 2, including, for example, an overlay presentation processor 232, movement presentation processor 234, or a force feedback processor 236. For example, the presentation controller 140 can provide instructions to cause the user interface 160 to present a graphical user interface via a display located at a client system 103. For example, the presentation controller 140 canprovide instructions to cause the user interface 160 to receive input from a graphical user interface via a capacitive input sensor on a display located at a client system 103. For example, the presentation controller 140 can provide instructions to cause the user interface 160 to output audio via a speaker located in the client system 103. For example, the presentation controller 140 can provide instructions to cause the user interface 160 to receive input audio via a microphone located in the client system 103.

[0041] The system memory 150 can store data associated with the system 100. The system memory 150 can include one or more hardware memory devices to store binary data, digital data, or the like. The system memory 150 can include one or more electrical components, electronic components, programmable electronic components, reprogrammable electronic components, integrated circuits, semiconductor devices, flip flops, arithmetic units, or the like. The system memory 150 can include at least one of a non-volatile memory device, a solid-state memory device, a flash memory device, or aNAND memory device. The system memory 150 can include one or more addressable memory regions disposed on one or more physical memory arrays. A physical memory array can include a NAND gate array disposed on, for example, at least one of a particular semiconductor device, integrated circuit device, and printed circuit board device. The system memory 150 can include a video data 152, robot metrics 154, performance metrics 156, and patient metrics 158. In an aspect, the system memory 150 can correspond to a non-transitory computer readable medium. In an aspect, the non-transitory computer readable medium can include one or more instructions executable by the system processor 110. The system processor 110 can overlay, on the user interface, one or more second visual indications according to the one or more actions, where the one or more second visual indications corresponds to a later action of the one or more actions, the later action is subsequent to a current action of the one or more actions, and the current action corresponds to the one or more visual indications.

[0042] The video data 152 can depict one or more medical procedures from one or more viewpoints associated with corresponding medical procedures. For example, the video data 152 can correspond to still images or frames of video images that depict at least a portion of a medical procedure, medical environment, or patient site from a given viewpoint. For example, the environment processor 120 can identify one or more depictions in an image or across a plurality of images. Each time can, for example, be associated with a given task or phase of a workflow as occurring during that task or phase.

[0043] The robot metrics 154 can be indicative of one or more states of one or more components of the robotic system 104. Components of the robotic system 104 can include, but are not limited to, actuators of the robotic system 104 as discussed herein. For example, the robot metrics 154 can include one or more data points indicative of one or more of an activation state (e.g., activated or deactivated), a position, or orientation of a component of the robotic system 104. For example, the robot metrics 154 can be linked with or correlated with one or more medical procedures, one or more phases of a given medical procedure, or one or more tasks of a given phase of a given medical procedure. For example, a robot metric among the robot metrics 154 can correspond to corresponding positions of one or more actuators of a given arm of the robotic system 104 at a given time or over a given time period. Each time can, for example, be associated with a given task or phase of a workflow as occurring during that task or phase.

[0044] The performance metrics 156 can be indicative of one or more actions during one or more medical procedures. For example, the performance metrics 156 can correspond to objective performance indicators (OPIs) as discussed herein. The patient metrics 158 can be indicative of one or more characteristics of a patient during one or more medical procedures. For example, the patient metrics 158 can indicate various conditions (e.g., diabetes, low blood pressure, blood clotting) or various traits (e.g., age, weight) corresponding to the patient in each medical procedure.

[0045] The client system 103 can include a computing system associated with a database system. For example, the client system 103 can correspond to a cloud system, a server, a distributed remote system, or any combination thereof. For example, the client system 103 can include an operating system to execute a virtual environment. The operating system can include hardware control instructions and program execution instructions. The operating system can include a high-level operating system, a server operating system, an embedded operating system, or a boot loader. The client system 103 can include a user interface 160.

[0046] The user interface 160 can include one or more devices to receive input from a user or to provide output to a user. For example, the user interface 160 can correspond to a display device to provide visual output to a user and one or more or user input devices to receive input from a user. For example, the input devices can include a keyboard, mouse or touch-sensitive panel of the display device, but are not limited thereto. The display device can display at least one or more presentations as discussed herein, and can include an electronic display. Anelectronic display can include, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, or the like. The display device can receive, for example, capacitive or resistive touch input. The display device can be housed at least partially within the client system 103.

[0047] The robotic system 104 can include one or more robotic devices configured to perform one or more actions of a medical procedure (e.g., a surgical procedure). For example, a robotic device can include, but is not limited to a surgical device that can be manipulated by robotic device. For example, a surgical device can include, but is not limited to, a scalpel or a cauterizing tool. The robotic system 104 can include various motors, actuators, or electronic devices whose position or configuration can be modified according to input at one or more robotic interfaces. For example, a robotic interface can include a manipulator with one or more levers, buttons, or grasping controls that can be manipulated by pressure or gestures from one or more hands, arms, fingers, or feet. The robotic system 104 can include a surgeon console in which the surgeon can be positioned (e.g., standing or seated) to operate the robotic system 104. However, the robotic system 104 is not limited to a surgeon console co-located or on-site with the robotic system 104. The robotic system 104 can include an instrument s) 170. The instrument s) 170 can include components of the robotic system 104 that can be moved in response to input by a surgeon at the surgeon console of the robotic device 104. The components can correspond to or include one or more actuators that can each move or otherwise change state to operate one or more of the instruments 170 of the robotic device 104. For example, each of the instruments can include one or more sensors or be associated with one or more sensors to provide haptic feedback from the robotic system 104. For example, the haptic feedback can include one or more data points according to the robot metrics 154, or that can be indicative of the robot metrics 154.

[0048] FIG. 2 depicts an example computer architecture, according to this disclosure. As illustrated by way of example in FIG. 2, a computer architecture 200 can include at least an environment processor 210, a robot state processor 220, and a presentation controller 230. The environment processor 210 can correspond at least partially in one or more of structure and operation to the environment processor 120. The environment processor 210 can include an image feature processor 212, a suture state processor 214, and a patient state processor 216. The robot state processor 220 can correspond at least partially in one or more of structure and operation to the robot state processor 130. The robot state processor 220 can include aninstrument metrics processor 222, a knot execution processor 224, and a knot sequence processor 226. The presentation controller 230 can correspond at least partially in one or more of structure and operation to the presentation controller 140. The presentation controller 230 can include an overlay presentation processor 232, a movement presentation processor 234, and a force feedback processor 236.

[0049] The image feature processor 212 can identify one or more features in depictions in video data as discussed herein. For example, the depictions can include portions of a patient site, one or more medical instruments, or any combination thereof, but are not limited thereto. The video data processor 120 can identify one or more edges, regions, or a structure within an image and associated with the depictions. For example, an edge can correspond to a line in an image that separates two depicted objects. For example, a region can correspond to an area in an image that at least partially corresponds to a depicted object. For example, the image feature processor 212 can execute a machine learning model configured to detect image features, and trained according to one or more of the data of the system memory 150 as discussed herein, but is not limited thereto.

[0050] The suture state processor 214 can identify a state of a suture, and can identify one or more aspects of the suture present in a physical environment or a virtual environment. For example, the suture state processor 214 can identify a state of the suture as tied, partially tied, untied, or any combination thereof. For example, the suture state processor 214 can identify aspects of a suture including one or more ends of a suture, one or more knots in a suture, one or more positions of material of the suture, or any combination thereof. The suture state processor 214 can identify the state or the aspect of the suture according to one or more features identified by the image feature processor 212 that correspond to the suture. For example, the suture state processor 214 can apply the image feature processor 212 to one or more images of a physical or virtual medical environment or camera viewpoint or field of view including the suture. For example, the suture state processor 214 can receive virtual environment data indicative of the aspects of a virtual presentation of the suture in a virtual medical environment.

[0051] The patient state processor 216 can identify a state of a patient site having the suture, and can identify one or more aspects of the patient site present in a physical environment or a virtual environment. For example, the patient state processor 216 can identify a state of the patient site as open, closed, blanched, or damaged (e.g., bleeding or tearing), or any combination thereof. For example, the patient state processor 216 can identify aspects of apatient site including one or more edges of an incision, one or more types of tissue, or any combination thereof. The patient state processor 216 can identify the state or the aspect of the patient site according to one or more features identified by the image feature processor 212 that correspond to the patient site. For example, the patient state processor 216 can apply the image feature processor 212 to one or more images of a physical or virtual medical environment or camera viewpoint or field of view including the patient site. For example, the patient state processor 216 can receive virtual environment data indicative of the aspects of a virtual presentation of the patient site in a virtual medical environment.

[0052] The instrument metrics processor 222 can determine or receive one or more metrics associated with one or more robotics arms or portions of one or more robotics arms of the robotic system 104. For example, the instrument metrics processor 222 can receive robot metrics as discussed herein from the robotic system 104, that are indicative of one or more positions, orientations, poses, or the like, of each of the arms of the robotic system 104. For example, the instrument metrics processor 222 can determine metrics for a pose of a right robotic arm manipulable by a right hand or arm of a surgeon, based on robot telemetry from the right robotic arm of the robotic system 104, or a virtual presentation of the right robotic arm of the robotic system 104. For example, the instrument metrics processor 222 can determine metrics for a pose of a left robotic arm manipulable by a left hand or arm of a surgeon, based on robot telemetry from the left robotic arm of the robotic system 104, or a virtual presentation of the left robotic arm of the robotic system 104.

[0053] The knot execution processor 224 can identify one or more states of one or more robot arms of the robotic system 104 that correspond to tying of a given knot or type of knot with the suture. For example, the knot execution processor 224 can identify a given knot based on a medical procedure, phase of the medical procedure, task of the phase, type of the patient site, one or more of the robot metrics associated with the right robot arm, one or more of the robot metrics associated with the left robot arm, or any combination thereof. For example, the knot execution processor 224 can determine that a one-handed or two-handed knot is the correct knot to be tied to close an incision having a shape, a size on a patient’s body, or a location on a patient’s body, or any combination thereof.

[0054] The knot sequence processor 226 can identify one or more movements of one or more robot arms of the robotic system 104 that result in tying of a given knot or type of knot with the suture. For example, the knot sequence processor 226 can identify one or more movementsin a sequence that result in tying of a knot as identified by the knot execution processor 224. For example, the knot sequence processor 226 can identify a previous or subsequent state of one or more of a position or pose of the right or left robot arm or a component of the right or left robot arm (e.g., one or more grasping digits of the right or left robotic arm). For example, a previous state can correspond to a state of a robot arm that occurs before a current state of the robot arm to execute tying of a given knot. For example, a subsequent state can correspond to a state of a robot arm that occurs after a current state of the robot arm to execute tying of a given knot. For example, the knot sequence processor 226 can include or be linked with a knot sequence memory that includes one or more criteria for executing a sequence of movement to achieve completion of tying of a given type of knot (e.g., right-over left to follow-left-over right).

[0055] The overlay presentation processor 232 can cause a user interface to modify a presentation of one or more portions of the robotic system 104 to identify movements corresponding to execution of tying of a given knot. For example, the overlay presentation processor 232 can generate user interface icon indications (and metrics therefor) indicative of one or more of user interface annotations of case video, highlight correct instrument to grasp correct suture end (e.g., via ML visual overlay), or any combination thereof. In an aspect, the one or more visual indications include an indication of an instrument of the one or more instruments to grasp at least a portion of the suture to tie the type of knot. In an aspect, the indication of the instrument can include an overlay of a portion of a depiction of the instrument in the video stream corresponding to the instrument. In an aspect, the overlay presentation processor 232 can overlay, on the user interface, one or more second visual indications according to the one or more actions. In an aspect, the one or more second visual indications corresponds to a later action of the one or more actions, the later action is subsequent to a current action of the one or more actions, and the current action corresponds to the one or more visual indications.

[0056] The movement presentation processor 234 can cause a user interface to augment a presentation of one or more portions of the robotic system 104 to identify movements corresponding to execution of tying of a given knot. For example, movement presentation processor 234 can present one or more visual indications corresponding to an action to tie a given knot at a given point in a sequence, with respect to a specific (e.g., left or right) robot arm of the robotic system 104. For example, the visual indications can include, but are notlimited to, wrapping direction, number of twists, linear arrows by each instrument indicating pulling direction for that instrument, rotational arrows by an instrument indicating wrapping direction for that instrument, indicators for previous and next movement, user interface error alerts for deviation of a given movement from the instructed movement, or any combination thereof. In an aspect, the one or more visual indications include an indication of a lateral direction of pulling corresponding to at least one of the instruments to tie the type of knot. In an aspect, the one or more visual indications include an indication of a rotation direction of wrapping of at least a portion of the suture on at least one of the instruments to tie the type of knot.

[0057] For example, the movement presentation processor 234 can modify a sequence associated with a knot, based on detection of deviation from a given sequence. For example, the movement presentation processor 234 can, based on one or more image features, determine in real time that a surgeon has committed an error in tying a knot, and can modify one or more subsequent steps to correct the error, including by modifying the given type of knot. For example, the movement presentation processor 234 can detect deviation based on case video analysis of one or more image features, one or more of the robot metrics 154, or any combination thereof. For example, the movement presentation processor 234 can change a knot type responsive to deviation / error, to modify a knot sequence memory corresponding to the given know. In an aspect, the movement presentation processor 234 can detect that one or more movements of one or more of the instruments meet a threshold indicative of deviation from one or more of the actions. In an aspect, the movement presentation processor 234 can detect, based on one or more features of the video stream, that the one or more movements meet the threshold. In an aspect, the movement presentation processor 234 can detect, based on one or more metrics corresponding to the robotic medical system, that the one or more movements meet the threshold. In an aspect, the movement presentation processor 234 can modify, in response to the detection of the deviation, the type of knot in a second type of knot. The movement presentation processor 234 can generate, based on the second type of knot, one or more second actions configured to cause the one or more instruments of the robotic medical system to tie the second type of knot with the suture on the anatomical structure.

[0058] The force feedback processor 236 can provide resistance to one or more manipulators of the robotic system 104 corresponding to resistance provided by one or more of the sutures, or the tissue of the patient site. For example, a robotic system 104 as discussed herein caninclude one or more instruments providing force feedback (e.g., one or more manipulators of a robotic system 104 each respectively corresponding to one or more robot arms. The robotic system 104 can provide force feedback data indicative of force applied to an instrument 170 by a patient site (e.g., patient tissue). For example, the robotic system can provide force feedback indicative of resistance of skin to cutting by a scalpel instrument. The resistance can correspond to a pressure at the instrument prior to or during breaking of the skin by the scalpel instrument. The robotic system 104 can provide force feedback data indicative of force applied by an instrument 170 to a patient site (e.g., patient tissue). For example, the robotic system can provide force feedback indicative of force applied by a scalpel instrument during cutting. The force applied can correspond to a force at a manipulator corresponding to the scalpel instrument prior to or during breaking of the skin by the scalpel instrument. For example, the force applied can be a fraction or a function of the force at the manipulator, normalized or corrected for scale of movement between the manipulator and the scalpel. For example, the scale of movement can be a ratio of 1 to 10 (1 : 10), corresponding to a movement of 1 mm by the scalpel in response to a movement of 1 cm at the manipulator.

[0059] The force feedback processor 236 can generate force feedback at one or more manipulators corresponding to one or more aspects of a medical procedure. The force feedback processor 236 can receive data from the suture state processor 214 indicative of a physical state of the suture, and can generate force feedback corresponding to the physical state of the suture. For example, the force feedback can correspond to a resistance to movement of the suture. For example, the resistance can correspond to a level of friction of the suture with respect to another surface of a suture (e.g., a previously tied knot). For example, the resistance can correspond to a level of friction of the suture with respect to a patient site (e.g., tissue surrounding an incision point). In yet another example, the force feedback can correspond to pulling a suture to cinch a knot down, which may facilitate ensuring knot tightness without damaging associated tissue. Thus, the force feedback processor 236 can generate a force feedback having a magnitude (e.g., in N) corresponding to a property of the suture, the patient site, or both, but is not limited thereto.

[0060] In an aspect, the force feedback can indicate tissue damage. For example, the force feedback processor 236 can determine an indication of force applied to the patient site, and can receive an identification of a type of the patient site (e.g., delicate tissue) or an identification of a property of patient site (e.g., muscle tissue). For example, the force feedback processor236 can determine or obtain a threshold for tissue responsiveness, based on the type of the patient site or the property of the patient site. The threshold for tissue responsiveness can be indicative of a magnitude of force applied to the patient site to cause blanching (e.g., movement of blood away from the point of application of pressure) of the patient site. The magnitude of force can correspond to a minimum level of force, or a range of force magnitudes that cause blanching in tissue having a given type or property. For example, the threshold for tissue responsiveness corresponding to causing blanching can include a range of force magnitudes below a threshold to cause tissue damage. The magnitude of force can vary according to a distance from the point of application of pressure at the patient site. For example, a level of white associated with the patient site according to blanching can decrease along a surface of the tissue as distance from the point of application of pressure increases. The force feedback processor 226 can determine a level of decrease of blanching according to distance, by a linear or nonlinear function indicative of propagation of blanching along the patient site. The propagation can be based on the type or the property of the patient site.

[0061] The threshold for tissue responsiveness can be indicative of a magnitude of force applied to the patient site to cause damage (e.g., breakage of tissue and bruising or bleeding in response to the breakage of the tissue from the point of application of pressure) of the patient site. The magnitude of force can correspond to a minimum level of force, or a range of force magnitudes that cause damage to tissue having a given type or property. The magnitude of force can vary according to a distance from the point of application of pressure at the patient site. For example, a level of red associated with the patient site according to damage can decrease along a surface of the tissue as distance from the point of application of pressure increases. The force feedback processor 236 can determine a level of decrease of damage according to distance, by a linear or nonlinear function indicative of propagation of bleeding or bruising along the patient site. The propagation can be based on the type or the property of the patient site.

[0062] In an aspect, the force feedback can provide an indication to guide or correct movement (e.g., higher resistance in wrong direction). For example, the knot execution processor 224 can identify one or more movements corresponding to execution of a given knot. The instrument metrics processor 222 can identify movement of the instruments 170 (e.g., the robotic arms as discussed herein) in one or more directions or along one or more axes. For example, the force feedback processor 236 can increase force feedback applied to a manipulator, in response to adetermination that a detected movement of an instrument 170 is different than an instructed movement indicated by the knot execution processor 224 for the execution of the given knot. For example, the detected movement causing the increased force feedback can correspond to movement in a linear direction or angular direction different than a linear direction or angular direction indicated by the knot execution processor 224 for the execution of the given knot. For example, the detected movement causing the increased force feedback can correspond to movement at a velocity, acceleration, or jerk greater than a velocity, acceleration, or jerk indicated by the knot execution processor 224 for the execution of the given knot.

[0063] In an aspect, the force feedback can be calibrated based on surgeon skill level metric / OPI (e.g., higher sensitivity for novice surgeon). For example, the robot state processor 130 can obtain one more performance metrics corresponding to a given user (e.g., surgeon) to operate the robotic system 104. For example, the robot state processor 130 can obtain one more performance metrics indicative of a novice, typical, or expert surgeon. The force feedback processor 236 can generate or modify one or more magnitudes of force feedback or one or more thresholds corresponding to force feedback based on the performance indicator for the user. For example, the force feedback processor 236 can lower one or more thresholds corresponding to force feedback, in response to identifying a novice surgeon, to increase potential responsiveness for a surgeon more likely to deviate from an instructed movement. For example, the force feedback processor 236 can increase one or more thresholds corresponding to force feedback, in response to identifying an expert surgeon, to reduce potential interference with an expert surgeon action. For example, the force feedback processor 236 can increase one or more magnitudes of force feedback in response to identifying a novice surgeon, to increase potential responsiveness for a surgeon more likely to deviate from an instructed movement. For example, the force feedback processor 236 can decrease one or more magnitudes of force feedback in response to identifying an expert surgeon, to reduce potential interference with an expert surgeon action. Thus, the force feedback processor 236 can provide a technical improvement at least to calibrate force feedback according to surgeon skill level.

[0064] In an aspect, the force feedback processor 236 can cause the user interface to display an indication of a state of the anatomical structure or site. The state of the anatomical structure can change or be reflected based on, or responsive to, a magnitude of force applied to the anatomical structure by the one or more instruments via the suture. The user interface can modify the presentation according to a magnitude of the force and a given threshold. Forexample, the user interface can cause the patient site to increase white balance according to a magnitude of force, in accordance with a determination the force satisfies the threshold for tissue responsiveness corresponding to causing blanching. For example, the patient tissue can increase color toward white according to a linear or non-linear function, as the magnitude of force applied to the patient site increases. Thus, the user interface can present gradually increasing blanching (e.g., increase in white appearance of an incision being closed by a suture). For example, the user interface can cause the patient site to increase red balance according to a magnitude of force, in accordance with a determination the force satisfies the threshold for tissue responsiveness corresponding to causing damage. For example, the patient tissue can increase color toward red to a linear or non-linear function, as the magnitude of force applied to the patient site increases. Thus, the user interface can present gradually increasing bleeding (e.g., increase in red appearance of an incision being closed by a suture). For example, the patient tissue can increase color toward blue or purple to a linear or non-linear function, as the magnitude of force applied to the patient site increases. Thus, the user interface can present gradually increasing bruising (e.g., increase in blue or purple appearance of an incision being closed by a suture). In an aspect, the force feedback processor 236 can cause the user interface to provide output that can include force feedback to the one or more instruments, the force feedback corresponding to a magnitude of force applied to the anatomical structure or anatomical site by the one or more instruments via the suture.

[0065] FIGs. 3 A-3F, 4A-4E, 5A-5C and 6A-6B are directed to user interface presentations of various medical procedures. FIGs. 3A-3F can correspond to user interface presentations at various times in a sequence from FIG. 3A to FIG. 3F. FIGs. 4A-4E can correspond to user interface presentations at various times in a sequence from FIG. 4A to FIG. 4E. FIGs. 5A-5C can correspond to user interface presentations at various times in a sequence from FIG. 5 A to FIG. 5C. FIGs. 6A-6B can correspond to user interface presentations at various times in a sequence from FIG. 6A to FIG. 6B. For example, the presentations of any of FIGs. 3 A-3F, 4A- 4E, 5A-5C and 6A-6B can depict a physical environment for a surgery performed by a surgeon using the robotic system 104. For example, the presentations of any of FIGs. 3A-3F, 4A-4E, 5A-5C and 6A-6B can depict a virtual environment for a simulated surgery performed by a surgeon using a computing system that simulates at least one aspect of a medical procedure or the robotic system 104.

[0066] FIG. 3 A depicts an example first knot tying presentation, according to this disclosure. As illustrated by way of example in FIG. 3 A, this technical solution can provide arrows on the suture to guide a grasping direction. The knot tying presentation 300 A can include at least an open patient site 302A, a suture 310A, a highlighted right robotic arm 320A, and a left robotic arm 330A. The open patient site 302 A can correspond to an opening at a patient site during a medical procedure.

[0067] For example, the open patient site 302A can be an incision of skin of a patient during a medical procedure, but is not limited thereto. The suture 310A can correspond to a surgical thread that is at least partially disposed through the open patient site 302A. The suture 310A can be depicted in the knot tying presentation 300 A according to video data, one or more annotation or overlays corresponding to the location of the suture 310, or any combination thereof. The suture 310A can include a grasping direction indicator 312A. For example, the arrow of the grasping direction indicator 312A indicates a direction of grasping. For example, the grasping direction from the grip joint to the tip is aligned with the arrow direction, to achieve parallel grasping. In the medical procedure, parallel grasping can increase ease of manipulation of a suture and thus provide a technical improvement to increase strength and durability of knot generation (e.g., compared to orthogonal grasping). This technical improvement is especially pronounced in scenarios as discussed herein by way of example directed to wrapping a suture, but is not limited thereto. The grasping direction indicator 312A can correspond to a visual indication located at or near the suture 310A, that is indicative of a direction of movement of the suture 310A in a direction corresponding to a length of the suture 310A. For example, the grasping direction indicator 312A can indicate that the suture 310A is to be grasp downward along a left side of the open patient site 302 A. The highlighted right robotic arm 320A can correspond to a visual indication having an overlay associated with the grasping direction indicator 312A. For example, the highlighted right robotic arm 320A can be highlighted with a translucent color fill (e.g., green) to indicate that the highlighted right robotic arm 320A is to be used to grasp the suture 310A and complete an action of grasping the suture according to the grasping direction indicated by the grasping direction indicator 312 A. The left robotic arm 330A can be in an idle state during a knot tying action corresponding to the knot tying presentation 300 A.

[0068] FIG. 3B depicts an example second knot tying presentation, according to this disclosure. As illustrated by way of example in FIG. 3B, a second knot tying presentation 300Bcan include at least a suture 310B, a right robotic arm 320B, a left robotic arm 330B, and a handedness indicator 340B. The suture 310B can correspond to a state subsequent to the state of the suture 310A. The right robotic arm 320B can grasp the suture 31 OB and perform a wrapping action according to the visual indications of the knot tying presentation 300B. The left robotic arm 33 OB can be held stationary and receive a portion of the suture 310B according to the wrapping action. The left robotic arm 33 OB can include a wrapping direction indicator 332B, and a wrapping number indicator 334B.

[0069] The wrapping direction indicator 332B can provide a visual indication of a path of wrapping of the suture 310B around the left robotic arm 330B. For example, according to a right-hand rule, when the thumb is pointed in a direction, the curled fingers point in the wrapping direction. Thus, a wrapping direction can be determined based on a handedness and a thumb direction. Thus, a hand figure with a thumb pointing portion can be indicative of a wrapping indicator. For example, the wrapping direction indicator 332B can indicate wrapping from a lower side of the left robotic arm 33 OB up a front of the left robotic arm 33 OB, and down a rear of the left robotic arm 330B. The wrapping number indicator 334B can provide a visual indication of a number of revolutions to perform in conjunction with the wrapping direction indicator 332B. For example, the wrapping number indicator 334B can indicate two revolutions (e.g., “x2”). The handedness indicator 340B can indicate a handedness of the left hand of the surgeon. For example, the handedness indicator 340B can correspond to a left hand of the surgeon. For example, the handedness indicator 340B can correspond to a wrapping indicator according to a determination that a robotic arm, or a hand corresponding to the robotic arm, is to be wrapped according to a determined knot as discussed herein. For example, the handedness indicator 340B can indicate the wrapping direction as a result of a chosen robotic arm (or hand) highlighted for grasping or wrapping as discussed herein.

[0070] FIG. 3C depicts an example third knot tying presentation, according to this disclosure. As illustrated by way of example in FIG. 3C, a third knot tying presentation 300C can include at least a suture 310C, a right robotic arm 320C, and a left robotic arm 330C. The suture 310C can correspond to a state subsequent to the state of the suture 310B. The right robotic arm 320C can grasp the suture 310C and perform a pulling action according to the pulling direction indicator 322C. The right robotic arm 320C can include a pulling direction indicator 322C. The pulling direction indicator 322C can provide a visual indication of a direction of pulling the suture 310C by the right robotic arm 320C. For example, the pulling direction indicator 322Ccan indicate pulling to the right by a right arrow. The left robotic arm 330C can grasp the suture 3 IOC and perform a pulling action according to the pulling direction indicator 332C. The left robotic arm 330C can include a pulling direction indicator 332C. The pulling direction indicator 332C can provide a visual indication of a direction of pulling the suture 3 IOC by the left robotic arm 330C. For example, the pulling direction indicator 332C can indicate pulling to the left by a left arrow.

[0071] FIG. 3D depicts an example fourth knot tying presentation, according to this disclosure. As illustrated by way of example in FIG. 3D, a fourth knot tying presentation 300D can include at least a closed patient site 302D, a suture 310D, a right robotic arm 320D, and a highlighted left robotic arm 330D. The closed patient site 302D can correspond to an opening at a patient site during a medical procedure that has been at least partially joined by the suture 310D. For example, the closed patient site 302D can be a closed incision of skin of a patient during a medical procedure, but is not limited thereto. The suture 310D can correspond to a state subsequent to the state of the suture 310C. The suture 310D can include a grasping direction indicator 312D. The grasping direction indicator 312D can correspond to a visual indication located at or near the suture 310D, that is indicative of a direction of movement of the suture 310D in a direction corresponding to a length of the suture 310D. For example, the grasping direction indicator 312D can indicate that the suture 310D is to be grasped downward along a right side of the patient site 302D. The right robotic arm 320D can be in an idle state during a knot tying action corresponding to the knot tying presentation 300D. The highlighted left robotic arm 330D can correspond to a visual indication having an overlay associated with the grasping direction indicator 312D. For example, the highlighted left robotic arm 320D can be highlighted with a translucent color fill (e.g., green) to indicate that the highlighted left robotic arm 320D is to be used to grasp the suture 310D and complete an action of grasping the suture 310D according to the grasping direction indicated by the grasping direction indicator 312D. For example, the knot tying presentation 300D can correspond to a first step or action in tying a second knot subsequent to the actions of 300A-C, where the subsequent knot is a mirror image or opposite to the first knot.

[0072] FIG. 3E depicts an example fifth knot tying presentation, according to this disclosure. As illustrated by way of example in FIG. 3E, a fifth knot tying presentation 300E can include at least a suture 310E, a right robotic arm 320E, a left robotic arm 330E, and a handedness indicator 340E. The suture 310E can correspond to a state subsequent to the state of the suture310D. The right robotic arm 320E can be held stationary and receive a portion of the suture 310E according to the wrapping action. The right robotic arm 320E can include a wrapping direction indicator 322E and a wrapping number indicator 324E. The wrapping direction indicator 322E can provide a visual indication of a path of wrapping of the suture 310E around the right robotic arm 320E. For example, the wrapping direction indicator 322E can indicate wrapping from a lower side of the right robotic arm 320E up a front of the right robotic arm 320E, and down a rear of the right robotic arm 320E. The wrapping number indicator 324E can provide a visual indication of a number of revolutions to perform in conjunction with the wrapping direction indicator 322E. For example, the wrapping number indicator 334B can indicate one revolution (e.g., “xl”). The left robotic arm 330E can grasp the suture 310E and perform a wrapping action according to the visual indications of the knot tying presentation 300E. The handedness indicator 340E can indicate a handedness of the left hand of the surgeon. For example, the handedness indicator 340E can correspond to a right hand of the surgeon.

[0073] FIG. 3F depicts an example sixth knot tying presentation, according to this disclosure. As illustrated by way of example in FIG. 3F, a sixth knot tying presentation 3 OOF can include at least a suture 310F, a right robotic arm 320F, and a left robotic arm 330F. The suture 310F can correspond to a state subsequent to the state of the suture 310E. The right robotic arm 320F can correspond at least partially in one or more of structure and operation to the right robotic arm 320C. The right robotic arm 320F can include a pulling direction indicator 322F. The pulling direction indicator 322F can correspond at least partially in one or more of structure and operation to the pulling direction indicator 322C. The left robotic arm 330F can correspond at least partially in one or more of structure and operation to the left robotic arm 330C. The left robotic arm 330F can include a pulling direction indicator 332F. The pulling direction indicator 332F can correspond at least partially in one or more of structure and operation to the pulling direction indicator 332C.

[0074] As illustrated herein by way of example, the data processing system 102 can perform one or more actions as discussed herein according to Figs. 4A-E, with respect to a one-handed technique in response to determining a horizontal defect associated with at least one of the patient site, the suture, the robotic arms, or any combination thereof, but is not limited thereto.

[0075] FIG. 4A depicts an example seventh knot tying presentation, according to this disclosure. As illustrated by way of example in FIG. 4A, a seventh knot tying presentation 400 A can include at least an open patient site 402 A, a suture 410A, a right robotic arm 420 A,and a highlighted left robotic arm 430 A. The open patient site 402 A can correspond at least partially in one or more of structure and operation to the open patient site 302 A, and can correspond to a medical procedure, phase of the medical procedure, task of the phase, or patient site distinct from the open patient site 302 A. The suture 410A can correspond to a surgical thread that is at least partially disposed through the patient site 402A. The suture 410A can be depicted in the knot tying presentation 400A according to video data, one or more annotation or overlays corresponding to the location of the suture 410A, or any combination thereof. The suture 410A can include a grasping direction indicator 412A. The grasping direction indicator 412A can correspond to a visual indication located at or near the suture 410A, that is indicative of a direction of movement of the suture 410A in a direction corresponding to a length of the suture 410A. For example, the grasping direction indicator 412A can indicate that the suture 410A is to be grasped downward along a rear side of the patient site 402 A. The right robotic arm 420A can be in an idle state during a knot tying action corresponding to the knot tying presentation 400A. The highlighted left robotic arm 430A can correspond to a visual indication having an overlay associated with the grasping direction indicator 412A. For example, the highlighted left robotic arm 430A can be highlighted with a translucent color fill (e.g., green) to indicate that the highlighted right robotic arm 430A is to be used to grasp the suture 410A and complete an action of grasping the suture according to the grasping direction indicated by the grasping direction indicator 412A.

[0076] FIG. 4B depicts an example eighth knot tying presentation, according to this disclosure. As illustrated by way of example in FIG. 4B, an eighth knot tying presentation 400B can include at least a suture 410B, a right robotic arm 420B, a left robotic arm 430B, and a handedness indicator 440B. The suture 410B can correspond to a state subsequent to the state of the suture 410A. The right robotic arm 420B can be held stationary and receive a portion of the suture 410B according to the wrapping action. The right robotic arm 420B can include a wrapping direction indicator 422B, and a wrapping number indicator 424B. The wrapping direction indicator 422B can correspond at least partially in one or more of structure and operation to the wrapping direction indicator 322E. The wrapping number indicator 424B can provide a visual indication of a number of revolutions to perform in conjunction with the wrapping direction indicator 422B. For example, the wrapping number indicator 424B can indicate two revolutions (e.g., “x2”). The left robotic arm 430B can grasp the suture 410B and perform a wrapping action according to the visual indications of the knot tying presentation 400B. The handedness indicator 440B can indicate a handedness of the right hand of thesurgeon. For example, the handedness indicator 440B can correspond to a right hand of the surgeon.

[0077] FIG. 4C depicts an example ninth knot tying presentation, according to this disclosure. As illustrated by way of example in FIG. 4C, a ninth knot tying presentation 400C can include at least a closed patient site 402C, a suture 410C, a right robotic arm 420C, and a left robotic arm 430C. The closed patient site 402C can correspond at least partially in one or more of structure and operation to the closed patient site 302D. The suture 410C can correspond to a state subsequent to the state of the suture 410B. The right robotic arm 420C can grasp the suture 310C and perform a pulling action according to a pulling direction indicator 422C. The pulling direction indicator 422C can provide a visual indication of a first direction of pulling the suture 410C by the right robotic arm 420C. For example, the pulling direction indicator 422C can indicate pulling to the back by an up arrow. For example, the knot tying presentation 400C can sequentially or concurrently present the pulling direction indicator 422C to guide the surgeon in pulling in real-time during the medical procedure. The left robotic arm 430C can grasp the suture 410C and perform a pulling action according to a pulling direction indicator 432C. The pulling direction indicator 432C can provide a visual indication of a direction of pulling the suture 410C by the left robotic arm 430C. For example, the pulling direction indicator 432C can indicate pulling forward by a down arrow.

[0078] FIG. 4D depicts an example tenth knot tying presentation, according to this disclosure. As illustrated by way of example in FIG. 4D, a tenth knot tying presentation 400D can include at least a suture 410D, a right robotic arm 420D, a left robotic arm 430D, and a handedness indicator 440D. The suture 410D can correspond to a state subsequent to the state of the suture 410C. The right robotic arm 420D can correspond at least partially in one or more of structure and operation to the right robotic arm 420B. The right robotic arm 420D can include a wrapping direction indicator 422D, and a wrapping number indicator 424D. The wrapping direction indicator 422D can provide a visual indication of a path of wrapping of the suture 410D around the right robotic arm 420D. For example, the wrapping direction indicator 422D can indicate wrapping from a lower side of the right robotic arm 320E up a front of the right robotic arm 320E, and down a rear of the right robotic arm 420D. The wrapping number indicator 424D can provide a visual indication of a number of revolutions to perform in conjunction with the wrapping direction indicator 422D. For example, the wrapping number indicator 424D can indicate one revolution (e.g., “xl”). The left robotic arm 430D can grasp the suture 410D andperform a wrapping action according to the visual indications of the knot tying presentation 400D. The handedness indicator 440D can indicate a handedness of the right hand of the surgeon. For example, the handedness indicator 440D can correspond to a right hand of the surgeon, and indicate at least one of a first handedness wrapping direction 442D and a second handedness wrapping direction 444D. For example, the knot tying presentation 400D can sequentially or concurrently present the first handedness wrapping direction 442D and the second handedness wrapping direction 444D to guide the surgeon in wrapping in real-time during the medical procedure.

[0079] FIG. 4E depicts an example eleventh knot tying presentation, according to this disclosure. As illustrated by way of example in FIG. 4E, an eleventh knot tying presentation 400E can include at least a suture 410E, a right robotic arm 420E, and a left robotic arm 430E. The suture 410E can correspond to a state subsequent to the state of the suture 410D. The right robotic arm 420E can grasp the suture 420E and be held stationary during an action to pull the suture 420E. The right robotic arm 420E can include a pulling direction indicator 422E. The pulling direction indicator 422E can provide a visual indication of a direction of pulling the suture 410E by the left robotic arm 430E with respect to a portion of the suture 410E grasped by the right robotic arm 420E. For example, the pulling direction indicator 422E can indicate movement toward the patient site by a down arrow. The left robotic arm 430E can correspond at least partially in one or more of structure and operation to the left robotic arm 330C. The left robotic arm 430E can include a pulling direction indicator 432E, and a pulling direction indicator 432E. The pulling direction indicator 432E can provide a visual indication of a direction of pulling the suture 410E by the left robotic arm 430E. For example, the pulling direction indicator 432E can indicate pulling to the rear by an up arrow.

[0080] As illustrated herein by way of example, the data processing system 102 can perform one or more actions as discussed herein according to Figs. 3 A-F, with respect to a two-handed technique in response to determining a vertical defect associated with at least one of the patient site, the suture, the robotic arms, or any combination thereof, but is not limited thereto.

[0081] FIG. 5A depicts an example first knot tying sequence presentation, according to this disclosure. As illustrated by way of example in FIG. 5A, a first knot tying sequence presentation 500A can include at least a suture 510, a right robotic arm 520, a highlighted left robotic arm 530, and a knot sequence presentation 540A. The suture 510 can correspond at least partially in one or more of structure and operation to the suture 410A. The right roboticarm 520 can correspond at least partially in one or more of structure and operation to the right robotic arm 420A. The highlighted left robotic arm 530 can correspond at least partially in one or more of structure and operation to the highlighted left robotic arm 430 A. The knot sequence presentation 540A can indicate a plurality of wrapping directions for the robotic arms 520 and 530. The knot sequence presentation 540A can include a next step presentation 550A, and a previous step presentation 560A. The next step presentation 550A can indicate a wrapping direction for each of the robotic arms 520 and 530 subsequent to a current action. The next step presentation 550A can include a left robotic arm wrapping direction indicator 552A, and a right robotic arm wrapping direction indicator 554 A. The left robotic arm wrapping direction indicator 552 A can indicate a wrapping direction in a first direction. The right robotic arm wrapping direction indicator 554 A can indicate a wrapping direction in a second direction opposite to the first direction. The wrapping direction can be altered to conform to a knot structure according to a surgeon’s knot. The surgeon’s knot has an advantageous structure to resist degradation and loosening, as a result of the sequential operations in opposite directions as discussed herein. The previous step presentation 560A can indicate a wrapping direction for each of the robotic arms 520 and 530 before a current action. The previous step presentation 560A can include a left robotic arm wrapping direction indicator 562A, and a right robotic arm wrapping direction indicator 564A. The left robotic arm wrapping direction indicator 562A can indicate a wrapping direction in the second direction. The right robotic arm wrapping direction indicator 564A can indicate a wrapping direction in the first direction.

[0082] FIG. 5B depicts an example second knot tying sequence presentation, according to this disclosure. As illustrated by way of example in FIG. 5B, a second knot tying sequence presentation 500B can include at least a knot sequence presentation 540B. The knot sequence presentation 540B can indicate a plurality of handedness directions for the robotic arms 520 and 530. The knot sequence presentation 540B can include a next step presentation 550B, and a previous step presentation 560B. The next step presentation 550B can indicate a handedness direction for each of the robotic arms 520 and 530 subsequent to a current action. The next step presentation 550B can include a left robotic arm handedness indicator 552B, and a right robotic arm handedness indicator 554B. The left robotic arm handedness indicator 552B can indicate a first handedness in a first wrapping direction. The right robotic arm handedness indicator 554B can indicate a second handedness in the first wrapping direction. The previous step presentation 560B can indicate a handedness wrapping direction for each of the robotic arms 520 and 530 before a current action. The previous step presentation 560B can include a leftrobotic arm handedness indicator 562B, and a right robotic arm handedness indicator 564B. The left robotic arm handedness indicator 562B can indicate the second handedness in a second wrapping direction. The right robotic arm handedness indicator 564B can indicate the first handedness in the second wrapping direction.

[0083] FIG. 5C depicts an example third knot tying sequence presentation, according to this disclosure. As illustrated by way of example in FIG. 5C, a third knot tying sequence presentation 500C can include at least a knot sequence presentation 540C. The knot sequence presentation 540C can indicate a plurality of wrapping directions for the highlighted robotic arm 530. The knot sequence presentation 540C can include a wrapping direction indicator 550C, and a wrapping direction indicator 560C. The wrapping direction indicator 550C can indicate a left wrapping direction for a previous step to the current step. The wrapping direction indicator 560C can indicate a right wrapping direction for a subsequent step to the current step.

[0084] FIG. 6A depicts an example first simulated knot tying presentation, according to this disclosure. As illustrated by way of example in FIG. 6A, a first simulated knot tying presentation 600 A can include at least an open patient site 602 A, a suture 610A, a right robotic arm 620 A, and a left robotic arm 630 A. The open patient site 602 A can correspond at least partially in one or more of structure and operation to the open patient site 302 A. The suture 610A can correspond at least partially in one or more of structure and operation to the suture 310A. The right robotic arm 620 A can correspond at least partially in one or more of structure and operation to the right robotic arm 320A. The left robotic arm 630A can correspond at least partially in one or more of structure and operation to the left robotic arm 330A. For example, the simulated knot tying presentation 600A can correspond to a presentation of a virtual environment including a virtual patient site 602 A, and virtual robotic arms 620 A and 630 A. For example, the virtual patient site 602 A can correspond to a simulated representation of the open patient site 302 A or the open patient site 602 A.

[0085] FIG. 6B depicts an example second simulated knot tying presentation, according to this disclosure. As illustrated by way of example in FIG. 6B, a second simulated knot tying presentation 600B can include at least a closed patient site 602B, a suture 610B, a right robotic arm 620B, and a left robotic arm 630B. The closed patient site 602B can correspond at least partially in one or more of structure and operation to the closed patient site 302D, and can have a visual property responsive to an action to move the suture 610B or execute an action to tie a knot by the suture 610B. For example, the closed patient site 610A can correspond to asimulated representation of the closed patient site 302D. For example, the closed patient site 602B can have a color lighter (e.g., modifying color toward white) than that of the surrounding tissue to indicate blanching of tissue in response to pressure applied by the suture 610B closing around the patient site 602B. For example, the closed patient site 602B can have a color darker (e.g., modifying color toward red) than that of the surrounding tissue to indicate bruising or bleeding of tissue in response to pressure applied by the suture 61 OB closing around the patient site 602B that exceeds a tissue integrity threshold. For example, the knot tying presentation 600B can modify a visual property of one or more of the suture 61 OB, the right robotic arm 620B, and the left robotic arm 630B, in response to modifying the visual property of the patient site 602B.

[0086] FIG. 7 depicts an example method of assisted knot tying, according to this disclosure. At least the data processing system 102 or any component thereof can perform method 700. The data processing system 102 can perform one or more portions of the method 700 to generate at least one of the presentations 300A-F, 400A-E, 500A-C, or 600A-B, but is not limited thereto. For example, the data processing system 102 can generate one or more presentations according to at least a portion of the method 700, that includes an overlay of a live medical procedure (e.g., intra-operative annotations during a surgical procedure involving the robotic system 104). For example, the data processing system 102 can generate one or more presentations according to at least a portion of the method 700, that includes an overlay of a complete medical procedure (e.g., post-operative annotations after a surgical procedure involving the robotic system 104). For example, the data processing system 102 can generate one or more presentations according to at least a portion of the method 700, that includes an overlay of an active simulation of a medical procedure (e.g., intra-operative annotations during a surgical procedure involving the robotic system 104). For example, the data processing system 102 can generate one or more presentations according to at least a portion of the method 700, that includes an overlay of a complete simulated medical procedure (e.g., post-operative annotations of a recorded simulation of surgical procedure involving the robotic system 104).

[0087] At 710, the method 700 can receive a video stream. For example, the interface controller 112 can receive the video stream. For example, the video stream can correspond to the video data 152. At 712, the method 700 can receive the video stream via a robotic medical system. For example, the robotic system 104 can provide the video stream to the interface controller 112 via the network 101. At 714, the method 700 can receive the video stream of a medicalprocedure performed on an anatomical structure. For example, the medical procedure can correspond to a knot-tying movement or plurality of knot-tying movement to suture an open patient site (e.g., to close an incision).

[0088] At 720, the method 700 can identify a type of knot to tie. For example, the environment processor 120 can identify the type of knot to tie. At 722, the method 700 can identify the type of knot to tie with a suture on the anatomical structure. For example, the environment processor 120 can identify the type of knot to tie with a suture on the anatomical structure. For example, the suture state processor 214 can identify the type of knot to tie with a suture on the anatomical structure. For example, the patient state processor 216 can identify the type of knot to tie with a suture on the anatomical structure. At 724, the method 700 can identify the type of knot based at least in part on the video stream. For example, the image feature processor 212 can identify the type of knot to tie based at least in part on the video stream.

[0089] FIG. 8 depicts an example method of assisted knot tying, according to this disclosure. At least the data processing system 102 or any component thereof can perform method 800. The data processing system 102 can perform one or more portions of the method 800 to generate at least one of the presentations 300A-F, 400A-E, 500A-C, or 600A-B, but is not limited thereto. For example, the data processing system 102 can generate one or more presentations according to at least a portion of the method 800, that includes an overlay of a live medical procedure (e.g., intra-operative annotations during a surgical procedure involving the robotic system 104). For example, the data processing system 102 can generate one or more presentations according to at least a portion of the method 800, that includes an overlay of a complete medical procedure (e.g., post-operative annotations after a surgical procedure involving the robotic system 104). For example, the data processing system 102 can generate one or more presentations according to at least a portion of the method 800, that includes an overlay of an active simulation of a medical procedure (e.g., intra-operative annotations during a surgical procedure involving the robotic system 104). For example, the data processing system 102 can generate one or more presentations according to at least a portion of the method 800, that includes an overlay of a complete simulated medical procedure (e.g., post-operative annotations of a recorded simulation of surgical procedure involving the robotic system 104).

[0090] At 810, the method 800 can generate one or more actions to tie the type of knot. For example, the robot state processor 220 can generate one or more actions to tie the type of knot. For example, the knot execution processor 224 can generate one or more actions to tie the typeof knot. For example, the knot sequence processor 226 can generate one or more actions to tie the type of knot. At 812, the method 800 can generate the actions to tie the type of knot with the suture on the anatomical structure. For example, the instrument metrics processor 222 can generate the actions to tie the type of knot with the suture on the anatomical structure in accordance with one or more constraints on force associated with the patient site. For example, the patient state processor can provide one or more patient metrics 158 indicative of force or properties that can be correlated with force associated with the patient site. At 814, the method 800 can generate the actions to cause one or more instruments of the robotic medical system to tie the type of knot. For example, the knot execution processor 224 can generate the actions to cause one or more instruments of the robotic medical system to tie the type of knot. For example, the knot sequence processor 226 can generate the actions to cause one or more instruments of the robotic medical system to tie the type of knot. At 816, the method 800 can generate the actions based on the type of knot. For example, the robot state processor 220 can generate the actions based on the type of knot.

[0091] At 820, the method 800 can overlay one or more visual indications according to the one or more actions. For example, the presentation controller 230 can overlay one or more visual indications according to the one or more actions. For example, the overlay presentation processor 232 can overlay one or more visual indications according to the one or more actions. In an aspect, the one or more visual indications include an indication of a lateral direction of pulling corresponding to at least one of the instruments to tie the type of knot. For example, the presentation controller 230 can generate one or more visual indications that include an indication of a lateral direction of pulling corresponding to at least one of the instruments to tie the type of knot. In an aspect, the one or more visual indications include an indication of a rotation direction of wrapping of at least a portion of the suture on at least one of the instruments to tie the type of knot. For example, the presentation controller 230 can generate one or more visual indications that include an indication of a rotation direction of wrapping of at least a portion of the suture on at least one of the instruments to tie the type of knot.

[0092] In an aspect, the one or more visual indications include an indication of an instrument of the one or more instruments to grasp at least a portion of the suture to tie the type of knot. For example, the presentation controller 230 can generate one or more visual indications that include an indication of an instrument of the one or more instruments to grasp at least a portion of the suture to tie the type of knot. At 822, the method 800 can overlay the visual indicationsto guide the one or more instruments. For example, the overlay presentation processor 232 can overlay the visual indications to guide the one or more instruments. At 824, the method 800 can overlay the visual indications to tie the type of knot with the suture on the anatomical structure. For example, the overlay presentation processor 232 can overlay the visual indications to tie the type of knot with the suture on the anatomical structure. At 826, the method 800 can overlay the visual indications on a user interface that displays the video stream of the medical procedure. For example, the overlay presentation processor 232 can overlay the visual indications to tie the type of knot with the suture on the anatomical structure. In an aspect, the method can include where the indication of the instrument can include an overlay of a portion of a depiction of the instrument in the video stream corresponding to the instrument.

[0093] Having now described some illustrative implementations, the foregoing is illustrative and not limiting, having been presented by way of example. In particular, although many of the examples presented herein involve specific combinations of method acts or system elements, those acts and those elements may be combined in other ways to accomplish the same objectives. Acts, elements and features discussed in connection with one implementation are not intended to be excluded from a similar role in other implementations.

[0094] The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," "characterized by," "characterized in that," and variations thereof herein, is meant to encompass the items listed thereafter, equivalents thereof, and additional items, as well as alternate implementations consisting of the items listed thereafter exclusively. In one implementation, the systems and methods described herein consist of one, each combination of more than one, or all of the described elements, acts, or components.

[0095] References to "or" may be construed as inclusive so that any terms described using "or" may indicate any of a single, more than one, and all of the described terms. References to at least one of a conjunctive list of terms may be construed as an inclusive OR to indicate any of a single, more than one, and all of the described terms. For example, a reference to "at least one of 'A' and 'B'" can include only 'A', only 'B', as well as both "A1and 'B'. Such references used in conjunction with "comprising" or other open terminology can include additional items. References to "is" or "are" may be construed as nonlimiting to the implementation or action referenced in connection with that term. The terms "is" or "are" or any tense or derivativethereof, are interchangeable and synonymous with "can be" as used herein, unless stated otherwise herein.

[0096] Directional indicators depicted herein are example directions to facilitate understanding of the examples discussed herein, and are not limited to the directional indicators depicted herein. Any directional indicator depicted herein can be modified to the reverse direction, or can be modified to include both the depicted direction and a direction reverse to the depicted direction, unless stated otherwise herein. While operations are depicted in the drawings in a particular order, such operations are not required to be performed in the particular order shown or in sequential order, and all illustrated operations are not required to be performed. Actions described herein can be performed in a different order. Where technical features in the drawings, detailed description or any claim are followed by reference signs, the reference signs have been included to increase the intelligibility of the drawings, detailed description, and claims. Accordingly, neither the reference signs nor their absence have any limiting effect on the scope of any clam elements.

[0097] Scope of the systems and methods described herein is thus indicated by the appended claims, rather than the foregoing description. The scope of the claims includes equivalents to the meaning and scope of the appended claims.

Claims

WHAT IS CLAIMED IS:

1. A system, comprising: one or more processors, coupled with memory, to: receive a video stream of a medical procedure performed on an anatomical structure via a robotic medical system; identify, based at least in part on the video stream, a type of knot to tie with a suture on the anatomical structure; generate, based on the type of knot, one or more actions configured to cause one or more instruments of the robotic medical system to tie the type of knot with the suture on the anatomical structure; and overlay, on a user interface that displays the video stream of the medical procedure, one or more visual indications according to the one or more actions to guide the one or more instruments to tie the type of knot with the suture on the anatomical structure.

2. The system of claim 1, comprising the one or more processors to: generate the one or more visual indications to include an indication of a lateral direction of pulling corresponding to at least one of the instruments to tie the type of knot.

3. The system of claim 1, comprising the one or more processors to: generate the one or more visual indications to include an indication of a rotation direction of wrapping of at least a portion of the suture on at least one of the instruments to tie the type of knot.

4. The system of claim 1, comprising the one or more processors to: generate the one or more visual indications include an indication of an instrument of the one or more instruments to grasp at least a portion of the suture to tie the type of knot.

5. The system of claim 4, comprising the one or more processors to: generate the indication of the instrument to include an overlay of a portion of a depiction of the instrument in the video stream corresponding to the instrument.

6. The system of claim 1, comprising the one or more processors to: overlay, on the user interface, one or more second visual indications according to the one or more actions.

7. The system of claim 6, wherein the one or more second visual indications corresponds to a later action of the one or more actions, the later action is subsequent to a current action of the one or more actions, and the current action corresponds to the one or more visual indications.

8. The system of claim 6, comprising the one or more processors to: cause the user interface to display an indication of a state of the anatomical structure, the state responsive a magnitude of force applied to the anatomical structure by the one or more instruments via the suture.

9. The system of claim 6, comprising the one or more processors to: cause the user interface to provide output including force feedback to the one or more instruments, the force feedback corresponding to a magnitude of force applied to the anatomical structure by the one or more instruments via the suture.

10. The system of claim 1, comprising the one or more processors to: detect that one or more movements of one or more of the instruments meet a threshold indicative of deviation from one or more of the actions.

11. The system of claim 10, comprising the one or more processors to: detect, based on one or more features of the video stream, that the one or more movements meet the threshold.

12. The system of claim 10, comprising the one or more processors to: detect, based on one or more metrics corresponding to the robotic medical system, that the one or more movements meet the threshold.

13. The system of claim 10, comprising the one or more processors to: modify, in response to the detection of the deviation, the type of knot in a second type of knot; andgenerate, based on the second type of knot, one or more second actions configured to cause the one or more instruments of the robotic medical system to tie the second type of knot with the suture on the anatomical structure.

14. A method, comprising: receiving, by one or more processors coupled with memory, a video stream of a medical procedure performed on an anatomical structure via a robotic medical system; identifying, by the one or more processors, based at least in part on the video stream, a type of knot to tie with a suture on the anatomical structure; generating, by the one or more processors, based on the type of knot, one or more actions configured to cause one or more instruments of the robotic medical system to tie the type of knot with the suture on the anatomical structure; and overlaying, by the one or more processors, on a user interface that displays the video stream of the medical procedure, one or more visual indications according to the one or more actions to guide the one or more instruments to tie the type of knot with the suture on the anatomical structure.

15. The method of claim 14, wherein the one or more visual indications include an indication of a lateral direction of pulling corresponding to at least one of the instruments to tie the type of knot.

16. The method of claim 14, wherein the one or more visual indications include an indication of a rotation direction of wrapping of at least a portion of the suture on at least one of the instruments to tie the type of knot.

17. The method of claim 14, wherein the one or more visual indications include an indication of an instrument of the one or more instruments to grasp at least a portion of the suture to tie the type of knot.

18. The method of claim 17, wherein the indication of the instrument includes an overlay of a portion of a depiction of the instrument in the video stream corresponding to the instrument.

19. A non-transitory computer readable medium including one or more instructions stored thereon and executable by a processor to: receive, by the processor, a video stream of a medical procedure performed on an anatomical structure via a robotic medical system; identify, by the processor and based at least in part on the video stream, a type of knot to tie with a suture on the anatomical structure; generate, by the processor and based on the type of knot, one or more actions configured to cause one or more instruments of the robotic medical system to tie the type of knot with the suture on the anatomical structure; and overlay, by the processor on a user interface that displays the video stream of the medical procedure, one or more visual indications according to the one or more actions to guide the one or more instruments to tie the type of knot with the suture on the anatomical structure.

20. The non-transitory computer readable medium of claim 19, the non-transitory computer readable medium further including one or more instructions executable by the processor to: overlay, by the processor on the user interface, one or more second visual indications according to the one or more actions, wherein the one or more second visual indications corresponds to a later action of the one or more actions, the later action is subsequent to a current action of the one or more actions, and the current action corresponds to the one or more visual indications.

Images