Image-guided autonomous robotic surgical resection systems and related methods

Abstract

Methods of producing an incised tissue are presented. The methods may include: contacting a portion of a tissue grasping implement of a first robotic manipulator with a tissue to produce contacted tissue, grasping the contacted tissue using the tissue grasping implement of the first robotic manipulator to produce grasped tissue, and creating an incision in the grasped tissue using a surgical cutting implement of a second robotic manipulator to produce the incised tissue. Additional methods as well as related systems and computer readable media are also presented.

Description

IMAGE-GUIDED AUTONOMOUS ROBOTIC SURGICAL RESECTION SYSTEMS AND RELATED METHODSCross-Reference To Related Applications

[0001] This application claims the benefit of U.S. Provisional Patent Application Ser. No. 63 / 704,251 , filed October 7, 2024, the disclosures of which is incorporated herein by reference in its entirety.Statement of Government Interest

[0002] This invention was made with government support under grant 2144348 awarded by the National Science Foundation (NSF). The government has certain rights in the invention.Field

[0003] This disclosure relates generally to robotic surgery.Background

[0004] Surgical cutting of soft tissues is important across various treatment modalities, encompassing traditional manual surgery, minimally invasive surgery (MIS), and robot-assisted minimally invasive surgery (RAMIS). Across these approaches, precision in tissue cutting is paramount for both the operation’s success and patient safety. Tissues cut in these procedures range from one-dimensional structures akin to vessels, to two-dimensional membranes like fascia or skin, and complex three-dimensional organs, referred to as volumetric entities, in which dimensions are comparably significant. The challenge of volumetric soft tissue cutting is heightened by significant deformation resulting from the interaction between the surgical instrument and the tissue. For example, in tumor resection, it is challenging but crucial to maintain an appropriate margin between the cut and the tumor edge toensure the complete excision of malignant tissue while minimizing healthy tissue removal. Margins too close to the tumor risk unsafe outcomes and increase the probability of cancer recurrence. Excessive removal of tissue, however, may negatively impact the patient’s quality of life.

[0005] Surgeons require rigorous training to master the precision needed for soft tissue cutting using hand-held surgical tools and teleoperated surgical robotic systems. Given the high costs and extensive efforts involved in creating physical phantoms, virtual simulations have been developed to enhance surgeon training. These simulations allow novice surgeons to practice essential techniques in a safe and risk-free environment before entering the operating room. Several virtual simulation platforms are available for training purposes, including Actaeon (BBZ Sri, Verona, Italy), dV Trainer (Mimic Technologies, Seattle, WA), and ROSS (Simulated Surgical Systems, San Jose, CA). Additionally, the da Vinci Skills Simulator (Intuitive Surgical, Sunnyvale, CA) appends a virtual simulation console to the robotic system for real-time control and haptic feedback. Moreover, recent advances in autonomous robotic surgery, a technique that aims to automate portions of the surgical procedure through robotics, has held interest in using virtual surgery simulations for data collection to train learning-based models. This approach is viewed as more efficient compared to data acquisition from real-world experiments.

[0006] Simulating soft tissue surgery with realistic physics is a complex engineering challenge that necessitates an accurate material deformation model to ensure successful integration. In practice, the physical properties of in vivo tissues must be classified and analyzed, acknowledging that these properties may differ among patients. Deformable objects such as tissues may contain infinite degrees of freedom which can be expensive to compute, and a balance must be achievedbetween the system fidelity and computational efficiency. The simulation of soft tissue cutting further compounds this complexity, as it involves topological changes and dynamic model updates, unlike the primarily surface-oriented deformation seen in grasping scenarios. Moreover, many simulation frameworks import objects as volumetric meshes that are broken down into smaller tetrahedral elements for performance optimization. However, the act of removing tetrahedral elements of disparate sizes to simulate a cutting contradicts the objective of replicating a precise surgical cut.

[0007] While the majority of soft tissue modeling research focuses on deformation analysis for surgical simulation, integrating these models into robotic control loops remains largely unexplored. This disconnect represents a significant research gap between surgery simulation and actual surgical practices. DeformerNet was introduced, as a neural network architecture designed for shape servoing tasks, demonstrating its surgical applicability through the precise manipulation of soft tissues using simulated training data. Similarly, a position-based dynamics simulation to train SuperPM was developed, as a software enhancing the surgical scene’s perception by accurately identifying and tracking essential tissues and organs. However, these works have yet to extend to surgical cutting, as DeformerNet didn’t address cutting tasks, and SuPerPM lacked post-cut tissue tracking capabilities. Finite element analysis (FEA)-based simulation has been utilized to study surgical cutting parameters focusing on straight-line incisions, neglecting the complex cutting patterns encountered in procedures like resection surgeries. LapGym was introduced, leveraging Simulation Open Framework Architecture (SOFA) to develop autonomous laparoscopic skills through reinforcement learning, albeit without real-world experimental validation. Among the rare frameworks available for simulating cuts indeformable objects, DiSECt was noteworthy as a differentiable simulator calibrated to precisely replicate force and deformation in a variety of cutting experiments on soft materials. However, its focus on simulating straight-line cuts with a large, all-purpose knife failed to address the nuanced requirements of complex surgical cutting scenarios.

[0008] Accordingly, there exists an unmet need for additional robotic surgical systems and related methods that create accurate and complex incisions, among other attributes.Summary

[0009] According to various embodiments, a surgical system is presented. The surgical system includes at least two robotic manipulators, wherein at least a first robotic manipulator comprises at least one tissue grasping implement and at least a second robotic manipulator comprises at least one surgical cutting implement; and, at least one controller operably connected at least to the first and second robotic manipulators, wherein the controller comprises at least one processor and at least one memory communicatively coupled to the processor, the memory storing non-transitory instructions which, when executed by the processor, perform operations comprising: contacting at least a portion of the tissue grasping implement of the first robotic manipulator with at least one tissue to produce contacted tissue; grasping the contacted tissue using the tissue grasping implement of the first robotic manipulator to produce grasped tissue; and, creating at least one incision in the grasped tissue using the electrosurgical cutting implement of the second robotic manipulator to produce incised tissue.

[0010] Various optional features of the above surgical system include the following. The method may include outputting the image of the target location. Thetissue grasping implement comprises a robotic gripping mechanism. The surgical cutting implement comprises an electrosurgical cutting implement. The tissue grasping implement comprises a vacuum grasping mechanism. The non-transitory instructions which, when executed by the processor, perform operations comprising: applying vacuum to the contacted tissue using the vacuum grasping mechanism of the first robotic manipulator to produce the grasped tissue. The system operates substantially autonomously. The electrosurgical cutting implement comprises at least one cautery device. The tissue comprises at least a portion of a tumor. The robotic manipulators are each configured to move with six degrees-of-freedom. The non-transitory instructions which, when executed by the processor, further perform operations comprising: removing at least a portion of the incised tissue using the first robotic manipulator. The system is configured to operate at least partially under human supervisory control. The human supervisory control comprises one or more tasks selected from the group consisting of: a real-time monitoring task, a surgical plan request task, a surgical plan approval task, and a selective intervention task. A subject comprises the tissue. An oral cavity of the subject comprises the tissue. The tissue comprises a soft tissue.

[0011] Various additional optional features of the above surgical system include the following. The surgical system further includes at least one camera operably connected to the controller, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: imaging at least a portion of the tissue using the camera to produce an imaged tissue data set; and, guiding movement and / or other operations of the robotic manipulators using the imaged tissue data set. The imaged tissue data set comprises at least one video image data set. The camera comprises a light source configured to illuminate one or moreareas disposed at least proximal to the tissue. The surgical system includes imaging at least the portion of the tissue and / or guiding movement and / or other operations of the robotic manipulators using at least one markerless tissue tracking technique and / or at least one closed-loop motion control technique. The markerless tissue tracking technique comprises at least pixel-level tracking resolution. The surgical system includes at least one tissue tracker, at least one path planner, at least one velocity controller, and at least one robot motion planner, wherein the tissue tracker receives at least a portion of the imaged tissue data set from the camera and computes tracking feedback value Dk using the portion of the imaged tissue data set, wherein the path planner and the velocity controller receive the tracking feedback value Dk from the tissue tracker, wherein the path planner updates a control input value p / <+i using the tracking feedback value Dk, wherein the velocity controller updates a control input value Vk+i using the tracking feedback value Dk, wherein the robot motion planner receives the control input value p / <+i from the path planner, wherein the robot motion planner receives the control input value Vk+i from the velocity controller, and wherein the robot motion planner updates a motion plan of one or more of the robotic manipulators using the control input value p / <+i and / or the control input value v +i . The tissue tracker comprises a three-dimensional (3D) tissue tracker. The camera comprises a depth sensing camera and / or a near infrared (NIR) camera. The non- transitory instructions which, when executed by the processor, further perform operations comprising: implementing one or more tissue incision planning procedures that use the imaged tissue data set, which tissue incision planning procedures are selected from the group consisting of: an NIR fluorescent landmark guidance procedure, a dynamic visual adjustment procedure, and a finite-element analysis optimization procedure. The depth sensing camera comprises an RGB-D camera. Thesurgical system further includes at least one smoke evacuator operably connected to the controller, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: evacuating smoke from one or more areas at least proximal to the tissue using the smoke evacuator. One or more of the robotic manipulators comprise the smoke evacuator.

[0012] According to various embodiments, a method of producing an incised tissue is presented. The method includes contacting at least a portion of at least one tissue grasping implement of at least a first robotic manipulator with at least one tissue to produce contacted tissue; grasping the contacted tissue using the tissue grasping implement of the first robotic manipulator to produce grasped tissue; and, creating at least one incision in the grasped tissue using at least one surgical cutting implement of at least a second robotic manipulator to produce incised tissue, thereby producing the incised tissue.

[0013] Various optional features of the above method include the following. The tissue grasping implement comprises a robotic gripping mechanism. The surgical cutting implement comprises an electrosurgical cutting implement. The tissue grasping implement comprises a vacuum grasping mechanism. The method includes applying vacuum to the contacted tissue using the vacuum grasping mechanism of the first robotic manipulator to produce the grasped tissue. The method includes performing the method using a system that operates substantially autonomously. The electrosurgical cutting implement comprises at least one cautery device. The tissue comprises at least a portion of a tumor. The first and second robotic manipulators are each configured to move with six degrees-of-freedom. The method includes removing at least a portion of the incised tissue using the first robotic manipulator.

[0014] Various additional optional features of the above method include the following. The method includes the human supervisory control comprises one or more tasks selected from the group consisting of: a real-time monitoring task, a surgical plan request task, a surgical plan approval task, and a selective intervention task. A subject comprises the tissue. An oral cavity of the subject comprises the tissue. The tissue comprises a soft tissue. The method further includes imaging at least a portion of the tissue using at least one camera to produce an imaged tissue data set; and, guiding movement and / or other operations of the robotic manipulators using the imaged tissue data set. The imaged tissue data set comprises at least one video image data set. The camera comprises a light source configured to illuminate one or more areas disposed at least proximal to the tissue. The method includes imaging at least the portion of the tissue and / or guiding movement and / or other operations of the first and second robotic manipulators using at least one markerless tissue tracking technique and / or at least one closed-loop motion control technique. The markerless tissue tracking technique comprises at least pixel-level tracking resolution. The method includes using at least one tissue tracker, at least one path planner, at least one velocity controller, and at least one robot motion planner, wherein the tissue tracker receives at least a portion of the imaged tissue data set from the camera and computes tracking feedback value Dk using the portion of the imaged tissue data set, wherein the path planner and the velocity controller receive the tracking feedback value Dk from the tissue tracker, wherein the path planner updates a control input value p / +i using the tracking feedback value Dk, wherein the velocity controller updates a control input value Vk+i using the tracking feedback value Dk, wherein the robot motion planner receives the control input value pk+i from the path planner, wherein the robot motion planner receives the control input value Vk+i from the velocity controller, and wherein the robot motion plannerupdates a motion plan of one or more of the robotic manipulators using the control input value p / <+i and / or the control input value V <+1.

[0015] Various additional optional features of the above method include the following. The tissue tracker comprises a three-dimensional (3D) tissue tracker. The camera comprises a depth sensing camera and / or a near infrared (NIR) camera. The method includes implementing one or more tissue incision planning procedures that use the imaged tissue data set, which tissue incision planning procedures are selected from the group consisting of: an NIR fluorescent landmark guidance procedure, a dynamic visual adjustment procedure, and a finite-element analysis optimization procedure. The depth sensing camera comprises an RGB-D camera. The method further includes evacuating smoke from one or more areas at least proximal to the tissue using at least one smoke evacuator. The first and / or second robotic manipulator comprises the smoke evacuator.

[0016] According to various embodiments, a computer readable media is presented. The computer readable media comprises: contacting at least a portion of at least one tissue grasping implement of at least a first robotic manipulator with at least one tissue to produce contacted tissue; grasping the contacted tissue using the tissue grasping implement of the first robotic manipulator to produce grasped tissue; and, creating at least one incision in the grasped tissue using at least one surgical cutting implement of at least a second robotic manipulator to produce incised tissue, thereby producing the incised tissue.

[0017] Various optional features of the above computer readable media include the following. The tissue grasping implement comprises a robotic gripping mechanism. The surgical cutting implement comprises an electrosurgical cutting implement. The tissue grasping implement comprises a vacuum grasping mechanism.The non-transitory instructions which, when executed by the processor, perform operations comprising: applying vacuum to the contacted tissue using the vacuum grasping mechanism of the first robotic manipulator to produce the grasped tissue. The electrosurgical cutting implement comprises at least one cautery device. The first and second robotic manipulators are each configured to move with six degrees-of- freedom. The computer readable media includes non-transitory instructions which, when executed by the processor, further perform operations comprising: removing at least a portion of the incised tissue using the first robotic manipulator. The non- transitory instructions which, when executed by the processor, further perform operations comprising: imaging at least a portion of the tissue using at least one camera to produce an imaged tissue data set; and, guiding movement and / or other operations of the robotic manipulators using the imaged tissue data set. The imaged tissue data set comprises at least one video image data set. The non-transitory instructions which, when executed by the processor, further perform operations comprising: imaging at least the portion of the tissue and / or guiding movement and / or other operations of the robotic manipulators using at least one markerless tissue tracking technique and / or at least one closed-loop motion control technique. The markerless tissue tracking technique comprises at least pixel-level tracking resolution.

[0018] Various additional optional features of the above computer readable media include the following. The non-transitory instructions which, when executed by the processor, further perform operations comprising: using at least one tissue tracker, at least one path planner, at least one velocity controller, and at least one robot motion planner, wherein the tissue tracker receives at least a portion of the imaged tissue data set from the camera and computes tracking feedback value Dk using the portion of the imaged tissue data set, wherein the path planner and the velocity controllerreceive the tracking feedback value Dk from the tissue tracker, wherein the path planner updates a control input value p / <+i using the tracking feedback value Dk, wherein the velocity controller updates a control input value Vk+i using the tracking feedback value Dk, wherein the robot motion planner receives the control input value Pk+i from the path planner, wherein the robot motion planner receives the control input value Vk+i from the velocity controller, and wherein the robot motion planner updates a motion plan of one or more of the robotic manipulators using the control input value Pk+i and / or the control input value V <+1. The tissue tracker comprises a three- dimensional (3D) tissue tracker. The camera comprises a depth sensing camera and / or a near infrared (NIR) camera. The non-transitory instructions which, when executed by the processor, further perform operations comprising: implementing one or more tissue incision planning procedures that use the imaged tissue data set, which tissue incision planning procedures are selected from the group consisting of: an NIR fluorescent landmark guidance procedure, a dynamic visual adjustment procedure, and a finite-element analysis optimization procedure. The depth sensing camera comprises an RGB-D camera. The non-transitory instructions which, when executed by the processor, further perform operations comprising: evacuating smoke from one or more areas at least proximal to the tissue using at least one smoke evacuator. One or more of the robotic manipulators comprise the smoke evacuator.Drawings

[0019] The above and / or other aspects and advantages will become more apparent and more readily appreciated from the following detailed description of examples, taken in conjunction with the accompanying drawings, in which:

[0020] Fig. 1 is a workflow diagram of a method of producing an incised tissue according to various embodiments.

[0021] Fig. 2 schematically depicts features of an autonomous robotic surgical system according to various embodiments.

[0022] Figs. 3A-G show images of the (a) ASTR, (b) monopolar electrosurgical instrument, (c) vacuum grasping instrument, d) dual-camera vision system, sample holder, grounding pad, smoke evacuation tube, linear motion stage, and (e) simulated clinical setting featuring a porcine tongue specimen stretched using retraction sutures. The close views during the (f) surface incision, and (g) deep margin dissection for a pseudotumor on a porcine tongue tissue according to various embodiments.

[0023] Fig. 4 shows a control diagram of the ASTR according to some embodiments.

[0024] Fig. 5 depicts a simplified workflow of the midline partial glossectomy (left), and the workflow using SPM representations at subtask granularity (right) according to various embodiments.

[0025] Fig. 6 shows an FSM state diagram of pseudotumor surface incision (left), and deep margin dissection (right) according to various embodiments.

[0026] Figs. 7A-7O show NIR images of (a) a porcine tongue featuring drawn pseudotumor contours and ten NIR markers, (b) an elongated tongue, (c) an estimated tumor bed after landmark-based deformable registration, and (d) a point cloud image displaying 3D overlay of pseudotumor contours (inner ring) and robotic incision path (outer ring) after applying a 5 mm surface offset. Images of (e) surface incision, (f) detected pseudotumor contours after incision (outlined), g) vacuum grasping instrument application, (g-k) iterations of autonomous pseudotumor deep margin exposure via lifting vacuum grasping instrument, k) the front edge of pseudotumor’s deep margin, (l-m) deep margin dissection movement pattern(arrows), and m-n) tissue retraction motion pattern (arrows) during the first iteration of dual-arm cooperativedissection, o) post complete pseudotumor removal according to various embodiments (note, All images are from the autonomous resection #1 .).

[0027] Fig. 8 are images that show pseudotumor surface shapes for six autonomous resections (#1 to #6) derived from three clinically reported tumor surface shapes according to various embodiments. The rulers included in the images display1 mm increments.

[0028] Figs. 9A-9E show aspects from an autonomous resection according to various embodiments. Images of the a) top, b) left, and c) right views of removed pseudotumor from the autonomous resection #1 . The actual cut (outer ring) and reference contours (inner ring) are shown. Jet color mapped errors are also highlighted in greyscale in the accompanying legend. Plots of the d) surface incision errors distribution, and e) lateral depth dissection errors distribution for both autonomous and manual resections.

[0029] Figs. 10A-10C are images of a) the robotic setup with modified ASTR, and b-c) comparison of simulated (b) with actual incisions on a real porcine tongue (c) according to the experimental embodiment. The complete real-to-sim-to-real workflow is explained on the right.

[0030] Figs. 11 A and 11 B are images that show sample preparation for electrosurgical incision, a) An ex vivo porcine tongue is marked with a 2 cm diameter contour representing the tumor edge, and a 3 cm diameter incision path, maintaining a 5 mm resection margin, b) The tongue is immobilized at the base and extended by2 cm at the tip with a retraction suture, replicating the clinical setup for glossectomy procedures.

[0031] Figs. 12A-12D are images that show visual assessment of tissue responses to various electrosurgical powers: b) noticeable bulging at 25W, c) reducedbut present bulging at 30W, d) the intact surface prior to incision at 40W, and e) postincision discoloration and denaturation at 40W.

[0032] Figs. 13A-13E show side-by-side comparison of physics-based simulated incision and real-world electrosurgical incision. Labels indicate a) CT scanning porcine tongue sample, b) volumetric tongue mesh, c) tongue mesh stretched 2cm at the tongue tip, d) and e) side-by-side simulated and experimental surgical scene composed of 1 ) porcine tongue, 2) electrosurgical instrument, 3) tissue incision contour, 4) 2cm anterior tongue stretch and mechanical constraint, 5) posterior tongue mechanical constraint, 6) stabilization support platform, 7) electrocautery tumor edge marking.

[0033] Figs. 14A-14E are images that show visualization of incision outcomes. Plots a) and b) show two tumor shapes (dashed inner ring), corresponding preoperatively intended incision paths in black after applying the 5mm margin to the tumor contour, and simulation improved incision paths. Dots indicate the incision waypoints for robot execution. Images c) and d) display incision results following preoperatively intended paths (outer rings). Images e) and f) showcase incision results guided by simulation improved paths. All incisions follow counterclockwise direction starting from the top of each contour. Each image includes a 5mm scale bar.

[0034] Figs. 15A-15D show box plots of a) absolute errors and b) directional errors for incision results, comparing preoperatively intended and simulation-improved cuts. The denoted zones indicate failed resections of positive margins, and excessively close margins, respectively. Images c) and d) display two incised samples before and after simulation refinements, with jet color mapping to visualize errors with direction and excessively close margins.

[0035] Figs. 16A-16D show aspects of an autonomous soft-tissue electrosurgery: a) an experimental robotic setup, b) a 3D tissue tracker computes tissue deformation Dk during electrosurgical cutting which is used by the path planner and velocity controller to update control inputs pk+i and vk+i, and c) the 3D tissue tracker monitors the extent of tissue deformation Dk induced by the electrode, and the robot controller uses Dk to reduce tissue deformation along the cutting path.

[0036] Figs. 17A-17C show a manual workflow of electrosurgery on ex vivo tissue with a) using a cautery pencil to mark the incision region on b) ex vivo tissue, and c) cauterizing the tissue margin with the cautery pencil.

[0037] Figs. 18A and 18B show a path planner of robotic cauterization: 1 ) 2D landmarks H were manually specified on the dot locations and landmarks mi were generated, and 2) 3D waypoints wi e Rswere computed via Dijkstra algorithm on tissue surface Ptissue-

[0038] Figs. 19A and 19B show a tool occlusion algorithm: pebase and petip are defined on the electrode, and 2) the projection of pebase and petip on the camera 2D view determines whether mi is occluded by the electrode.

[0039] Figs. 20A and 20B show a design of the one-dimensional fuzzy controller with input of tissue alternation D and output as feed rate v of robot motion.

[0040] Figs. 21A-21 C show an experimental setup to a) evaluate markerless soft-tissue tracker with cadaver pork tongue with b) three tissue patterns for closed- loop robotic cauterization, and c) evaluate cutting-margin accuracy in 3D via RGBD camera.

[0041] Figs. 22A-22D show experimental results of the robotic autonomous electrosurgical cuts with a) process scenes with each corresponding tissuedeformation and control output, b) closed-loop results, c) open-loop result, and d) cutting margin of closed-loop (N=6) and three open-loop (N=3) results measured w.r.t. reference contour (i.e. , dash line on tissue surface).

[0042] Fig. 23 shows tracking results of movement distance measurement in 3D from initial landmarks (i.e., hashed points) to final tracked positions (i.e., open points).

[0043] Figs. 24A and 24B are images showing a representation of a porcine tongue prepared for electrosurgical incision, a. An ex vivo porcine tongue is marked with a 2cm diameter outline representing the tumor edge, and a 3cm diameter incision path, maintaining a 5mm resection margin, b. The tongue is immobilized at the base and extended by 2cm at the tip with a retraction suture, replicating the clinical setup for glossectomy procedures.

[0044] Figs. 25A-25D are images showing a visual assessment of tissue responses to various electrosurgical voltages: a. Noticeable bulging at 25v, b. Reduced but present bulging at 30v, c. The intact surface prior to incision at 40v, and d. Post-incision discoloration and denaturation at 40v.Definitions

[0045] In order for the present disclosure to be more readily understood, certain terms are first defined below. Additional definitions for the following terms and other terms may be set forth throughout the specification. If a definition of a term set forth below is inconsistent with a definition in an application or patent that is incorporated by reference, the definition set forth in this application should be used to understand the meaning of the term.

[0046] As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictatesotherwise. Thus, for example, a reference to “a method” includes one or more methods, and / or steps of the type described herein and / or which will become apparent to those persons skilled in the art upon reading this disclosure and so forth.

[0047] It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. Further, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. In describing and claiming the methods, systems, and computer readable media, the following terminology, and grammatical variants thereof, will be used in accordance with the definitions set forth below.

[0048] About: As used herein, “about” or “approximately” or “substantially” as applied to one or more values or elements of interest, refers to a value or element that is similar to a stated reference value or element. In certain embodiments, the term “about” or “approximately” or “substantially” refers to a range of values or elements that falls within 25%, 20%, 19%, 18%, 17%, 16%, 15%, 14%, 13%, 12%, 11 %, 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1 %, or less in either direction (greater than or less than) of the stated reference value or element unless otherwise stated or otherwise evident from the context (except where such number would exceed 100% of a possible value or element).

[0049] Subject: As used herein, “subject” or “test subject” refers to an animal, such as a mammalian species (e.g., human) or avian (e.g., bird) species. More specifically, a subject can be a vertebrate, e.g., a mammal such as a mouse, a primate, a simian or a human. Animals include farm animals (e.g., production cattle, dairy cattle, poultry, horses, pigs, and the like), sport animals, and companion animals (e.g., pets or support animals). A subject can be a healthy individual, anindividual that has or is suspected of having a disease or pathology or a predisposition to the disease or pathology, or an individual that is in need of therapy or suspected of needing therapy. The terms “individual” or “patient” are intended to be interchangeable with “subject.” A “reference subject” refers to a subject known to have or lack specific properties (e.g., a known pathology, such as melanoma and / or the like).

[0050] System: As used herein, "system" in the context of instrumentation refers to a group of objects and / or devices that form a network for performing a desired objective.Detailed Description

[0051] Embodiments as described herein are described in sufficient detail to enable those skilled in the art to practice the invention and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the invention. The present description is, therefore, merely exemplary.

[0052] A. Introduction

[0053] Tumor resection surgery, a vital cancer treatment, involves the complete removal of tumors and adjacent healthy tissues, demanding high surgical precision for optimal oncologic outcomes. Developed for this need, as disclosed herein, the autonomous system for tumor resection (ASTR), as sometimes referred to herein, represents a pioneering dual-arm, vision-guided autonomous robotic system. In some embodiments, ASTR integrates advanced hardware — including two six- degree-of-freedom manipulators, laparoscopic instruments for vacuum grasping and electrosurgical cutting, RGB-D and near-infrared (NIR) cameras with a dedicated light source, and an electrosurgical smoke evacuator — with sophisticated software forcoordinated system registration, safety-assured supervisory control, and specialized tracking and planning. In some embodiments, the present disclosure provides resection planning and execution techniques, validated in a glossectomy-mimicking setup with porcine tongue samples, that demonstrate high precision and consistency in autonomous glossectomy, rivalling manual techniques by experienced ENT surgeons. This underscores ASTR's ability to enhance surgical accuracy and patient outcomes in tumor resection, among many other types of surgeries.

[0054] In some aspects, the present disclosure provides a dual-arm robotic resection system that integrates electrosurgical cutting with vacuum gripping, a novel approach in tissue manipulation, enhancing surgical (e.g., tumor resection) efficiency and precision. In some embodiments, the present disclosure provides various tissue incision planning techniques, including NIR fluorescent landmark guidance, dynamic visual adjustment, and finite-element analysis optimization, that elevate the accuracy of surgical cuts. In some embodiments, the present disclosure also provides a supervised autonomous control, which operates under a human supervisory framework, ensuring safety and efficiency, with capabilities for real-time monitoring, plan request or approval, and immediate intervention in case of irregularities, among other features. Some examples disclosed herein demonstrate high system precision and consistency in ex vivo porcine tongue tumor resection, which match, and at some aspects exceed, manual resection results by experienced otolaryngologist.

[0055] The Autonomous System for Tumor Resection (ASTR) offers a high- level autonomous robotic solution to complex interventional oncology procedures, aiming at mitigating human error and reducing variability in surgeon performance. This innovation is designed to achieve consistently high surgical precision, reducing recurrence and metastasis risks, and thereby enhancing patient recovery and qualityof life. Ex vivo studies, disclosed further herein, used a clinically mimicked setup to demonstrate ASTR’s efficacy in autonomous partial midline glossectomy, a procedure that involves the precise removal of a shallow mass extending into deeper tissue layers.

[0056] In addition to glossectomies, ASTR can be used for broader applications even beyond interventional oncology, extending to various surgical resections, such as: (1 ) Other Head and Neck Cancer Resection Surgery: Effective in resecting superficial lesions in areas such as the hard palate, tongue base, tonsil, and oropharynx; (2) Urological Surgery: Suitable for partial nephrectomy in cases of superficial renal tumors; (3) Gastrointestinal Surgery: Applicable for partial hepatectomy for superficial liver lesions and resections in the colon; (4) Dermatological Surgery: Utilized for the precise excision of skin cancers, cysts, and moles; (5) Gallbladder Surgery: Beneficial for cholecystectomy, particularly in benign conditions like gallstones or cholecystitis; (6) Cardiovascular Surgery: Readily adapted for procedures such as myectomy in the treatment of hypertrophic cardiomyopathy; and (7) Gynecological Surgery: Appropriate for oophorectomy or salpingectomy in the removal of benign ovarian or fallopian tube masses, among many others applications.

[0057] The techniques and other aspects detailed in this disclosure represents a significant advancement in current surgical technology. For example, unlike existing commercialized robotic solutions for soft tissue surgery, which are largely teleoperated and possess minimal autonomous functionality, the present systems and related methods pioneer the realm of autonomous tumor resection, among other surgical interventions. These systems and related aspects stand apart not only in the commercial sector but also in academic and industrial research, as they achieve autonomous tumor resection with consistent precision. This leap from teleoperation tohigh autonomy in surgical robotics marks a significant paradigm shift, underscoring the unique and innovative nature of the present disclosure in the field of surgical oncology, among many other areas. These and other attributes will be apparent upon a complete review of the present disclosure, including the accompanying figures.

[0058] B. Exemplary Methods

[0059] In some aspects, the present disclosure provides various methods of performing at least aspects of surgical tasks. To illustrate, Fig. 1 is a workflow diagram of a method of producing an incised tissue according to various embodiments. As shown, method 100 includes contacting at least a portion of at least one tissue grasping implement of at least a first robotic manipulator with at least one tissue to produce contacted tissue (step 102). In some embodiments, the tissue comprises a soft tissue or at least a portion of a tumor. In some embodiments, the first and second robotic manipulators are each configured to move with six degrees-of-freedom. In some embodiments, a subject comprises the tissue (e.g., an oral cavity of the subject comprises the tissue, etc.). Method 100 also includes grasping the contacted tissue using the tissue grasping implement of the first robotic manipulator to produce grasped tissue (step 104). In addition, method 100 also includes creating at least one incision in the grasped tissue using at least one surgical cutting implement of at least a second robotic manipulator to produce incised tissue (step 106). Typically, method 100 is performed using a system as described herein that operates substantially autonomously.

[0060] The tissue grasping implements and surgical cutting implements include various embodiments. In some embodiments, for example, the tissue grasping implement comprises a robotic gripping mechanism. In some embodiments, the tissue grasping implement comprises a vacuum grasping mechanism. In someembodiments, the surgical cutting implement comprises an electrosurgical cutting implement. In some embodiments, the electrosurgical cutting implement comprises at least one cautery device.

[0061] In some embodiments, method 100 includes applying vacuum to the contacted tissue using the vacuum grasping mechanism of the first robotic manipulator to produce the grasped tissue. In some embodiments, method 100 further includes removing at least a portion of the incised tissue using the first robotic manipulator. In some embodiments, method 100 further includes evacuating smoke from one or more areas at least proximal to the tissue using at least one smoke evacuator. In some embodiments, the first and / or second robotic manipulator comprises the smoke evacuator.

[0062] In some embodiments, method 100 is implemented using a system that is configured to operate at least partially under human supervisory control. Exemplary systems are described further herein. In some embodiments, the human supervisory control comprises one or more tasks selected from a real-time monitoring task, a surgical plan request task, a surgical plan approval task, and / or a selective intervention task, as described herein. In some embodiments, method 100 further includes imaging at least a portion of the tissue using at least one camera to produce an imaged tissue data set, and guiding movement and / or other operations of the robotic manipulators using the imaged tissue data set. In some embodiments, the imaged tissue data set comprises at least one video image data set. In some embodiments, the camera comprises a light source configured to illuminate one or more areas disposed at least proximal to the tissue. In some embodiments, method 100 includes imaging at least the portion of the tissue and / or guiding movement and / or other operations of the first and second robotic manipulators using at least one markerless tissue trackingtechnique and / or at least one closed-loop motion control technique. In some embodiments, the markerless tissue tracking technique comprises at least pixel-level tracking resolution.

[0063] In some embodiments, method 100 includes using at least one tissue tracker, at least one path planner, at least one velocity controller, and at least one robot motion planner, wherein the tissue tracker receives at least a portion of the imaged tissue data set from the camera and computes tracking feedback value Dk using the portion of the imaged tissue data set, wherein the path planner and the velocity controller receive the tracking feedback value Dk from the tissue tracker, wherein the path planner updates a control input value p / <+i using the tracking feedback value Dk, wherein the velocity controller updates a control input value Vk+i using the tracking feedback value Dk, wherein the robot motion planner receives the control input value pk+i from the path planner, wherein the robot motion planner receives the control input value Vk+i from the velocity controller, and wherein the robot motion planner updates a motion plan of one or more of the robotic manipulators using the control input value p / <+i and / or the control input value V <+1. In some embodiments, the tissue tracker comprises a three-dimensional (3D) tissue tracker. In some embodiments, the camera comprises a depth sensing camera and / or a near infrared (NIR) camera. In some embodiments, method 100 further includes implementing one or more tissue incision planning procedures that use the imaged tissue data set, which tissue incision planning procedures are selected from the group consisting of: an NIR fluorescent landmark guidance procedure, a dynamic visual adjustment procedure, and a finite- element analysis optimization procedure. In some embodiments, the depth sensing camera comprises an RGB-D camera.

[0064] C. Exemplary Systems

[0065] The present disclosure also provides various systems and computer program products or machine readable media. In some aspects, for example, the methods described herein are optionally performed or facilitated at least in part using systems, distributed computing hardware and applications (e.g., cloud computing services), electronic communication networks, communication interfaces, computer program products, machine readable media, electronic storage media, software (e.g., machine-executable code or logic instructions) and / or the like. To illustrate, Fig. 2 provides a schematic diagram of an exemplary system suitable for use with implementing at least aspects of the methods disclosed in this application. As shown, system 200 includes at least one controller or computer, e.g., server 202 (e.g., a search engine server), which includes processor 204 and memory, storage device, or memory component 206, and one or more other communication devices 214, 216, (e.g., client-side computer terminals, telephones, tablets, laptops, other mobile devices, etc. (e.g., for receiving data sets or results, etc.) in communication with the remote server 202, through electronic communication network 212, such as the Internet or other internetwork. Communication devices 214, 216 typically include an electronic display (e.g., an internet enabled computer or the like) in communication with, e.g., server 202 computer over network 212 in which the electronic display comprises a user interface (e.g., a graphical user interface (GUI), a web-based user interface, and / or the like) for displaying results upon implementing the methods described herein. In certain aspects, communication networks also encompass the physical transfer of data from one location to another, for example, using a hard drive, thumb drive, or other data storage mechanism. System 200 also includes program product 208, as described herein, stored on a computer or machine readable medium, such as, for example, one or more of various types of memory, such as memory 206of server 202, that is readable by the server 202, to facilitate, for example, a guided search application or other executable by one or more other communication devices, such as 214 (schematically shown as a desktop or personal computer). In some aspects, system 200 optionally also includes at least one database server, such as, for example, server 210 associated with an online website having data stored thereon (e.g., entries corresponding to tissue tracking data sets, etc.) searchable either directly or through search engine server 202. System 200 optionally also includes one or more other servers positioned remotely from server 202, each of which are optionally associated with one or more database servers 210 located remotely or located local to each of the other servers. The other servers can beneficially provide service to geographically remote users and enhance geographically distributed operations.

[0066] As understood by those of ordinary skill in the art, memory 206 of the server 202 optionally includes volatile and / or nonvolatile memory including, for example, RAM, ROM, and magnetic or optical disks, among others. It is also understood by those of ordinary skill in the art that although illustrated as a single server, the illustrated configuration of server 202 is given only by way of example and that other types of servers or computers configured according to various other methodologies or architectures can also be used. Server 202 shown schematically in Fig. 2, represents a server or server cluster or server farm and is not limited to any individual physical server. The server site may be deployed as a server farm or server cluster managed by a server hosting provider. The number of servers and their architecture and configuration may be increased based on usage, demand and capacity requirements for the system 200. As also understood by those of ordinary skill in the art, other user communication devices 214, 216 in these aspects, for example, can be a laptop, desktop, tablet, personal digital assistant (PDA), cell phone,server, or other types of computers. As known and understood by those of ordinary skill in the art, network 212 can include an internet, intranet, a telecommunication network, an extranet, or world wide web of a plurality of computers / servers in communication with one or more other computers through a communication network, and / or portions of a local or other area network.

[0067] As further understood by those of ordinary skill in the art, exemplary program product or machine readable medium 208 is optionally in the form of microcode, programs, cloud computing format, routines, and / or symbolic languages that provide one or more sets of ordered operations that control the functioning of the hardware and direct its operation. Program product 208, according to an exemplary aspect, also need not reside in its entirety in volatile memory, but can be selectively loaded, as necessary, according to various methodologies as known and understood by those of ordinary skill in the art.

[0068] As further understood by those of ordinary skill in the art, the term "computer-readable medium" or “machine-readable medium” refers to any medium that participates in providing instructions to a processor for execution. To illustrate, the term "computer-readable medium" or “machine-readable medium” encompasses distribution media, cloud computing formats, intermediate storage media, execution memory of a computer, and any other medium or device capable of storing program product 208 implementing the functionality or processes of various aspects of the present disclosure, for example, for reading by a computer. A "computer-readable medium" or “machine-readable medium” may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks. Volatile media includes dynamic memory, such as the main memory of a given system. Transmission mediaincludes coaxial cables, copper wire and fiber optics, including the wires that comprise a bus. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications, among others. Exemplary forms of computer-readable media include a floppy disk, a flexible disk, hard disk, magnetic tape, a flash drive, or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave, or any other medium from which a computer can read.

[0069] Program product 208 is optionally copied from the computer-readable medium to a hard disk or a similar intermediate storage medium. When program product 208, or portions thereof, are to be run, it is optionally loaded from their distribution medium, their intermediate storage medium, or the like into the execution memory of one or more computers, configuring the computer(s) to act in accordance with the functionality or method of various aspects disclosed herein. All such operations are well known to those of ordinary skill in the art of, for example, computer systems.

[0070] In some aspects, program product 208 includes non-transitory computer-executable instructions which, when executed by electronic processor 204, perform at least: contacting at least a portion of at least one tissue grasping implement of at least a first robotic manipulator with at least one tissue to produce contacted tissue, grasping the contacted tissue using the tissue grasping implement of the first robotic manipulator to produce grasped tissue, and creating at least one incision in the grasped tissue using at least one surgical cutting implement of at least a second robotic manipulator to produce incised tissue, thereby producing the incised tissue.

[0071] In some embodiments, system 200 includes surgical system or subassembly 218, which is described in great detail herein.

[0072] D. Examples

[0073] Example 1 : Autonomous System for Tumor Resection (ASTR) - Dual-Arm Robotic Midline Partial Glossectomy

[0074] I. Introduction

[0075] In this example, we developed an autonomous system for tumor resection (ASTR), and conducted autonomous pseudotumor resections on porcine tongue tissues under human supervision. Ourfirst contribution is developing the ASTR, a dual-arm robot system designed for autonomous HNSCC resections. Specifically, the current scope of ASTR is primarily focused on early-stage, midline tongue tumors. The ASTR integrates a novel autonomous control strategy, soft tissue resection planners, and a new laparoscopic vacuum grasping robot, enhancing our previously developed laparoscopic electrosurgical robot and a dual-camera vision system. The ASTR presents multiple advancements: 1 ) deformed tissue tracking for precise planning; 2) closed-loop vacuum robot control; 3) integration of industry- grade equipment for reliability; 4) integration of the spherical linear interpolation (SLERP) algorithm for robust orientation control; 5) soft real-time control for timely response; and 6) a three-layer supervisory control ensuring procedural safety. The second contribution is establishing a novel ontology-based research framework for complex autonomous robotic surgeries. In essence, a surgical workflow should be formed using consistent terminologies (e.g. OntoSPM) in surgical process model (SPM) representations at subtask granularity, and paired with the finite state machine (FSM) state diagram for further procedural elaboration and assured implementation, as applied in our autonomous glossectomy case study. This approach provides autonomous robotsurgery data for SPM, broadening its scope to encompass both human and robotic operators, and potentially influence management practices across operating rooms (ORs), hospitals, and overall patient care in the future. For our third contribution, we not only prove the feasibility of employing the ASTR within the ontology-based research framework, but also report the first study that accomplishes accurate supervised autonomous tumor resection on animal tissues. In a surgical site setup mimicking clinical glossectomy, the autonomous surgeries succeed in all six consecutive experiments under human supervision, creating no positive margins. The resection accuracy exhibits great improvement in contrast to our prior study. Comparing with three manual resections performed by an experienced otolaryngologist, both the autonomous and manual surface incisions display submillimeter accuracy. Autonomous resections show slightly better depth dissection accuracy than manual studies, but the difference isn’t statistically significant.

[0076] II. Procedure of Glossectomy

[0077] The HNSCC resection workflows are influenced by the tumor’s location, size, and depth. This example focuses on the midline partial glossectomy, a procedure employed to address early-stage, small-sized squamous cell carcinoma (SCO) that primarily arises on the midline superior surface of the tongue. Such a glossectomy is classified as a type 1 , or mucosectomy. A clinical report on an SCC (T1 N0M0) glossectomy offers valuable insights Into this procedure. This procedure essentially resembles the excavation of a shallow mass growing from the tissue surface, and its progression into deeper tissue layers.

[0078] The choice of glossectomy is influenced by three factors. First, it fits the HNSCC resection introduction, which covers tongue SCC resection. Second, midline partial glossectomy shares characteristics with other HNSCC surgeries, likethose targeting the hard palate, tongue base, tonsil, and oropharynx. Third, its transoral approach offers a more spacious workspace than surgeries in deeper head and neck regions, making it an appropriate initial step toward autonomous HNSCC resection. In practice, surgeons use oral gags to open the mouth, and traction sutures at the tongue tip to pull the anterior tongue out of the oral cavity. The standard method involves a hand-held electrocautery pen and surgical forceps to remove lesions, approximating an open surgical setting.

[0079] In this example, we designed a pseudotumor on a porcine tongue to simulate an early-stage SCC that invades the mucosa, akin to a tumor chip. SCC surface boundaries are derived from clinical reports, and marked on the porcine tongue’s midline surface using black ink. The pseudotumor has a surface dimension of 2 cm and an invasion depth of « 1 mm. Taking clinical resection margins into account, we aim to extend beyond the pseudotumor by a margin of 5 mm both laterally and in depth.

[0080] III. Methods

[0081] A. Autonomous System for Tumor Resection (ASTR)

[0082] The ASTR, an autonomous, dual-arm, vision-guided robotic system, adopts certain technical insights related to low-level control from the STAR system. While STAR focuses on anastomosis, ASTR is specifically tailored for tumor resection, representing a unique application.

[0083] 1 ) ASTR Hardware Architecture: Fig. 3A-3D demonstrates the ASTR. T o facilitate translation to clinical settings, we primarily integrate products from industryleading manufacturers. For example, UR manipulators (Universal Robots, Odense, Den- mark) are used in several FDA-approved surgical robots like the TMS-Cobot (Axilum Robotics, Strasbourg, France).

[0084] The primary instrument for tumor resection, the electrosurgical instrument, is used to cut tissues. It consists of a standard 25 mm length and 1 mm diameter monopolar electrode (Bovie, Clearwater, FL), a customized 35 cm laparoscopic extension, a grounding pad, and an electrosurgical generator (ASG- 300ESU, DRE Veterinary, Louisville, KY). A portable smoke evacuator (Smoke Shark, Bovie) is also included to reduce electrosurgical smoke. The vacuum grasping instrument serves as a secondary tool for maneuvering tissues and exposing the area of interest for easier cutting, with its design detailed in Section III-A2. The electrosurgical and vacuum grasping instruments are mounted on the flanges of a UR10e and a UR5 robotic manipulators, respectively. These manipulators carry and direct the instruments to reach various positions and orientations within the surgical sites. The vision system incorporates an RGBD camera (D415, Intel, Santa Clara, CA), a 2D near-infrared (NIR) cam- era (acA2040-90umNIR, Basler AG, Ahrensburg, Germany), a 845 ± 27.5 nm band-pass filter (Chroma Technology, Bellows Falls, VT), and a 760 nm high power light-emitting diode (North Coast Technical, Chesterland, OH). The NIR camera and light source are chosen to achieve high SNRs of ICG, the most frequently used FDA-approved NIRF dye. Images from both cameras are fused, allowing the retrieval of 3D positions of color and NIR signals to create surgical plans and guide the robots. The development details of the electrosurgical tool and vision system are explained in our previous work.

[0085] 2) Vacuum Grasping Instrument: The vacuum grasping is an alternative to forceps grasping for soft tissue manipulation, reducing potential tissue damage. For tumor resections, this approach reduces the risk of inadvertently breaking tumor tissues, leaving undetected fragments, and potentially causing cancer recurrence. Furthermore, since SCC originates from the epithelium, using vacuumgrasping on the tumor surface simultaneously immobilizes the tumor and retracts the tissue.

[0086] The vacuum grasping instrument is comprised of an electric vacuum gripper (EPick, Robotiq, Levis, Canada), a moisture trap, a vacuum pad, and a stainless steel extension tube and fittings (Fig. 3C). The tube has a length of 35 cm and an outer diameter (OD) of 10 mm. The soft Bellows vacuum pad has a 12 mm inner diameter (ID) and a 24 mm OD, with food grade FDA compliant replacement options for future use. An in-line moisture trap connects the tube and EPick using pipe fittings, preventing blood, water, and tissue fluid from being sucked into the EPick and causing damage or corrosion to the internal electric panels. Custom-programmed robot operating system (ROS) service servers, developed in Python, control EPick vacuum between continuous on and off modes. The negative pressure is preset at “70 kPa while the vacuum mode is on, with the theoretical vacuum force on the 12 mm ID Bellows pad equaling 7.92 N.

[0087] 3) Autonomous Control Strategy: The system control software is developed using the ROS and Open Robot Control Soft- ware (OROCOS) frameworks, and operates on RT___PREEMPT Linux for real-time functionality. This architecture is also employed in dVRK, a pioneering surgical robotics project with documented bench-to-bedside initiatives.

[0088] The ASTR’s control strategy encompasses six components (Fig. 4). Initially, a checkerboard is placed near the target tissue serving as a shared world coordinate frame. Following standard hand-eye calibrations for all robots and cameras, the frames of each component are integrated, aligning them with this world frame for a unified system registration. Consequently, the dual-camera fusion is achieved. For safe autonomous surgery executions, a human supervisor oversees the ASTR systemin three main ways: i) They monitor live image streams, determining if the system is prepared to execute the next subtask; ii) They review and approve initial plans before the robot executes a sub- task, requesting changes if necessary; iii) They can immediately halt all robot operations if any irregularities are detected. Once the initial checks are done, the ASTR’s tracking and planning modules take over the control. These modules are tailored to their respective surgical workflows. Two subtask planners are developed in this study (explained in Section III-B2), but more can be developed and integrated into the ASTR. Commands generated by these planners can be position and orientation targets forwarded to the robot’s trajectory planner, or requests for specific instrument functions directed to ROS service servers. The trajectory planner incorporates robot kinematics, and controls speed modes like fast, accurate, or idle to cater to different needs. This ensures smooth robotic movements towards targets, adapting in real-time, even if a new target is received before the current one is reached. The ROS services for vacuum functions can be called and activated, and a confirmation signal is received from the EPick low-level controller regarding the vacuum’s functional status. Upon completion of a surgical sub- task, the human supervisor decides whether to proceed with the next subtask. Iterations continue until the final surgical subtask is completed.

[0089] The trajectory planner is implemented using OROCOS Real-Time Toolkit (RTT) for real-time capabilities, OROCOS Kinematics and Dynamics Library (KDL) for kinematics calculations, and Reflexxes Motion Library (RML) for instantaneous robot joint trajectory interpolations. The SLERP algorithm is integrated to enhance orientation control, utilizing quaternion representations. This approach not only ensures the shortest path between two orientations, but also avoids challenges as- sociated with Euler angle interpolation, such as gimbal lock and ambiguities wheremultiple Euler angle sets produce the same orientation. In terms of motion parameters, The maximum linear velocity along each Cartesian axis for the fast and accurate speed modes is set to 6 cm / sec and 2 mm / sec, respectively, while the maximum angular velocity is set to 15° / sec and 5° / sec, respectively.

[0090] B. A Research Framework for Autonomous Robotic Surgeries

[0091] In this example, we propose a two-step research framework to design and implement an autonomous robotic surgery workflow. First, we use ontologies to formalize the workflow using SPM representations at subtask level. Second, we utilize FSM techniques to further elaborate the procedure and achieve implementation. OntoSPM ontology and midline partial glossectomy are chosen to illustrate this approach.

[0092] 1) Ontology-Based Surgical Workflow Using SPM Representations AtSubtask Granularity: The clinical workflow of midline partial glossectomy can be decomposed into three tasks: diagnostic imaging, tumor resection, and surgical site reconstruction. This resection study focuses on the first two tasks, and simplifies the workflow at subtask level for the pseudotumor model (Fig. 5). To clarify, TBRs of a pseudotumor equal one, making medical imaging and pathological evaluation of tumor resection infeasible. The workflow is then formalized using OntoSPM terminologies, resulting in a SPM activity diagram (Fig. 5). Semantic adjustments are made for clarity, such as substituting grasping instrument for suction instrument, because the latter is defined as: a sucking instrument has the function to suck and is used to remove blood, fluid or debris from operative sites. Since traditional SPM nodes exclusively repre- sent human operations, three non-ontology terms (i.e. ASTR, robot, single) are used to indicate robot operators without distinguishing between left or right hand. Whenintegrated with the surgery time from experimental results for SPM nodes, the reported autonomous robotic surgeries can be used for SPM.

[0093] 2) Subtasks Implementation and Detailed Workflow: Fig. 6 shows theFSM state diagrams of the two subtasks, and representative images for better illustrating the workflow.

[0094] Subtask 1 is a four-state FSM that employs both cameras and the electrosurgical robot. Preoperatively, a NIR image is taken to record the original tumor bed (Fig. 7(a)). The porcine tongue is then stretched from the tongue tip by 2 cm to imitate the clinical workflow of exposing tumor, and another NIR image is taken (Fig. 7(b)). A landmark-based deformable image registration technique, using Insight Segmentation and Registration Toolkit (ITK) libraries, is performed to recover the 2D deformed pseudotumor surface boundary (Fig. 7(c)). The 3D positions of the deformed pseudotumor boundary are generated from the dual-camera image fusion design, and down-sampled to way- points with an average distance of 2 mm (green dots in Fig. 7(d)). Then, a point 10 mm above the center of waypoints is planed as the reach-in goal for the electrosurgical instrument. The surface incision path is planned by applying a 5 mm lateral offset (red dots in Fig. 7(d)), and a 5 mm incision depth (Fig. 7(e)) to all waypoints. The reach-in point and incision path are executed sequentially, with the electrosurgical instrument maintaining a vertical orientation (Fig. 7(e)). The electrosurgical robot then retracts to its initial configuration to prepare the next subtask.

[0095] Subtask 2 is a six-state FSM that employs the RGBD camera and both robots. Once the human supervisor evaluates the performance of subtask 1 and deems it satisfactory, a standard Collins tongue forceps is manually applied on porcine tongue to expose the pseudotumor deep margin (Fig. 7(f)). A color-based region growingsegmentation technique, utilizing Point Cloud Library (PCL) libraries, is employed to detect the current pseudotumor surface contours (Fig. 7(f)). Both robots then move near the sample concurrently in preparation for deep margin dissection. The pseudotumor surface center point and a vertical orientation are used as the target pose of the vacuum grasping instrument. Once the target is reached, the vacuum is turned on by executing the customized Robotiq vacuum ROS service (Fig. 7(g)). Subsequently, the vacuum grasping instrument is raised in 2 mm increments to reveal the front edge of the pseudotumor’s deep margin to the RGBD camera (Fig. 7(g)-7(k)). The exposed area of pseudotumor’s side margin is calculated at each step using a simple HSV thresholding of the dark brown color beneath the vacuum instrument. If the area stops increasing after one step during tissue retraction, the side margin is considered fully exposed (i.e. Fig. 7(j)-7(k). The front edge of the pseudotumor’s deep margin is acquired by performing a vertical slicing of 1 mm thickness to the side margin point cloud in the grasping instrument frame, and keeping the lowest slice (Fig. 7(k)). As the end effector frames of the instruments can be retrieved from the robotic kinematics, the complete pseudotumor deep margin contours are generated by projecting the pseudotumor surface contours (Fig. 7(f)) onto the xy-plane of the grasping instrument frame (Fig. 7(g)), and offsetting to the depth of the aforementioned lowest slice (Fig. 7(k)). The pseudotumor deep margin contours are further downsampled with an average distance of 2 mm along the y axis of the grasping instrument frame, and grouped into left and right pairs. The electrosurgical instrument is guided by the paired poses with a 20° downward-tilting orientation to perform the deep margin dissection in iterations (Fig. 7(l)-7(m)). The vacuum instrument is then lifted by 1 mm, with horizontal offset from the electrosurgical instrument by 1 mm, and tilted towards the RGBD camera by 2° for improved tissue retraction (Fig. 7(m) and7(n)). This repetitive dual-arm cooperative dissection process continues until all paired poses are traversed, and the pseudotumor is removed from the sample (Fig. 7(o)). Finally, both robots retract concurrently to their initial configurations.

[0096] IV. Experiments and Results

[0097] A Sample and Surgical Site Preparation

[0098] Three distinct oral tongue SCC surface shapes are identified from clinical case reports, and scaled to fit in a 2 cm diameter bounding circle, corresponding to T1 / T2 stage SCC dimensions. Nine porcine tongues (A / = 9), obtained from a grocery store (H Mart, Lyndhurst, NJ), are used as samples due to their anatomical resemblance to human tongues. Each surface contour is manually drawn on the anterior midline of two porcine tongues ( / V = 2 ■ 3 = 6) for autonomous resections using a black ink pen and laser-cut pattern films, with ten NIRF markers evenly distributed along each contour (Fig. 6). Each surface contour is manually drawn on one porcine tongue ( / V = 1 ■ 3 = 3) for manual resections as com- parison studies using a black ink pen.

[0099] A clinically-mimicked glossectomy setup is used for both autonomous and manual resections (Fig. 3(e)). We 3D printed a sample holder to replicate the jawbone and mouth floor, and added a clamp on top to stabilize posterior tongue motion. Before each test, tongue tips are sutured and extended by 2 cm using a linear stage. In autonomous resections, after subtask 1 execution, a Collin tongue forceps is manually clamped on the anterior tongue’s edge (Fig. 7(f)). This technique exposes the front edge of the pseudotumor deep margin during subtask 2, while minimizing deformation from vacuum tissue retraction. In manual resections, the otolaryngologist uses a handheld Bovie electrosurgical pen, a surgical forceps, and a ruler, bypassingthe ASTR components. The ruler is actively used during resection to provide surgeons with dimensional feedback in millimeters.

[0100] B. Evaluation Metrics

[0101] Upon pseudotumor removal, images of the top, left, and right views are captured for all nine studies. Using a ruler in these images for pixel-to-millimeter scaling, we measure a resolution of approximately 0.02 mm. A custom Python script, developed using OpenCV libraries, processes the top view images to calculate surface incision errors. The incision error for each point on the actual incision contour (to, red in Fig. 9(a)) is determined by the distance to closest point on the pre-drawn pseudotumor contour (pdc, greyscale in Fig. 9(a)), subtracting the desired surface margin of 4.5 mm (due to a 5 mm offset and a 0.5 mm radius electrosurgical electrode). This is mathematically represented asinciSioncontour. Similarly, a custom Python script calculates depth dissection errors using the left and right view images. The dissection error for each point on the actual dissection contour (de, greyscale in Fig. 9(b), (c)) is measured by the distance to closest point on the surface contour (sc, greyscale in Fig. 9(b), (c)), subtracting the 4.5 mm depth margin. This is represented asmm for each point / on the dissection contour. The average, standard deviation, maximum, and minimum of surface incision and depth dis- section errors for each resection experiment will be calculated and reported. The two-tailed t-test will assess differences in incision and dissection errors between autonomous and manual resections, with a significance level of 0.05.

[0102] Clinically, successful tumor resection requires complete tumor removal with negative tumor margins. In this study, a glossectomy is deemed successful if: i) the entire pseudotumor is removed, ii) its surface contour remains outside the pre-drawn contour (i.e. minimum surface incision error > -4.5 mm), and iii) its depth contourdoesn't intersect the surface contour (i.e. minimum depth dissection error > “4.5 mm). We will report the procedure duration for each subtask within each experiment. We will also document the number of supervisor requested replannings, and the computation time for replannings at subtasks 1 -1 and 2-1 .TABLE 1Table of Results (A for Autonomous Resections, and M for Manual Resections

[0103] C Resu / ts

[0104] The results of all nine experiments are presented in Fig. 9 and Table I. In six autonomous experiments, the average surface incision and depth dissection errors were -0.73 ± 0.60 and 1.89 ± 0.54 mm, respectively. The average autonomous surgery duration for subtask 1 and 2 were around 1 min 21 sec and 8 min 10 sec, respectively. No emergency stop of the ASTR was triggered by the human supervisor. In three manual experiments, the average surface incision and depth dissection errors were -0.32 ± 0.60, and 3.02 ± 1.19 mm, respectively. The average manual surgery duration for subtask 1 and 2 were around 1 min 48 sec and 2 min 10 sec, respectively. All six autonomous and three manual glossectomies successfully removed the pseudotumors, with no positive tumor margin identified in any of the nine samples. Both autonomous and manual resections achieved sub-millimeter accuracy in the surface incision, with no significant difference (p = 0.098). While autonomous resections showed better deep margin dissection accuracy than manual resections (1.89 ± 0.54mm vs. 3.02 ± 1.19 mm), the difference wasn’t significant (p = 0.118). Our current autonomous resection results showed accuracy improvements over the previous study, which reported averages of -1.19 mm for surface incision, and -1.83 mm for depth dissection errors. Two- tailed t-tests indicated near significance for surface incision (p = 0,052), and definitive significance for depth dissection improvements (p < 0,001 ).

[0105] V. Discussion

[0106] In this example, we have identified several key observations from the resection results. First, both autonomous and manual resections of pseudotumors exhibit a trend of narrower average surface margins (-0.73 mm and -0.32 mm, respectively) and thicker depth margins (1.89 mm and 3.02 mm, respectively). The former could be influenced by the electrosurgical electrode vaporizing more tissue than its radius due to heat effects, while the latter might result from the stretching of dead tissues during the procedure, causing removed pseudotumors to be thicker than initially intended. Second, a spike in autonomous surface resection errors near the cameras’ perspectives suggests the need for improved camera setup, calibration, and dual-camera fusion in future work (Fig. 9(a)). Third, large autonomous depth resection inconsistencies occur at the front and back of the removed pseudotumor (Fig. 9(e)). The former may result from the initial deep margin dissection process using a 20 downward-tilting electrode angle, while the latter could be due to decreasing retraction force in the latter half of the dissection iterations. Fourth, subtask 1 -3 in manual resection is a repetitive task. Surgeons lack robot precision and vision feedback in units, and require repetitive ruler measurements for accurate margin control. As a result, autonomous incisions averaged 1 min 21 sec, compared to the manual incisions which took 1 min 48 sec. Fifth, the hand motion in manual resections is significantlyfaster than the ASTR’s preset 2 mm / sec speed for precise movements. Consequently, the average time for deep margin resection was reported 8 min 10 sec in autonomous operations, compared to 2 min 10 sec for manuai procedures. Future work shouid evaluate and optimize the ASTR's speed settings to potentially decrease resection duration, while achieving a comparable or improved resection accuracy.

[0107] Automating a complex surgery entails the autonomous and continuous execution of a detailed, lengthy sequence of predefined surgical workflows. Errors can accumulate throughout the procedure, as the image processing and robot control algorithms in each sequence interact. Consequently, a failure in one part of the algorithmic sequence can jeopardize the entire procedure. Ensuring near 100% success rates and improved accuracy for each algorithm is crucial for both safety and effectiveness. Another challenge arises from the depth camera’s inaccuracy at incision sites, where deep gaps and small openings hinder the camera from providing millimeter-accurate depth information. Robot kinematics can be employed alongside precise camera data to address this issue. For instance, we use the accurate depth of the pseudotumor surface at subtask 2-1 , precise color projection of the pseudotumor deep margin at subtask 2-4, and accurate kinematics of the vacuum grasping robot during the six states of subtask 2 to determine the accurate depth of the pseudotumor deep margin. The ASTR and experimental design exhibit two primary limitations. First, the pseudotumor model, while useful, is an oversimplified representation of HNSCC, resembling a tumor chip. Consequently, our evaluation metrics, tailored to a pseudotumor with a TBR of 1 , might not translate directly to clinical scenarios, particularly for pathological evaluations. Second, by focusing on the midline partial glossectomy, which mirrors an open surgical environment without simulating oralaccess constraints, we haven't fully address challenges related to HNSCC resection in deeper regions of the head and neck.

[0108] VI. Conclusion

[0109] This example reports the first accurate supervised autonomous tumor resection using animal tissues. Our primary contributions include the development of the ASTR for autonomous HNSCC resections, where we integrate vacuum grasping as an alternative to traditional forceps, aiming to reduce potential tissue damage and subsequent cancer recurrence. We also develop a safe and reliable control strategy for the autonomous robot system under human supervisory. Furthermore, we establish an ontology- based research framework for complex autonomous robotic surgeries, which bridges medical and programming knowledge, offering a structured and accurate representation of procedures and ensuring their robotic implementation. This framework pro- vides autonomous robotic surgery data for SPM, extending its reach to include both human and robot operators, and holds the potential to reshape future OR management practices. Lastly, our results from supervised autonomous tumor resections on porcine tissues demonstrate a precision that competes with, and at some aspects exceeds, manual resections performed by an experienced otolaryngologist. This also underscores the feasibility of the proposed ontology-based research framework.

[0110] For future work, the ASTR will be utilized to perform HNSCC resections in more realistic and challenging scenarios, such as an autonomous tongue base tumor resection with blood obfuscation and spatial constraints. The ASTR will be integrated with advanced deep learning-based vision techniques to improve the system’s perception capabilities. We will also develop more subtask planners, and employ a more complex control scheme to provide decision-making capabilities, such aspausing resections to ciear blood. Force sensors will be integrated to the instrument tips to measure the cutting force, allowing path planning adjustments to increase safety. These future directions will enhance the efficacy of the proposed method in real- world surgical settings.

[0111] Example 2: Enhancing Surgical Precision in Autonomous Robotic Incisions via Physics-Based Tissue Cutting Simulation

[0112] I. Introduction

[0113] In this example, we addressed the identified research gap by developing an accurate soft tissue cutting simulation, and integrating it into robotic control loops. This integration allowed for the preoperative prediction and compensation of deformation resulting from tool-tissue interactions, guiding our robotic system towards autonomous planning and execution with improved precision. We conducted and evaluated autonomous incisions on ex vivo animal tissues in clinically mimicked scenarios, affirming the efficacy of our methodology. Our contributions include: 1 ) Developing a novel physics- based simulated surgical scene using SOFA that accurately models tissue deformation during cutting. This model not only considers the mechanical properties of tissues but also the sharpness and speed of the cutting tool as key factors of deformation extent. It assesses errors arising from tool-tissue interactions, and refines cutting plans to better align actual incisions with the intended surgical plan. 2) Demonstrating a statistically significant increase in cutting precision through ex vivo experiments utilizing our approach. Additionally, our method has shown improvements in maintaining the intended shapes and locations of cuts, which could potentially decrease the likelihood of adverse oncologic outcomes. This is the first work that attempts to improve clinically relevant soft tissue cutting accuracy using data gathered from a physics-based soft tissue cutting simulation.

[0114] II. Methods

[0115] A. Characterization of Electrosurgical Incision Task

[0116] Surgical cutting, involving tasks like incision, dissection, and excision, commonly utilizes instruments such as scalpels, scissors, harmonic scalpels, and electrosurgery devices. In particular, our study addresses incisions executed via electrosurgical techniques, pivotal in numerous surgical procedures. Surgical incision involves a precise cut breaching the epidermis for diagnostic or therapeutic intervention. Electrosurgery, employing high-frequency electrical currents to cut or coagulate tissue, is a standard technique in pathology management, where precision is key to successful oncologic outcomes. Contrary to most research in robotic cutting control of deformable objects focusing on straight or near- straight line cuts using general-purpose knives, our study explores more complex and clinically relevant scenarios. We investigate incisions along irregular, non- planar contours using a monopolar electrosurgical needle electrode, commonly employed in surgeries like nephrectomy, hepatectomy, cholecystectomy, and glossectomy.

[0117] This example focuses on midline partial glossectomy, essential for early-stage tumor resection on the tongue’s superior midline surface, classified as type 1 glossectomy or mucosectomy. Our approach involves defining the tumor surface shape based on a clinical case report, adjusting it to a 2cm diameter to match T1 / T2 stage cancer dimensions. We also adhere to a 5mm resection margin both laterally and in depth, following standard clinical practices. Surgical setup often includes oral gags and traction sutures for anterior tongue access, simulating an open surgical environment. For experiments, we opt for porcine tongue specimens, chosen for their anatomical resemblance to human tongues and ease of accessibility. We 3D print a sample holder mimicking the jawbone and mouth floor, with a clamp to stabilize theposterior tongue. Each tongue sample’s tip was sutured and extended by 2cm using a linear stage, replicating clinical tongue positioning. The experiment’s goal, illustrated in Fig. 11 , is to achieve accurate autonomous electrosurgery along the identified incision path.

[0118] B. Incision Following Preoperatively Intended Paths

[0119] The robotic incision accuracy following preoperatively intended paths on porcine tongues is evaluated, with the findings set to inform subsequent simulation models.

[0120] 1) Clinically Relevant Tissue Incision Speed: Cutting speed is a critical factor that influences both tissue deformation and the duration of surgery. We enlisted an experienced otolaryngologist to determine a clinically applicable speed for electrosurgical incisions. Utilizing two porcine tongues from a local grocery (H Mart, Lyndhurst, NJ), each tongue was marked with three 10cm lines for incision. The surgeon employed an electrosurgical pen (Bovie, Clearwater, FL) to incise at a consistent depth of 5mm, as per our study’s parameters. The duration of each cut was timed by reviewing the recorded video. The cutting speeds recorded were 11.11 , 6.66, 7.14, 7.14, 6.25, 5.00 mm / s respectively, resulting in an average speed of 7.22 ± 1 .89mm / s. Based on these findings, a rounded cutting speed of 7mm / s was chosen for subsequent experiments.

[0121] 2) Robotic Experimental Setup: Leveraging our prior development of the autonomous system for tumor resection (ASTR), this example’s testbed comprised five main components (Fig. 10. a): 1 ) a 6-DOF UR10e manipulator (Universal Robots, Odense, Denmark), 2) a customized laparoscopic electrosurgical instrument, 3) an RGBD camera (D405, Intel, Santa Clara, CA), 4) custom robot operating system (ROS) programs for tissue tracking and incision planning, and 5) a tongue sampleholder. A standard monopolar needle electrode with a 25mm length and 1mm diameter(Bovie, Clearwater, FL), a grounding pad, and an electrosurgical power generator (ASG-300ESU, DRE Veterinary, Louisville, KY) were key components. To manage electrosurgical smoke, we utilized both a portable smoke evacuator (Smoke Shark, Bovie, Clearwater, FL), and an air purifier (GC Multigas, IQAir, Goldach, Switzerland). The sample holder was described in Section ll-A, and the software’s role in the experimental process will be presented in the following sections.

[0122] 3) Electrosurgical Cutting Power Setting: Electrosurgical cutting power significantly affects the electrode’s sharpness, akin to the blade of a scalpel. Adequate power ensures the electrode cuts smoothly, reducing tissue bulging and deformation, which introduce inaccuracies. While higher power increases sharpness, it also risks thermal damage and tissue denaturation, negatively impacting post- surgical recovery and organ functionality. To identify the lowest effective power, we utilized four grocery porcine tongues (H Mart), each subjected to incisions at varying powers of 25, 30, 35, and 40 Watts, corresponding to our experimental setup as described in Section ll-A. A custom C++ ROS program enabled the Intel D405 camera to track the incision path through color thresholding, plan electrosurgical incision paths with 5mm depth, and guide the robot for execution at 7mm / s autonomously. Visual observations revealed that at powers of 35 W or higher, there was no noticeable tissue bulging, while at 4CW , discoloration suggested tissue denaturation. Consequently, 35W was chosen for subsequent experiments.

[0123] 4) Experimental Design to Evaluate Precision: Two ex vivo porcine tongues were acquired (Animal Technology, Tyler, TX), and CT-scanned (Loop-X, BrainLab, Munich, Germany). To assess incision accuracy, two samples were prepared as described in Section ll-A. The same C++ ROS program was utilized fortracking the marked path, generating the incision plan, and guiding the robotic execution. Key parameters, informed from clinical reports and preliminary findings, included the same near 2cm diameter tumor contour, a 5mm lateral margin, a 5mm incision depth, a 7mm / s cutting speed, and a 35 W electrosurgical power. The detailed evaluation metrics are reported in Section III-A.1.

[0124] C. Development and Optimization of Tissue Cut Simulation

[0125] This section outlines the development of soft tissue cutting models and simulated surgical setups. It introduces the optimization of simulation parameters to mirror deformation observed in real tissue experiments, and details our approach to refining incision paths for potentially improved precision.

[0126] 1) Simulation Framework Selection: To accurately replicate tissue cuts, specific criteria were established: 1 ) replicating clinical setups through the importation of tool and tissue models alongside mechanical and boundary constraints, 2) employing FEA to model nonlinear deformation, and 3) facilitating the division of volumetric objects through cutting interactions. The search for a suitable physics- based simulation framework to meet these criteria led to the evaluation of FEBio, Asynchronous MultiBody Framework (AMBF), Isaac Gym, and SOFA. FEBio focused on deformation and stress analysis, only allowing scripted cuts in predetermined separation locations. AMBF lacked support for volumetric cutting. Isaac Gym was used for data generation in DeformerNet, however, despite DiSECt’s recent advancements, it lacked a dedicated open-source plugin for simulating complex surgical cuts. Other platforms like Gazebo and Ignition were also considered but fell short in soft body mechanics and lacked the necessary cutting features. Ultimately, SOFA was chosen for its extensive use in medical simulation and soft robots, offering comprehensive FEA capabilities and the SofaCarving plugin, enabling realistic volumetric cutting simulations.

[0127] 2) Soft Tissue Cutting Simulation: SOFA imports tissue models as volumetric meshes consisting of smaller tetrahedral elements. The SofaCarving plugin simulates cutting by removing a tetrahedron of the carving surface mesh upon contact with a carving tool. Rather than incorporating tetrahedron subdivision, which is limited to straight-line cuts on tetrahedra, a volumetric mesh densely packed with tetrahedra is preferred to create a refined incision geometry. In these models, the area designated for cutting is densely populated with small tetrahedra, while larger tetrahedra fill the remaining space. This technique improves performance by minimizing unnecessary elements while producing a precise incision edge.

[0128] As aforementioned, the surgical instrument sharpness plays a critical role in tissue incision accuracy due to material deformation. Given that no real-world tool is perfectly sharp, the SofaCarving’s approach of instantaneously removing mesh elements upon contact appears unrealistic. To address this, we introduced a tool dullness simulation parameter, defined as the required contact duration between the carving tool and a tetrahedron for its removal. A greater dullness leads to more significant tissue deformation and lower incision precision.

[0129] 3) Simulated Surgical Scene: The experimental surgical scene was recovered in simulation (Fig. 13). A 3D model of the electrocautery tool was supplied to the scene as the cutting tool model. Porcine tongues were CT-scanned using Loop-X, and converted to volumetric meshes following the description in Section II-C.2. Using the SOFA object TetrahedralCorotationalFEMForceField, the tissue object was empirically assigned a Young’s modulus of 70kPa and a Poisson’s ratio of 0.49. The electrocautery collision model and tongue collision model acted as the carving tool and carving surface, respectively. The tongue model was given a posteriormechanical constraint and underside support platform for stabilization, and the tongue model’s tip was extended 2cm from the initial position.

[0130] 4) Optimizing Simulation Parameters for Real-to-Sim Reg- ist ration:Assuming that physics-based simulations cannot perfectly replicate real-world phenomena, we further tuned two simulation parameters to align the magnitude of tissue deformation observed in simulations with actual results. To achieve this real-to- sim registration, input simulation tool speed and tool dullness were optimized across simulated experiments. Initial robotic incision experiments described in Section II-B.4 were used as a benchmark for real-world outcomes. We varied input tool speed and tool dullness on simulated incisions performed on two tissue models representing the two CT-scanned porcine tongues from the real-world experiments in Section II-B.4. The experiments encompassed tool speeds of 1, 5, 9, 13, 17mm / s, and dullness levels of 1, 5, 9, 13, 17, culminating in a total of 25 experimental settings. Results and equations used for the real-to-sim parameter registration are reported in Section III-A.2.

[0131] 5) Improving Preoperatively Intended Tool Paths Using SimulatedData: Following the sim-to-real registration and achieving more accurately simulated tissue deformability, we simulated incisions on newly CT-scanned and uncut porcine samples. Our objective was to predict deformation-induced errors and refine incision plans preoperatively for robotic surgery, thereby completing a real-to-sim-to-real workflow to enhance accuracy. Two new porcine tongues (Animal Technology) were CT scanned using Loop-X, and converted into volumetric meshes for simulation. The tumor shape was imported into the simulation, and an initial tool path was set with a 5mm lateral and depth offsets, respectively. Subsequent simulated procedures applied tool speed and dullness settings refined through real-to-sim registration, with incision result evaluation metrics explained in Section III-A.2. Evaluations led to theadjustment of waypoint coordinates on each tool path, shifting them towards or away from the tumor center to correct for specific incision errors identified at those locations. For example, waypoints too close to the tumor were moved away from the tumor center by the measured error distance to counteract deformation-induced inaccuracies, and the opposite was done for waypoints too far. Ideally, the expected incision error post-simulation adjustments would be less than the 0.5mm radius of the electrosurgical electrode, beyond which further precision improvements would be impractical. Noteworthy, empirical findings indicated that these adjustments should be applied only once to prevent unpredictable oscillations and potentially decreased accuracy in subsequent robotic executions on actual porcine tongues.

[0132] D. Incision Following Simulation-Improved Paths

[0133] Real tissue experiments are designed to validate the efficacy of our proposed simulation-improved incision planning approach. It completes the real-to- sim-to-real workflow.

[0134] 1) Sim-to-Real Registration: The simulation-improved incision path, created in the simulation coordinate frame, required alignment with the real-world coordinate frame to guide our electrosurgical robot, necessitating a sim-to-real registration. After stretching the tongue in the simulation, a point cloud was produced from the volumetric mesh to represent the tongue tissue. Following the complete simulation, this was coupled with the tumor edge and improved incision path. This tongue point cloud was processed to retain the top surface, using a custom Python program with Open3D libraries. In parallel, the same CT-scanned real porcine tongue was prepared in the 3D printed holder, as described in Section II- A. Its superior surface point cloud was cropped and acquired from Intel D405 streamed point cloud using a custom ROS GUI interface. Subsequently, the coherent point drift (CPD)method, augmented with scaling, was applied to achieve registration, employing a Python-based implementation of this algorithm. This process generated a transformation matrix that bridged the simulation and real-world coordinate frames, enabling the precise overlay of the tumor edge and improved incision path from the SOFA simulation onto the actual surgical context.

[0135] 2) Experimental Design to Evaluate Precision: To assess the effectiveness of robotic incision following simulation- improved versus preoperatively intended incision paths, all key parameters as described in Section II-B.4 were kept consistent. The two newly CT-scanned ex vivo porcine tongues (Animal Technology) were prepared as delineated in Section ll-A. Since the paths were derived from simulation, the method of manually drawing tumor edges on porcine tongues was impractical. Instead, a C++ ROS program projected the simulated tumor edge and improved incision path onto the tongue sample’s surface. The program also implemented a 1 mm inferior offset to the tumor edge, guiding the electrosurgical robot to mark the edge at 2mm / s with minimal tissue deformation. The 5mm incision depth was then applied to the projected incision path for autonomous execution. The evaluation metrics are detailed in Section III-A.1.

[0136] In this example, we utilized a four-layer metric to evaluate the performance of autonomous incisions for tumor resection surgeries. First, an incision was considered as successful if the aic didn’t intersect with and remained exterior to the tec, reflecting a negative margin status where no tumor cells are present at the resected margin. Second, we quantified the dimensional accuracy of the surface margin using a custom Python script employing OpenCV libraries to process the image data and compute surface incision errors. The error for each point on the aic was calculated as the shortest distance to the greyscale tec, subtracting theintended surface margin of 4.5mm (considering a 5mm incision margin and a 0.5mm electrosurgical electrode radius), and then calculating the absolute values. Mathematically, the error is presented amm for each point i on the aic. Additionally, the two-tailed t-test will assess differences in these errors before and after applying our simulation- based adjustments, with a significance level of 0.05. Third, shape similarity and centroid displacement between the aic and greyscale tec of each porcine sample were evaluated to determine if the desired incision shape and location were achieved. Shape matching scores were calculated using OpenCV libraries based on Hu Moments. Let D(aic, tec) be the shape difference between aic and greyscale tec, andHi'cbethel log transformed Hu Moments for shapes aic and tec, respectively. The difference measure was quantified as D(aic,tec) -a va|ue ofzeroindicating perfect shape match and increasing values denoting greater disparities. Centroid displacements were determined by calculating the Euclidean distance between the centers of mass for the aic and greyscale tec of each sample. Fourth, this study further explored oncologic safety by analyzing sites with excessively close margins, which literature suggested may negatively influence local recurrence rates and disease-free survival (DFS). Similar to the second metric, incision errors were quantified by computing data analysisviewpoint, incision sites with errors exceeding -4.5mm were considered to have negative margins. However, clinical research indicated a negative margin cutoff of 2.2mmas prognostic of diminished DFS. To elaborate, incision sites with errors ranging from -4.5mm to -2.3mm (-4.5 + 2.2 = -2.3mm) resulted in excessively close margins. Such margins might encapsulate microscopic tumor cells that are challenging todetect, elevating the risk of false negative margin assessments and, consequently, worse DFS.TABLE ITable of Simulated Incision Results for Real-to-Sim Parameter Registration (Note:Values Round for Space)

[0137] 2) Simulated Incision Performances: After completing a simulated incision, tongue mesh tetrahedra removed during the procedure were recovered and skeletonized to produce a midline incision contour. This recovered incision contour is evaluated as the aic for simulation results reporting.

[0138] Simulated incision results were evaluated using a two-layer metric, mirroring real-world result reporting. First, an incision was deemed successful if the maximum error point stayed within 4.5mm, indicating a negative margin status, and if the simulation produced a closed-loop incision. Second, similar to real-world experiments, the error for each point on the aic was calculated as the shortest distance to the tec using absolute values, as mathematically described in Section III-A.1.

[0139] Simulated incisions results, used for real-to-sim parameter optimization as described in Section II-C.4, were evaluated and registered to real-world results in Section III-B.1 , using a three-layer metric. First, the simulation’s success was measured by the achievement of a negative margin and the completion of a closed- loop incision. Second, the absolute average incision error across two samples under consistent simulation parameters was examined, and third, the maximum incision error was assessed. Mathematically, real-to-sim registration is defined asII 0-o- l|max(aic) for each pOjnt2 onthe aic, reported as a percentage of the baseline value in Table I. Unsuccessful incisions are reported with a 0% match since all baseline real-world experiments were successful. A real-to-sim parameter registration of 103.6% motivated the selection of simulation parameters: a tool speed of 13mm / s and a dullness setting of 9 for subsequent simulations.

[0140] B. Results

[0141] Quantitative and visual outcomes are detailed in Fig. 14, 15 and Table II, as elaborated in subsequent subsections.

[0142] 1) Incision Results Following Preoperatively Intended Paths: As shown in Fig. 14.c,d, and S1 , S2 in Table II. Two out of two ex vivo incisions were successful with negative margins. The average absolute incision errors were 1.57 ±0.66mm and 1.91 ± 0.90mm, respectively, leading to a combined average absolute error of 1.73 ± 0.80mm. Shape similarity, assessed through Hu Moments, yielded scores of 0.08 and 0.12 for each specimen. Centroid deviations were measured at 1.96mm and 2.22mm, respectively.

[0143] 2) Simulated Incisions: The simulation-based incision path improvements were conducted on two porcine tongues, which had been CT-scanned and converted into mesh files, corresponding to the samples shown in Fig. 14, and S3, S4 in Table II. Pre-tool path improvement, two out of two simulated incisions were successful with negative margins and closed-loop incisions. The absolute incision errors were 1.84+3.53mm and 2.14+4.41mm, respectively, leading to a combined average absolute error of 1.99+2.20mm, Post-tool path improvement, two out of two simulated incisions were successful with negative margins and closed-loop incisions. The absolute incision errors were 0.33 + 1.16mm and 0.33 + 1.02mm, respectively, leading to a combined average absolute error of 0.33 +0.76mm,

[0144] 3) Incision Results Following Simulation-Improved Paths: As shown inFig. 14, and S3, S4 in Table II. Two out of two ex vivo incisions were successful with negative margins. The average absolute incision errors were 1.37+1.19mm and 1.57 + 0.97mm, respectively, leading to a combined average absolute error of 1.46 + 1.09mm. Shape similarity, assessed through Hu Moments, yielded scores of 0.06 and 0.06 for each specimen. Centroid displacements were measured at 1.58mm and 1.07mm, respectively.TABLE IITabe of Incision Results on Porcine Samples

[0145] The box plots depicted in Fig. 15.a,b reported both the absolute incision errors and the directional discrepancies of incision errors, and the Fig. 15.c,d displayed error distributions and excessively close margins. Improvements were observed when comparing results post-application of our method to pre- application, including: 1 ) a statistically significant decrease in average absolute incision errors from 1.73mm to 1.46mm (p < 0.001) based on two-tailed t-test, 2) enhanced fidelity in replicating intended incision geometries with shape matching scores improving from 0.10 to 0.06, 3) more precise incision placements with centroid shifts reducing from 2.09mm to 1.33mm, and 4) prevention of excessively close margins associated with worse DFS.

[0146] 4) Limitations Highlighted by a Case of Reduced Accuracy: A third porcine tongue subjected to the complete real-to-sim-to-real workflow exhibited decreased incision accuracy, marking it as a failed case and revealing a limitation of our proposed method. The sample underwent CT scanning and simulation with optimized parameters aimed at incision path enhancement. Despite these efforts, the simulated absolute incision error marginally improved from 1.93 + 3.57mm to 0.63 + 1.97mm, failing to meet our anticipated 0.5mm accuracy threshold detailed in Section II-C.5. Subsequent simulation iterations adjusted incision waypoints closer or further from the tumor, using the same explained method, attaining an improved accuracy of 0.33+0.85mm, However, when this simulation-modified path was executedrobotically on the actual porcine tongue, the absolute incision errors worsened to 2.11± 1.64mm, with a shape matching score of 0.10, and a centroid displacement of 1.62mm. These outcomes not only showed reduced incision precision but also poorer maintenance of incision shape and location, thereby underscoring a critical limitation of our approach as highlighted in Section II-C.5.

[0147] III. Discussion

[0148] In previous research, we achieved sub-millimeter precision in incision accuracy at a slow cutting speed of 2mm / s, which resulted in extended surgery duration and increased thermal tissue damage. Building upon this, our current example explores a clinically more relevant speed of 7mm / s, at the cost of increased incision error. Despite simulation- based improvements, we could not replicate the precision of slower speeds at this higher rate. This finding necessitates a discussion among surgical professionals to find an optimal balance between efficiency and accuracy, tailored to specific clinical needs and priorities.

[0149] The challenge of accurately simulating soft tissue de- formation for robotic surgery remains significant, as evidenced by research primarily concentrating on simpler linear cuts from 2007 to 2023. The variability in patient-specific tissue properties further complicates the development of a universal physics-based model for precise organ deformability prediction. Our study, despite its limited sample size, serves as a preliminary investigation rather than comprehensive research aimed at definitive precision enhancement. It underscores the importance of advancing towards more realistic surgical simulations and developing models that account for individual tissue differences.

[0150] Concerning potential discrepancies between preoperative and intraoperative imaging, our method proposes scanning patients on the day ofdiagnosis, followed by overnight simulation (approximately 1 -2 hours using 2020 MacBook Pro) for next-day surgery. Although this does not account for real-time surgical adjustments due to unanticipated tissue retraction, it offers a preparatory framework that improves upon existing methods. The inability to modify surgical plans in real-time remains a significant limitation.

[0151] The failed case study underscores another limitation of our method: its effectiveness largely hinges on the initial simulation’s accuracy. Our empirical findings suggest that even if desired precision in simulated incisions is achieved after several iterations, the robotic execution on porcine tongues may result in diminished precision, likely due to unpredictable oscillations accumulated through simulation adjustments. This illustrates that our method is selective in terms of patient eligibility, which, although a limitation, does not invalidate the applicability of our method to surgical procedures. Such selectivity exists in many existing surgical techniques like LAS IK eye surgeries, where individuals with thin corneas may not be suitable candidates.

[0152] IV. Conclusion

[0153] In conclusion, this study demonstrates a successful in- tegration of physics-based tissue cutting simulations with autonomous robotic execution, targeting the precision of incisions along complex contours in animal tissues. We introduced an innovative simulation model on the SOFA platform, featuring a concept of simulated cutter dullness to mirror the deformability observed in real tissue experiments accurately. This model was utilized to foresee and compensate for the deformations arising from tool-tissue interactions preoperatively, leading to enhanced surgical path planning. The findings confirm our approach enhances incision accuracy, reducing the average absolute error from 1.73mm to 1.46mm (p < 0.001) and improves the preservation of intended cutting shapes and locations — shape matching scoresimproved from 0.10 to 0.06, and centroid shifts decreased from 2.09mm to 1.33mm.This precision in execution potentially improves DFS by reducing risks associated with margins closer than the clinically suggested threshold of 2.2mm.

[0154] Looking ahead, our research will explore more sophisti- cated simulations of deep tumor margins, extending beyond mere incision to encompass entire tumor resection processes. We aim to refine the simulation’s efficiency, enabling patient- specific surgical planning on the day of the surgery itself. Furthermore, we will explore innovative solutions to broaden the applicability of our method to a wider patient demographic, reducing the current limitations regarding patient selection.

[0155] Example 3: Autonomous Closed-Loop Control for Robotic Soft- Tissue Electro-Surgery Using RGBD Image Guidance

[0156] I. Introduction

[0157] In this example, an autonomous system that incorporates tissue tracking was proposed to enable closed-loop robotic soft- tissue cauterization. The proposed system aims to enhance the margin precision via the feedback of tissue tracking to reduce tissue deformation in the cautery process. First, a robotic control workflow of closed-loop cauterization was designed, shown in Fig. 16. a, in which a 3D tissue tracker estimates the movement of tracked keypoints followed by updating control actions via a path planner and a velocity controller. Second, the 3D tissue tracker was developed by integrating a CoTracker to enable markerless pixel-level tracking capability on soft tissue. Additionally, an algorithm for tool occlusion was developed within the 3D tissue tracker for online tracking throughout the robotic cautery procedure. Next, a velocity controller was developed using a fuzzy logic approach, incorporating tissue tracking as feedback to control the feed rate of cautery motion. Lastly, the robotic control workflow, shown in Fig. 16. b, was designed ontothe Autonomous Surgery Interface (ASI) to achieve system-level implementation for online soft-tissue tracking and feedback motion control. The feasibility of the system was assessed through autonomous incision cauterization on cadaver pork tongue and the results were compared to the cautery outcomes obtained using open-loop control strategies. The remainder of this paper is organized as follows: the robotic system as well as the design of the control methods are detailed in Section II, the results of tests via the autonomous system are presented in Section III and discussed in Section IV, and Section V concludes the paper.

[0158] II. Method

[0159] A. Problem Formulation

[0160] Fig. 17 shows actions using a cautery pencil to perform an incision in electrosurgery on a cadaver pork tongue tissue. In the clinical procedure of tumor resection, a surgeon first identifies the approximate tumor location using pre-operative imaging followed by removing the tissue in the tumor region via a resection procedure. This study focuses on the resection task by giving the circular contour drawn on the tissue surface representing a pseudo tumor region, shown in Fig.17. a. The surgeon used measurement tape and a cautery pencil to mark the dots on the tissue surface with a distance opposed to the contour, with results shown in Fig.17.b. Next, the resection procedure is carried out by inserting the electrode of the cautery pencil into the tissue with a certain depth followed by connecting each marked dot, where the cautery pencil delivers energy to vaporize tissue near the pencil tip. The surgeon relies on visual cues observed from the tissue surface to adjust the motion of the cauterization to maintain a precise cutting contour (Fig.17.c).

[0161] B. Robotic System Setup

[0162] The experiment setup is shown in Fig. 1 .b. The robotic system consists of A 7-DOF KUKA LBR Med (KUKA AG, Augsburg, Germany) attached to a customized electro- cautery tool. An Intel Realsense D405 (Intel Corp., Santa Clara, California) was used as an imaging system with the 3D perception capability of detecting tissue surface information. The D405 camera also provides a 2D-3D image mapping functionality, therefore the coordinate system of the 3D image can be extracted directly from the pixel location in 2D image space. A hand-to-eye calibration registered the 3D camera coordinate system onto the robot coordinate system via a touch-point registration with a checkerboard. The 3D locations of the target point w.r.t. robot frame were obtained and used by a path planner to guide the robotic arm.

[0163] C. Autonomous Cauterization Workflow

[0164] Algorithm 1 details the workflow of autonomous cauterization. In the initialization, the user manually specified the 2D locations on the image to retrieve 3D landmarks L and the path planner took L to generate a trajectory W (detailed in Section ll-D). We defined k as a counter and will be increased with every completion of reaching a waypoint in W . The robot system iterated through the W to control the electro-cautery tip moving from the start to the end of the waypoint in W . The motion planner took pk (i.e., current target waypoint) and Vk (i.e., planned velocity) as input to plan the trajectory of the tip motion. As the motion was executed between consecutive waypoints, the tissue tracker monitored the tissue deformation Dk(detailed in Section II- E). Upon completing each waypoint segment, Dk was used to update the feed rate Vk+i (detailed in Section I l-F) for the next iteration, serving as thenext control input for moving the electro-cautery tip. In addition, motion planner issued relative motion with amount Dk from pk toward pk-i to alleviate the tissue deformation. Autonomous Cauterization Workflow wLandmarksQ > Manual Selection ewPaMJL) > Path Planner< W.sizeQ do5: Pk F- W[k]6: Movefaky Vk) > Motion Planner7: Dk F" NewDeformationQ > Tissue Tracker8: tOe-pi F- FeedRateUpdate(Dk) > ControllerM ove (pk > i?Vk , Dk ) > Motion Planner10: k F~ k + 111: end while

[0165] D. Path Planner

[0166] The goal of the path planner is to generate a trajectory that connects each marked dot in 3D, shown in Fig 17.b. A set of landmarks L (red markers in Fig.3. a), containing it G R2, was manually specified by the user on the 2D image. In this example, ten landmarks were needed based on pattern size and the surgeon’s prior experience. The order of A was defined as starting from the image top-mid landmark and increasing sequentially in a counter-clockwise direction. An additional set of landmarks M (green markers in Fig. 18. a), containing mi e R2, was generated by the path planner via Eq. (1 ) and (2) / C U?2 II II where is the centroid location of L, d & N is a pixel offset, and II 112 denotes asEuclidean norm. The point cloud of the target tissue is denoted as Ptissue which is generated via the Intel Realsense D405 camera. The 3D location of li and rrn are denoted as 'Ph £r3audpmier3» respectively. Dijkstra algorithm was utilized to compute trajectories on the Ptissue from pitto piw. An equal-spacing algorithm was applied to downsample the path with an equally spaced waypoint G R3within the trajectory W shown in Fig 17.b.

[0167] E. Markerless Tissue Tracker

[0168] Recent years have seen significant research interest in soft tissue tracking driven by advancements in machine learning technologies. The markerless soft-tissue tracking techniques are intriguing as they eliminate the need for deploying fiducial markers to the scene, particularly beneficial for intra-operative surgical applications. In this study, we built a markerless tissue tracker for autonomous soft- tissue cauterization by integrating CoTracker to achieve online tracking for the location change of M (green markers in Fig. 18. a) in image frames. CoTracker is a transformer based model capable of tracking a group of pixel points in a frame jointly across an image sequence. The neural tracker is flexible in tracking arbitrary points selected in the image at any spatial location and time, which fits well in theautonomous workflow of robotic cautery. The markerless tissue tracker built in this study streams images directly from the RGBD camera to enable online tracking capability. The tissue tracker took n image frames from k - n to k and estimated M on k where k is the timestamp and n was set to 8 to maximize the tracking update rate.

[0169] One limitation of CoTracker is that if the tissue tracker processes short windows of points at a time, the tracking could fail for points that stay occluded for the length of multiple sliding windows. Therefore, additional occlusion handling was developed in this study for the tissue tracker for robotic cauterization to avoid tracking failure. We defined the tip and base location of the electrode, shown in Fig. 19. a, asw.r.t. robot coordinates. Fig. 19.b illustrates the concept of algorithm. Because the camera was registered to the robot coordinate system via hand-eye calibration, both PeilPand Pebasecan be transformed into eup € R2and ebaseG R2w rt. camera frame, respectively. Givenebase andebaseand etip in the camera scene, a linear function A was defined. Moreover, an additional linear function h is defined orthogonal to f passing through ebase. A parameter gmj is referred to as which side of h does a landmark mi lie. If h(mi) is less than 0, we considered gmj e Gbase, and if h(mi) is greater than 0, we considered gmj eGtip. An orthogonal distance from each mi to f was defined as dmj,orth viaEq.(3)vector from etiPto ebase,G r2is a vector from etip to mi, and II II2denotes as Euclidean norm. If gme GtiP, vi is defined from ebase to etiP, and vmj is the vector from ebase to mt.

[0170] In addition, if g™ / e GtiP, a projection distance dmj,proj is computed via Eq. (4)

[0171] Once dm.proj and dmhOrth were computed, parameters phase and ptiPwere defined as a threshold for determining whether m, was occluded by the cautery tool in the scene via (5) and was specified prior to the experiment.

[0172] Examples (mi, rm, m3, and m4) are illustrated in Fig. 19.b. Since mi eGbase and dm±,orth < phase, mi is considered occluded by the tool (i.e., Om± =True). Next, since m2e GaPand dm2,orth < paP, m2is not considered occluded by the tool (i.e. , Om2 = False). For m3and m4, even though both landmarks are inGtiPand are satisfied dmj,orth < paP, dm3,proj does not satisfy dmj,proj < 1 1 vi| |2;therefore, m3is not considered occluded by tool but m4is considered (i.e.,Om3= False, Omi= True).

[0173] We designed the tracker to estimate the 3D change of the landmark locally near the tip. We defined N as the subset of M where all rrn e N satisfied Omj= True. The tracker computed 3D coordinates of rrn e N and the 3D coordinate of pmj was determined by finding the closest projected tissue point cloud on the 2D location of the tracked landmarks, given intrinsic and extrinsic camera parameters. Finally, the tracker computed the mean magnitude of the position change of landmarks via Eq. (6)where k is the timestamp and n is the total number of land- marks in N . Dk indicates the level of landmarks alternation.

[0174] F. Velocity Controller Design

[0175] Considering a process of mono-polar cauterization illustrated in Fig.20. a, there are three factors that affect the level of tissue deformation in the process: the power level of the electrocautery (i.e. , how much current is applied to dissect the tissue), cutting depth of the cauterization (i.e., how deep the electrode is inserted into the tissue), and feed rate of the electrode (i.e., how fast the movement of the electrode tip). For instance, as a high power level is set with a low feed rate and shallow depth in the cauterization, the electrode vaporizes the surrounding tissue faster before contacting the electrode tip, which results in a low magnitude of tissue contact force to deform local tissue. In contrast, if a low power level is set with ahigh feed rate and the same cutting depth, the tip might not be capable of burning tissue due to insufficient energy applied to the tissue, which further generates a large magnitude of force applied on the tissue and leads to deforming the contour of the removal margins. In this example, because the setting of the power level was fixed to insufficient energy applied to the tissue, which further generates a large magnitude of force applied on the tissue and leads to deforming the contour of the removal margins. In this study, because the setting of the power level was fixed to insufficient energy applied to the tissue, which further generates a large magnitude of force applied on the tissue and leads to deforming the contour of the removal margins. In this study, because the setting of the power level was fixed milling machining where feedback control is implemented for controlling the feed rate to regulate the contact force between the tool and the rigid workpiece, ensuring the machining quality of the target workpiece. Similarly, the robot controlled the speed of the cautery tip, mitigating deforming tissue to achieve precise resection results.

[0176] Inspired by the controller design in CNC machining, we implemented a 1 -DoF fuzzy controller to modulate the feed rate for robotic cauterization. Fuzzy control allows the experience and knowledge gained in previous experiments to be applied to the process control, which is effective in the scenario in which the model to be controlled is complex but the tasks are already successfully performed by humans. In this study, the design of a fuzzy controller consists of three fuzzy sets with a bell membership function for both input and output, shown Fig. 20. b. The x- axis represents a normalized variable (e.g., deformation or velocity) and the y-axis represents how completely the variable belongs to each fuzzy set. The controller first took the deformation and found the degree of membership on each input membership function. Each degree of membership was used to fill the area below thecorresponding output membership function. The controller output (i.e., v) was computed by finding the centroid of the filled union area of output membership functions. The values of Vmin, Vmax, Dmin, and Dmax were determined via experiments explained in the Sec. II-H.2.

[0177] G. Software Architecture

[0178] We extend the software architecture of the Autonomous Surgery Interface (ASI) with additional development for autonomous robotic cauterization. The ASI consists of a kernel, a graphical user interface (GUI), and peripheral components to allows an operator to perform cautery work- flow with a combination of autonomous and manual modes. In the experiment, the cautery workflow is built on GUI using a customized robot programming language, and the kernel sequentially interprets commands to determine which peripheral component executes the action. Three software components (e.g., 3D tissue tracker, path planner, and velocity controller) developed in this study were connected to the kernel with distributed infrastructure via Robot Operating System (ROS). The 3D tissue tracker was running on an RTX 2080 Ti, a graphical processing unit (GPU) from Nvidia (Santa Clara, California, U.S.). The motion module of ASI consists of Movelt and Pilz industrial motion planner to generate point-to-point and linear motion waypoints. A joint trajectory controller was chosen in position control mode that interpolates the waypoints into smooth trajectories. In the low-level layer, components of the OROCOS real- time toolkit (RTT) were deployed to achieve real-time control. Fast Research Interface (FRI) was used to synchronize the command with the KUKA driver and control the motion of the robotic arm.

[0179] H. Experimental Design and Evaluation Criteria

[0180] The hypothesis of this example is that the designed closed- loop autonomous control workflow will improve the robotic results of cauterization with a more precise incision margin. To test our hypothesis and proposed techniques, three ex- periments were carried out systematically via 1 ) evaluation of the 3D tissue tracker, 2) identification of parameters of the velocity controller, and 3) evaluation with autonomous robotic cauterization.

[0181] 1) Markerless Tissue Tracking: We first evaluated the online tracking performance of the 3D tissue tracker using a cadaver tongue tissue via a robotic setup, shown in Fig 21 .a. The tissue was placed on a rigid acrylic plate attached to the flange of the robot arm, and the robot was set to move at a constant translational motion 3mm in Cartesian space. The D405 camera was placed in a stationary location facing the tissue at a distance of 10cm which satisfies the minimum sensing distance of the 3D camera. Tissue patterns shown in Fig 21. b were drawn on the tongue surface and the 3D tracker tracked the locations of M . The tracked positions were recorded and compared to ground-truth motion (i.e., robot motion).

[0182] 2) Parameters Identification of Velocity Controller: The velocity controller for autonomous cauterization in Sec. II- F requires parameters Vmin, Vmax,Dmin, and Dmax to operate. To identify the parameters that best match the actual experiment, the parameters were experimentally derived by performing a small segment of robotic cauterization on pork tongues with a defined velocity profile to identify the relationship between the corresponding deformation. The velocity Vmin andVmax were chosen 1 mm / s and 2 / mm, respectively, based on prior robotic experiments.The cutting depth was set at 5mm and the cutting waveform was set at 20W. Two landmarks were specified on both sides of the cautery path and the path distance was chosen to match the spacing length of the waypoint segment in Sec.ll-D The 3D tissue tracker measured the average deformation D during the cauterization to determine the maximum and minimum deformation (i.e., Dmin, and Dmax).

[0183] 3) Autonomous Cauterization: The proposed robotic system performed closed-loop robotic cauterization on the cadaver tissue surface. The workflow of Algorithm 1 was used in the autonomous cauterization with the same cutting depth and cutting waveform used in Sec. II-H.2. Three cutting patterns were chosen in the experiment, shown in Fig. 21. b, with a 5mm margin between the pre-defined landmark and the contour. Ten landmarks were marked on porcine tongue tissue via transparent plastic sheets and were specified as L in the initialization of the cautery workflow in Algorithm 1. The results of the cauterization were evaluated by measuring cutting-margin accuracy, which is defined as the 3D differences between the reference contour of the tissue pattern and the actual cutting trajectory. To measure the cuttingmargin accuracy, a top-view snapshot was taken for each of the cautery results via an RGBD camera followed by comparing reference contour and cutting path in 3D, shown in Fig. 21. c. In addition, the closed-loop robotic results were compared to the openloop robotic cauterization in which the constant velocity Vmax was set for the robotic motion with no tracking feedback to modulate the feed rate (i.e., cautery workflow in Algorithm 1 without the steps of 7,8, and 9).

[0184] III. Results

[0185] A. Markerless Tissue Tracking

[0186] Fig. 23 shows the results tracking accuracy of the 3D markerless tracker with tissue moving 3mm in Cartesian space w.r.t. robot coordinate system.Ten landmarks were initiated (red sphere) and the final positions (green sphere) were recorded w.r.t. camera frame. The measurement is defined as an average Euclidean distance movement of the tracked landmarks between the initial and final positions. The table summarizes the measurement for tissue motion in each axis, with the total average motion 2.7 ± 0.3mm.

[0187] B. Autonomous Cauterization

[0188] The robotic system performed six autonomous closed-loop cauterizations on cadaver pork tongues, with two cautery results for each tissue pattern. In addition, a total of three open-loop robotic cauterizations were carried out using the same experimental setup, with one cautery result for each tissue pattern. Fig. 22 shows snapshots of a closed-loop robotic cauterization with feedback of tissue deformation and control of velocity input in waypoint sequence. The representative closed-loop cautery samples shown in Fig. 22, as opposed to open-loop cautery samples also shown in Fig. 22. d. Fig. 22 also shows the comparison of cutting-margin accuracy between closed-loop and open-loop robotic results. The cutting margin is plotted in the counter-clockwise angle unit with the solid line representing the average cutting margin, and a shaded color representing the union of the cutting margin on each angle. In six closed-loop experiments, the average margin error (i.e. , error to the ground truth 5mm) was 0.54±0.35mm; whereas in three open-loop experiments, theaverage margin error was 1.2 ± 0.88mm. Two-sample t- tests indicated the definitive significance of cautery margin improvements for the closed-loop strategy (p < 0.001 ).

[0189] IV. Discussion

[0190] The robotic system with the proposed autonomous workflow demonstrates a better accurate resection margin in cauterizing soft tissue. Markerless tracking and closed-loop motion control contribute significantly to improved accuracy of cauterization in robotic results. It is noticeable that between the range of 180° and 270° in Fig. 22 in which the open-loop control failed to maintain the cautery margin. This is because once an incision was made on the tissue surface from 0° to 180°, the remaining half segment of the tissue portion became easier to deform due to less tissue bounding to restrict tissue surface motion, and therefore, the cautery margin changed drastically when no feedback control approaches for the cautery task. One solution to reduce tissue deformation when using an open-loop control strategy is to either increase the power level setting (i.e., increase current to vaporize tissue) or control the robot with a constant low feedrate; however, either of the methods will apply larger energy concentration to the surrounding tissue along the cautery trajectory and generate thermal damage, which is considered undesired. In this study, we proposed a solution to achieve the cautery task with efficiency by modulating robot motion using tissue deformation, achieving better contour without compromising the low velocity. In the future, incorporating thermal imaging feedback will further complete the control scheme for the robotic soft-tissue electro-surgery.

[0191] While these results show promising of the proposed tech- niques in robotic soft-tissue cauterization, limitations exist in the current system and the accuracy of the robotic resection could be further improved. For instance, thetracking rate of the developed 3D tissue tracking achieved an average of 2.9 frames per second and could be further increased via a more powerful graphics processing unit (GPU) and a camera with a higher frame rate. Moreover, the parameters for the velocity controller were designed particularly for the robotic experiment setup and configuration, which is considered less generic for other soft-tissue cautery applications. In addition, the algorithm of tool occlusion developed within the tissue tracker requires knowledge of robot kinematics with accurate hand-eye calibration. If either robot calibration or kinematics is inaccurate, the tool-occlusion algorithm could fail and the tracking performance would be compromised. The alternative solution is to apply an image-based tool- tracking technique

[0026] directly using the image stream without needing robot kinematics, which is considered a more generic implementation for complex tool geometry.

[0192] V. Conclusion

[0193] We built an autonomous system to achieve closed-loop robotic cauterization via soft tissue tracking and motion planning techniques. We proposed an autonomous workflow of closed-loop robotic cauterization and developed peripheral components. A 3D soft tissue tracker was developed by integrating CoTracker to achieve online markerless tracking capability on soft tissue assisted with an algorithm for tool occlusion utilizing robot kinematics. Moreover, a velocity controller was developed via a fuzzy logic approach to modulate the feed rate in robotic cauterization using tracking feedback from the tissue tracker. The proposed robotic workflow and system were evaluated via an actual robotic cautery task on cadaver pork tongue tissue, demonstrating the feasibility of the techniques. The proposed autonomous robotic techniques achieve results with better accuracy in resection margin compared to the approaches of open-loop

[0194] Example 4: Enhancing Surgical Precision in Autonomous Robotic Incisions via Physics-Based Tissue Cutting Simulation

[0195] 1 Introduction

[0196] Robot-assisted surgery (RAS) represents the current clinical state-of- the-art in managing pathologies that require a cutting procedure. Several robotic systems for RAS exist which incorporate haptic-feedback, surgical perception, and minimally invasive technology. Producing accurate cuts is critical to improving patient care, and automating RAS procedures, such as cutting, has the potential to reduce surgeon fatigue and increase accuracy and efficiency.

[0197] Virtual environments simulating RAS and endoscopic surgery are often used to enhance surgeon training. This allows novice surgeons to practice common techniques in a safe and risk-free environment before entering the operating room. Several virtual simulation environments for surgeon train- ing are available, including Actaeon (BBZ Sri, Verona, Italy), dV Trainer (Mimic Technologies, Seattle, WA), RobotiX Mentor (3D Systems USA Corps, Rock Hill, South Carolina), ROSS (Simulated Surgical Systems, San Jose, CA), and Sepp (SimSurgery, Oslo, Norway). Additionally, the da Vinci Skills Simulator (Intuitive Surgical, Sunnyvale, CA) appends a virtual simulation console to the robotic system for real-time control and haptic feedback. Virtual reality in the form of VR headsets have also been deployed to produce realistic and engaging virtual training environments.

[0198] Realistic simulation of soft tissue mechanics is a complex engineering task that requires an accurate material deformation model for successful integration in medical simulation systems. The physical properties of in vivo tissue must first be classified and analyzed, and tissue may differ from patient to patient. Appropriate numerical schemes for calculating internal stress and pressure must be chosen toproduce stable, accurate, and converging results. Soft tissue objects may contain limitless degrees of freedom which can be expensive to compute, and a decision must be made between system accuracy and run-time. Additional complexity is added when modeling deformation that results from a surgical cut.

[0199] Soft tissue surgery simulation has often been limited to deformation analysis, and is infrequently extended to a connection with the surgical robot. Therefore, there is a potential to integrate virtual simulation of surgical procedures into the robot control loop to plan and produce more precise motions during surgical tasks. By incorporating virtual simulation into the robot con- trol loop, more complicated operations can be automated with a greater degree of accuracy, reducing surgeon fatigue and expense. Virtual surgery simulation can be broken down into four aspects, computational approach, interaction devices, system architectures, and clinical validations. Virtual surgery computed in the pre-operative planning phase could predict material deformation and interact with the robot through integration in the control structure.

[0200] Previous research has been conducted in integrating soft tissue deformation prediction and analysis into the robot control loop. Others have developed SuPerPM, a surgical perception framework that continuously collects geometric information and tracks surgical tool paths in the endoscopic environment. The framework is demonstrated as a real-time solution to tracking tissue deformation and surgical tool orientation in unstructured surgery environments, and is shown to successfully complete autonomous soft tissue manipulation tasks. Thach et al. further incorporate a virtual environment simulating material deformation into the control of robotic manipulators. The neural network architecture of this work, dubbed DeformerNet, is demonstrated on a formulated shape servoing task in which a pair ofrobotic manipulators deforms an object from an initial shape to a goal shape. The model was trained on simulated object deformation data, and has relevant surgical applications, however, cutting soft tissue was not considered in this work. There are currently few options for simulating object deformation during a cut, Heiden et al. overcome this limitation by introduced DiSECt, a differentiable simulator for cutting soft materials. The simulator is calibrated to match force and deformation fields across a variety of experiments, how- ever, this work did not aim to achieve a simulation within the robot control loop nor is it clinically validated to model soft tissue cutting.

[0201] This study sought an open source framework for the virtual simulation of soft tissue surface topology change during cutting. We analyzed several avail- able frameworks using a comparison metric, including NVIDIA Omniverse Isaac Sim, FEBio, AMBF, etc. and chose simulation open frame- work architecture (SOFA). SOFA is a multi-physics engine that primarily focuses on real-time medical simulation and is often used in the development and prototyping of biomedical models and algorithms. The framework allows a developer to define collision behavior, material properties, and boundary conditions, and deformation mechanics are solved with finite element analysis (FEA). SOFA is often used in the testing of control techniques in soft robots, and medical robot applications. For example, others incorporate OpenAI Gym reinforcement learning (RL) training architecture into SOFA in the form of LapGym, a virtual environment for developed RL models to control laparoscopic robotic surgery. The training environment is demonstrated using a variety of example virtual scenes that use various surgical tasks, such as suturing and cutting.

[0202] In this example, we aim to integrate physics-based simulation of a soft tissue incision on complex contours into the control loop of a surgical robot. We focus on interactions between the cutting tool and soft tissue to show that the pre-operativetool path can be improved in simulation and pro- duce more accurate results in the real-world. First, we performed several soft tissue incisions on complex contours using cadaver porcine tongue samples to demonstrate the relationship between tool-tissue interactions and incision error. Second, we developed a FEA-based simulation replicating the incision scenarios. Upon simulation completion, incision results are analyzed and the procedure is re-attempted with a corrected tool-path. Third, computed tomography (CT) scanned porcine tongue samples were supplied to the simulation as mesh models to predict material deformation during the incision task and output a corrected tool-path that minimizes incision error. These corrected tool-paths were exported back to the robotic system for use in soft tissue incisions experiments on the porcine tongue samples for clinical validation. The contributions of this paper are as follows 1 ) an evaluation of robotic soft tissue surface incision accuracy as guided by a pre-planned path resulting from material deformation, 2) development of a simulated surgical scene capable of predicting material deformation caused by tool-tissue cutting interactions, 3) demonstration through clinical validation that the virtual simulation method can compute more accurate surface incision paths.

[0203] 2 Methods

[0204] 2. 1 Characterization of Electrosurgical Incision Task

[0205] Surgical cutting, involving tasks like incision, dissection, and excision, commonly utilizes instruments such as scalpels, scissors, harmonic scalpels, and electrosurgery devices. In particular, our study addresses incisions executed via electrosurgical techniques, pivotal in numerous surgical procedures. Surgical incision involves a precise cut breaching the epidermis for diagnostic or therapeutic intervention. Electrosurgery, employing high-frequency electrical currents to cut or coagulate tissue, is a standard technique in pathology management, where precision iskey to successful oncologic outcomes. Contrary to most research in robotic cutting control of deformable objects focusing on straight or near-straight line cuts using general-purpose knives, our exam p le explores more complex and clinically relevant scenarios. We investigate incisions along irregular, non-planar contours using a monopolar electrosurgical needle electrode, commonly employed in surgeries like nephrectomy, hepatectomy, and glossectomy.

[0206] Our primary focus is on midline partial glossectomy, essential for early- stage tumor resection on the tongue’s superior midline surface, classified as type 1 glossectomy or mucosectomy. Our approach involves defining tumor surface boundaries based on clinical case studies, adjusting them to a 2cm diameter to match T1 / T2 stage cancer dimensions. We also adhere to a 5mm resection margin both laterally and in depth, following standard clinical practices. Surgical setup often includes oral gags and traction sutures for anterior tongue access, simulating an open surgical environment. For our experiments, we opt for porcine tongue specimens, chosen for their anatomical resemblance to human tongues and ease of accessibility. We 3D print a sample holder mimicking the jawbone and mouth floor, with a clamp to stabilize the posterior tongue. Each tongue sample’s tip was sutured and extended by 2cmusing a linear stage, replicating typical clinical tongue positioning. The experiment’s goal, illustrated in Fig. 24, is to achieve accurate robotic electrosurgical incisions along the predefined path to reduce residual tumor tissue and the risk of cancer recurrence, ultimately improving oncologic outcomes.

[0207] 2.2 Surface Incision Accuracy Following Pre-Planned Paths in TongueSamples

[0208] Building on previous research that predominantly assessed depth precision in straight or near-straight robotic incisions, our example shifts focus tosurface accuracy along irregular contours. We specifically evaluate the fidelity of robotic incisions to pre-planned paths on tongue samples, with the findings set to inform subsequent simulation models.

[0209] 2.2. 1 Clinically Relevant Tissue Incision Speed

[0210] Incision speed is a critical factor that influences both tissue deformation and the duration of surgery. We enlisted an experienced otolaryngologist to determine a clinically applicable speed for electrosurgical incisions. Utilizing two porcine tongues from a local grocery (H Mart, Lyndhurst, NJ), each tongue was marked with three lOcmlines for incision. The surgeon employed an electrosurgical pen (Bovie, Clearwater, FL) to incise at a consistent depth of 5mm, as per our study’s parameters. The duration of each cut was timed by reviewing the recorded video. The cutting speeds recorded were 11.11 , 6.66, 7.14, 7.14, 6.25, 5.00mm / s respectively, resulting in an average speed of 7.22±1.89mmDs. Based on these findings, a rounded cutting speed of 7mm / s was chosen for subsequent experiments.

[0211] 2.2.2 Robotic Experimental Setup

[0212] Leveraging our prior development of an autonomous system for tumor resection (ASTR), this example’s testbed comprised five main components: 1 ) a 6- DOF UR10e manipulator (Universal Robots, Odense, Denmark), 2) a customized laparoscopic electrosurgical instrument, 3) an RGBD camera (D405, Intel, Santa Clara, CA), 4) custom robot operating system (ROS) programs for tissue tracking and incision planning, and 5) a tongue sample holder. A standard monopolar needle electrode with a 25mmlength and I mmdiameter (Bovie, Clearwater, FL), a grounding pad, and an electrosurgical power generator (ASG-300ESU, DRE Veterinary, Louisville, KY) are key components. To manage electrosurgical smoke, we utilized both a portable smoke evacuator (Smoke Shark, Bovie, Clearwater, FL), and an air purifier(GC Multigas, IQAir, Goldach, Switzerland). The sample holder has been described in Section 2.1 , and the software’s role in the experimental process will be presented in the following sections.

[0213] 2.2.3 Electrosurgical Cutting Voltage Setting

[0214] Electrosurgical cutting voltage crucially affects the electrode’s sharpness, akin to the blade of a scalpel. Adequate voltage ensures the electrode cuts smoothly, reducing tissue deformation and bulging, which can introduce inaccuracies. While higher voltage increases sharpness, it also risks thermal damage and tissue denaturation. To identify the lowest effective voltage, we utilized four grocery porcine tongues (H Mart, Lyndhurst, NJ), each subjected to incisions at varying voltages of 25, 30, 35, and 40v, corresponding to our experimental setup as described in Section 2.1 and illustrated in Fig. 24. A custom C++ ROS program enabled the Intel D405 camera to track the incision path through color thresholding, plan electrosurgical incision paths with 5mmdepth, and guide the robot for execution at 7mmDsautonomously. Visual observations revealed that at voltages above 35v, there was no noticeable tissue bulging, while at 40v, discoloration suggested tissue denaturation and thermal damage. Consequently, 35v was chosen for subsequent experiments.

[0215] 2.2.4 Experimental Design for Incision Accuracy Evaluation

[0216] To assess incision accuracy, two ex vivo porcine tongues (AnimalTechnology, Tyler, TX) were prepared as delineated in Section 2.1. The same C++ ROS program was utilized for tracking the marked path, generating the incision plan, and guiding the robotic execution. Key parameters, informed from clinical reports and preliminary findings (Sections 2.2.1 and 2.2.3), included a near 2cm diameter tumor boundary, a 5mm safety margin, a 5mm incision depth, 7mm / s cutting speed, and a35v electrosurgical power. The primary measure was the deviation between the robot-executed incision and the pre-drawn guide on the tongue specimens, aiming to refine the fidelity of simulation models to actual soft tissue deformability. The detailed evaluation metric and results will be reported in Section 3.

[0217] 2.3 Development and Optimization of Tissue Incision Simulation

[0218] Our study sought to develop a FEA-based virtual simulation of a soft tissue incision on a complex contour. We focus on tool-tissue interactions during a soft tissue cut to produce accurate results with parameters optimized to match real world results.

[0219] 2.3.1 Design Specifications

[0220] We considered several important design considerations in the development of the soft tissue incision simulation. These considerations included the ability to import tissue models from medical scans, capture nonlinear tissue deformation, establish mechanical constraints and boundary conditions to recover clinical scene, define tool-tissue interactions in terms of applied force, and separate volumetric meshes during cutting interactions. FEA was emphasized to handle tissue deformation.

[0221] 2.3.2 Simulation Software Comparison and Selection

[0222] The software frameworks considered were evaluated based on incorporation of FEA, ability to handle a cut, and existence of an open source repository to aid development. Open simulation framework architecture (SOFA) was chosen as the simulation framework for this project because of its ability to model soft tissue deformation using FEM, native support for different nonlinear material types, and SofaCarving plugin which allows for a cut to be simulated. SOFA is a mesh-based soft tissue simulation framework, meaning that each model is defined as a volumetric mesh composed of smaller tetrahedral elements. SofaCarving is a plugin extension of SOFA that defines object collision in terms of carving interactions by removing tetrahedralmesh elements from a carving surface object when in collision with a carving tool object. SofaPython3 is another plugin extension of SOFA that includes python bindings that allow for interactive scripts to be developed, allowing for the recovery of incision geometry produced through the simulated incision. Moreover, SofaCaribou is a plugin extension of SOFA that provides extra capability to the framework, such as nonlinear nonlinear integration solvers to improve the performance of material deformation. The combined integration of each of these plugins and SOFA allowed for this project to develop a soft tissue cutting simulation of curved surface incisions.

[0223] 2.3.3 Soft Tissue Mesh Model Design

[0224] Volumetric mesh models consisting of tetrahedral elements are a common input into medical simulations. The simulation of volumetric mesh object cuts must consider a method for producing an accurate and precise incision geometry. For example, tetrahedral mesh subdivision is a technique used to re-mesh the volumetric object during the cutting procedure to produce a more accurate and straight-edged object incision. However, large mesh subdivision serves only for a straight cut on each tetrahedron, while in our assumption the tool can cut through the mesh following a curved spline path. SofaCarving achieves volumetric cuts by removing individual tetrahedral mesh elements that come into contact with a carving tool object. Therefore, a volumetric mesh with a high density of tetrahedral elements is preferred to create a smooth and accurate incision geometry. However, a denser mesh has an increased number of elements for which material deformation must be calculated, thus increasing computational load. To overcome this challenge, the soft tissue volumetric meshes used in this simulation were given dense regions of tetrahedral elements in the pre-defined cutting area, as well as a reduced amount of tetrahedral elements across the remainder of the object to reduce computational load.

[0225] 2.3.4 Simulation Overview

[0226] The simulation scene is composed of a volumetric mesh model of a cadaver porcine tongue soft tissue sample, an electrosurgical instrument model as inspired by our previous work, and a platform that supports the tongue model under gravity. Inspired from a prior experimental setup of autonomous tumor resection, this simulation aims to model soft tissue surface incision on complex contours used to reveal the tumor’s inferior edge. Prior to the incision procedure, mechanical constraints are placed on the posterior and anterior of the tongue model, and the tongue model tip is stretched 2cm to replicate the linear stage extension and to recover clinical positioning. Psuedotumor boundary extents are imported to the simulation from clinical records of head-and-neck cancer. Following standard clinical practice in tumor resection, as described previously, the target simulation incision path includes a 5mmlateral offset and 5mmdepth offset to insure appropriate resection margins. The incision contour is downsampled to 45 tool waypoints, and the electrocautery object traverses the path sequentially until procedure completion. Upon procedure completion, the incision geometry is recovered, error between the target incision path and the recovered incision path is calculated, and the simulation is reset with an updated incision path.

[0227] The soft tissue material of the tongue model is calculated as a hyperelastic neo-Hookean solid to replicate hydrostat-like specimens. As the electrocautery model impacts the tongue model mesh surface, tetrahedral elements are removed from the mesh to produce the incision, and two steps are taken to ensure the collision interaction is continuous such as to produce applied force and deformation of the surface topology. The first step is that a minimum distance necessary for a tetrahedral mesh element to be carved is smaller than the minimum distance set for contactbetween models. This produces a penetration-like effect during the cutting interaction. The second step is that the open source SofaCarving plugin is modified so then tongue mesh elements must be in contact with the electrocautery tool for at least a pre-set time- frame before the element can be carved. This acts to transform the cutting interaction into a more continuous collision and produces tissue bulges and drag similar to those observed experimentally. This innovation is critical in producing a simulated cut with error that varies with increases toolspeed.

[0228] This simulation seeks to improve soft tissue incision accuracy by using a previously simulated incision attempt to improve the electrocautery tool-path over a new iteration. A modification to the open source SofaCarving plugin allowed for the simulation to record which elements were removed from the tongue mesh model during the incision procedure. These removed vertices undergo data pre-processed using a Gaussian blur technique and are then skeletonized to produce a continuous cutting contour for incision accuracy analysis. Waypoints from the target incision path are compared to the recovered cutting contour as polar coordinates. Each incision path waypoint is then updated by the radial difference between the intended incision and recovered incision. The simulation is then reset with the new tool path, and iterations are run until error convergence. In the surgical robot control loop, the goal is to use the simulation to predict the tissue deformation resulting from the cut, update the trajectory until a satisfactory path is discovered, and then output the path to the robotic system.

[0229] 2.4 Surface Incision Accuracy Following Simulation-OptimizedPaths in Tongue Samples

[0230] To validate our soft tissue cutting simulation, which recommends an optimized incision path minimizing the gap between actual and pre-planned paths, realtissue experiments are essential. These experiments will determine if our simulationbased predictions effectively enhance surface incision precision on porcine tongue tissues.

[0231] 2.4.1 Sim-to-Real Tissue Registration

[0232] The simulation-optimized incision path, created in the simulation coordinate frame, required alignment with the real-world coordinate frame to guide our electrosurgical robot, necessitating a sim-to-real registration. A porcine tongue sample was CT scanned and converted to volumetric mesh for simulation. Upon simulation start, a point cloud of the tongue mesh is output with an attached point set representing the tumor edge and the optimized incision path. This point cloud was processed to preserve the top surface, using a custom Python program with Open3D libraries. After the simulation, the same CT scanned porcine tongue sample was prepared in the 3D printed holder, as described in Section 2.2. Its superior surface point cloud was cropped and acquired from Intel D405 streamed point cloud using a custom ROS GUI. Wethen applied the coherent point drift (CPD) technique with scaling for registration, using a Python algorithm implementation. A transformation matrix was produced, linking simulation to real-world frames, and allowed us to position the 2D tumor edge and optimized incision path from the SOFA simulation into the real-world coordinates.

[0233] 2.4.2 Robotic Surface Incision Experimental Design

[0234] To assess the effectiveness of robotic incision following simulation- optimized versus prior pre-planned incision paths, all key parameters as described in Section 2.2.4 were kept consistent. Since the paths were derived from simulation, the method of manually drawing tumor edges on porcine tongues was impractical. Instead, a C++ ROS program projected the 2D tumor edge and optimized incision path ontothe tongue sample’s surface. The program also implemented a 1 mm inferior offset to the tumor edge, guiding the electro- surgical robot to mark the edge at 2mmDswith minimal tissue deformation. The 5mm incision depth was then applied to the projected incision path for autonomous execution. Post-execution, the distance between the actual incision and the electrocautery-marked tumor edge was measured, with the surface incision error calculated as the difference between this distance and the predefined incision margin of 5mm.

[0235] 3 Experiments and Results

[0236] 3. 1 Evaluation Metrics

[0237] After completing the surface incisions, top view images were captured for all studies. These images, scaled using a ruler for pixel-to-millimeter conversion, yielded a resolution of about 0D02mm. A custom Python script, employing OpenCV libraries, processed these images to compute surface incision errors. The error for each point on the actual incision path (aip) was calculated as the shortest distance to the drawn tumor edge (dtp), minus the intended surface margin of 4D5mm(considering a 5mm incision margin and a 05mmelectrosurgical electrode radius). Mathematically, this is expressedpOjnt jjonthe incision contour.

[0238] Some further aspects are defined in the following clauses:

[0239] Clause 1 : A surgical system, comprising: at least two robotic manipulators, wherein at least a first robotic manipulator comprises at least one tissue grasping implement and at least a second robotic manipulator comprises at least one surgical cutting implement; and, at least one controller operably connected at least to the first and second robotic manipulators, wherein the controller comprises at least one processor and at least one memory communicatively coupled to the processor,the memory storing non-transitory instructions which, when executed by the processor, perform operations comprising: contacting at least a portion of the tissue grasping implement of the first robotic manipulator with at least one tissue to produce contacted tissue; grasping the contacted tissue using the tissue grasping implement of the first robotic manipulator to produce grasped tissue; and, creating at least one incision in the grasped tissue using the electrosurgical cutting implement of the second robotic manipulator to produce incised tissue.

[0240] Clause 2: The surgical system of Clause 1 , wherein the tissue grasping implement comprises a robotic gripping mechanism.

[0241] Clause 3: The surgical system of Clause 1 or Clause 2, wherein the surgical cutting implement comprises an electrosurgical cutting implement.

[0242] Clause 4: The surgical system of any one of the preceding Clauses 1-3, wherein the tissue grasping implement comprises a vacuum grasping mechanism.

[0243] Clause 5: The surgical system of any one of the preceding Clauses 1-4, wherein the non-transitory instructions which, when executed by the processor, perform operations comprising: applying vacuum to the contacted tissue using the vacuum grasping mechanism of the first robotic manipulator to produce the grasped tissue.

[0244] Clause 6: The surgical system of any one of the preceding Clauses 1-5, wherein the system operates substantially autonomously.

[0245] Clause 7: The surgical system of any one of the preceding Clauses 1-6, wherein the electrosurgical cutting implement comprises at least one cautery device.

[0246] Clause 8: The surgical system of any one of the preceding Clauses 1-7, wherein the tissue comprises at least a portion of a tumor.

[0247] Clause 9: The surgical system of any one of the preceding Clauses 1- 8, wherein the robotic manipulators are each configured to move with six degrees-of- freedom.

[0248] Clause 10: The surgical system of any one of the preceding Clauses 1 -9, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: removing at least a portion of the incised tissue using the first robotic manipulator.

[0249] Clause 11 : The surgical system of any one of the preceding Clauses 1 -10, wherein the system is configured to operate at least partially under human supervisory control.

[0250] Clause 12: The surgical system of any one of the preceding Clauses 1 -11 , wherein the human supervisory control comprises one or more tasks selected from the group consisting of: a real-time monitoring task, a surgical plan request task, a surgical plan approval task, and a selective intervention task.

[0251] Clause 13: The surgical system of any one of the preceding Clauses 1 -12, wherein a subject comprises the tissue.

[0252] Clause 14: The surgical system of any one of the preceding Clauses 1 -13, wherein an oral cavity of the subject comprises the tissue.

[0253] Clause 15: The surgical system of any one of the preceding Clauses 1 -14, wherein the tissue comprises a soft tissue.

[0254] Clause 16: The surgical system of any one of the preceding Clauses 1 -15, further comprising at least one camera operably connected to the controller, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: imaging at least a portion of the tissue using thecamera to produce an imaged tissue data set; and, guiding movement and / or other operations of the robotic manipulators using the imaged tissue data set.

[0255] Clause 17 : The surgical system of any one of the preceding Clauses 1 -16, wherein the imaged tissue data set comprises at least one video image data set.

[0256] Clause 18: The surgical system of any one of the preceding Clauses 1 -17, wherein the camera comprises a light source configured to illuminate one or more areas disposed at least proximal to the tissue.

[0257] Clause 19: The surgical system of any one of the preceding Clauses 1 -18, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: imaging at least the portion of the tissue and / or guiding movement and / or other operations of the robotic manipulators using at least one markerless tissue tracking technique and / or at least one closed-loop motion control technique.

[0258] Clause 20: The surgical system of any one of the preceding Clauses 1 -19, wherein the markerless tissue tracking technique comprises at least pixel-level tracking resolution.

[0259] Clause 21 : The surgical system of any one of the preceding Clauses 1 -20, comprising at least one tissue tracker, at least one path planner, at least one velocity controller, and at least one robot motion planner, wherein the tissue tracker receives at least a portion of the imaged tissue data set from the camera and computes tracking feedback value Dk using the portion of the imaged tissue data set, wherein the path planner and the velocity controller receive the tracking feedback value DR from the tissue tracker, wherein the path planner updates a control input value p / <+i using the tracking feedback value Dk, wherein the velocity controller updates a control input value Vk+i using the tracking feedback value Dk, wherein the robot motion plannerreceives the control input value p / <+i from the path planner, wherein the robot motion planner receives the control input value Vk+i from the velocity controller, and wherein the robot motion planner updates a motion plan of one or more of the robotic manipulators using the control input value p / <+i and / or the control input value V <+1.

[0260] Clause 22: The surgical system of any one of the preceding Clauses 1 -21 , wherein the tissue tracker comprises a three-dimensional (3D) tissue tracker.

[0261] Clause 23: The surgical system of any one of the preceding Clauses 1 -22, wherein the camera comprises a depth sensing camera and / or a near infrared (NIR) camera.

[0262] Clause 24: The surgical system of any one of the preceding Clauses 1 -23, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: implementing one or more tissue incision planning procedures that use the imaged tissue data set, which tissue incision planning procedures are selected from the group consisting of: an NIR fluorescent landmark guidance procedure, a dynamic visual adjustment procedure, and a finite- element analysis optimization procedure.

[0263] Clause 25: The surgical system of any one of the preceding Clauses 1 -24, wherein the depth sensing camera comprises an RGB-D camera.

[0264] Clause 26: The surgical system of any one of the preceding Clauses 1 -25, further comprising at least one smoke evacuator operably connected to the controller, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: evacuating smoke from one or more areas at least proximal to the tissue using the smoke evacuator.

[0265] Clause 27 : The surgical system of any one of the preceding Clauses 1 -26, wherein one or more of the robotic manipulators comprise the smoke evacuator.

[0266] Clause 28: A method of producing an incised tissue, the method comprising: contacting at least a portion of at least one tissue grasping implement of at least a first robotic manipulator with at least one tissue to produce contacted tissue; grasping the contacted tissue using the tissue grasping implement of the first robotic manipulator to produce grasped tissue; and, creating at least one incision in the grasped tissue using at least one surgical cutting implement of at least a second robotic manipulator to produce incised tissue, thereby producing the incised tissue.

[0267] Clause 29: The method of Clause 28, wherein the tissue grasping implement comprises a robotic gripping mechanism.

[0268] Clause 30: The method of Clause 28 or Clause 29, wherein the surgical cutting implement comprises an electrosurgical cutting implement.

[0269] Clause 31 : The method of any one of the preceding Clauses 28-30, wherein the tissue grasping implement comprises a vacuum grasping mechanism.

[0270] Clause 32: The method of any one of the preceding Clauses 28-31 , comprising applying vacuum to the contacted tissue using the vacuum grasping mechanism of the first robotic manipulator to produce the grasped tissue.

[0271] Clause 33: The method of any one of the preceding Clauses 28-32, comprising performing the method using a system that operates substantially autonomously.

[0272] Clause 34: The method of any one of the preceding Clauses 28-33, wherein the electrosurgical cutting implement comprises at least one cautery device.

[0273] Clause 35: The method of any one of the preceding Clauses 28-34, wherein the tissue comprises at least a portion of a tumor.

[0274] Clause 36: The method of any one of the preceding Clauses 28-35, wherein the first and second robotic manipulators are each configured to move with six degrees-of-freedom.

[0275] Clause 37: The method of any one of the preceding Clauses 28-36, further comprising removing at least a portion of the incised tissue using the first robotic manipulator.

[0276] Clause 38: The method of any one of the preceding Clauses 28-37, comprising performing the method using a system that is configured to operate at least partially under human supervisory control.

[0277] Clause 39: The method of any one of the preceding Clauses 28-38, wherein the human supervisory control comprises one or more tasks selected from the group consisting of: a real-time monitoring task, a surgical plan request task, a surgical plan approval task, and a selective intervention task.

[0278] Clause 40: The method of any one of the preceding Clauses 28-39, wherein a subject comprises the tissue.

[0279] Clause 41 : The method of any one of the preceding Clauses 28-40, wherein an oral cavity of the subject comprises the tissue.

[0280] Clause 42: The method of any one of the preceding Clauses 28-41 , wherein the tissue comprises a soft tissue.

[0281] Clause 43: The method of any one of the preceding Clauses 28-42, further comprising: imaging at least a portion of the tissue using at least one camera to produce an imaged tissue data set; and, guiding movement and / or other operations of the robotic manipulators using the imaged tissue data set.

[0282] Clause 44: The method of any one of the preceding Clauses 28-43, wherein the imaged tissue data set comprises at least one video image data set.

[0283] Clause 45: The method of any one of the preceding Clauses 28-44, wherein the camera comprises a light source configured to illuminate one or more areas disposed at least proximal to the tissue.

[0284] Clause 46: The method of any one of the preceding Clauses 28-45, comprising imaging at least the portion of the tissue and / or guiding movement and / or other operations of the first and second robotic manipulators using at least one markerless tissue tracking technique and / or at least one closed-loop motion control technique.

[0285] Clause 47: The method of any one of the preceding Clauses 28-46, wherein the markerless tissue tracking technique comprises at least pixel-level tracking resolution.

[0286] Clause 48: The method of any one of the preceding Clauses 28-47, comprising using at least one tissue tracker, at least one path planner, at least one velocity controller, and at least one robot motion planner, wherein the tissue tracker receives at least a portion of the imaged tissue data set from the camera and computes tracking feedback value Dk using the portion of the imaged tissue data set, wherein the path planner and the velocity controller receive the tracking feedback value DR from the tissue tracker, wherein the path planner updates a control input value p / +i using the tracking feedback value Dk, wherein the velocity controller updates a control input value Vk+i using the tracking feedback value Dk, wherein the robot motion planner receives the control input value p / +i from the path planner, wherein the robot motion planner receives the control input value Vk+i from the velocity controller, and wherein the robot motion planner updates a motion plan of one or more of the robotic manipulators using the control input value p^+i and / or the control input value V <+1.

[0287] Clause 49: The method of any one of the preceding Clauses 28-48, wherein the tissue tracker comprises a three-dimensional (3D) tissue tracker.

[0288] Clause 50: The method of any one of the preceding Clauses 28-49, wherein the camera comprises a depth sensing camera and / or a near infrared (NIR) camera.

[0289] Clause 51 : The method of any one of the preceding Clauses 28-50, further comprising implementing one or more tissue incision planning procedures that use the imaged tissue data set, which tissue incision planning procedures are selected from the group consisting of: an NIR fluorescent landmark guidance procedure, a dynamic visual adjustment procedure, and a finite-element analysis optimization procedure.

[0290] Clause 52: The method of any one of the preceding Clauses 28-51 , wherein the depth sensing camera comprises an RGB-D camera.

[0291] Clause 53: The method of any one of the preceding Clauses 28-52, further comprising evacuating smoke from one or more areas at least proximal to the tissue using at least one smoke evacuator.

[0292] Clause 54: The method of any one of the preceding Clauses 28-53, wherein the first and / or second robotic manipulator comprises the smoke evacuator.

[0293] Clause 55: A computer readable media, comprising non-transitory computer executable instructions which, when executed by a processor, perform operations comprising: contacting at least a portion of at least one tissue grasping implement of at least a first robotic manipulator with at least one tissue to produce contacted tissue; grasping the contacted tissue using the tissue grasping implement of the first robotic manipulator to produce grasped tissue; and, creating at least one incision in the grasped tissue using at least one surgical cutting implement of at leasta second robotic manipulator to produce incised tissue, thereby producing the incised tissue.

[0294] Clause 56: The computer readable media of Clause 55, wherein the tissue grasping implement comprises a robotic gripping mechanism.

[0295] Clause 57: The computer readable media of Clause 55 or Clause 56, wherein the surgical cutting implement comprises an electrosurgical cutting implement.

[0296] Clause 58: The computer readable media of any one of the preceding Clauses 55-57, wherein the tissue grasping implement comprises a vacuum grasping mechanism.

[0297] Clause 59: The computer readable media of any one of the preceding Clauses 55-58, wherein the non-transitory instructions which, when executed by the processor, perform operations comprising: applying vacuum to the contacted tissue using the vacuum grasping mechanism of the first robotic manipulator to produce the grasped tissue.

[0298] Clause 60: The computer readable media of any one of the preceding Clauses 55-59, wherein the electrosurgical cutting implement comprises at least one cautery device.

[0299] Clause 61 : The computer readable media of any one of the preceding Clauses 55-60, wherein the first and second robotic manipulators are each configured to move with six degrees-of-freedom.

[0300] Clause 62: The computer readable media of any one of the preceding Clauses 55-61 , wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: removing at least a portion of the incised tissue using the first robotic manipulator.

[0301] Clause 63: The computer readable media of any one of the preceding Clauses 55-62, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: imaging at least a portion of the tissue using at least one camera to produce an imaged tissue data set; and, guiding movement and / or other operations of the robotic manipulators using the imaged tissue data set.

[0302] Clause 64: The computer readable media of any one of the preceding Clauses 55-63, wherein the imaged tissue data set comprises at least one video image data set.

[0303] Clause 65: The computer readable media of any one of the preceding Clauses 55-64, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: imaging at least the portion of the tissue and / or guiding movement and / or other operations of the robotic manipulators using at least one markerless tissue tracking technique and / or at least one closed-loop motion control technique.

[0304] Clause 66: The computer readable media of any one of the preceding Clauses 55-65, wherein the markerless tissue tracking technique comprises at least pixel-level tracking resolution.

[0305] Clause 67: The computer readable media of any one of the preceding Clauses 55-66, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: using at least one tissue tracker, at least one path planner, at least one velocity controller, and at least one robot motion planner, wherein the tissue tracker receives at least a portion of the imaged tissue data set from the camera and computes tracking feedback value Dk using the portion of the imaged tissue data set, wherein the path planner and the velocity controllerreceive the tracking feedback value Dk from the tissue tracker, wherein the path planner updates a control input value p / <+i using the tracking feedback value Dk, wherein the velocity controller updates a control input value Vk+i using the tracking feedback value Dk, wherein the robot motion planner receives the control input value Pk+i from the path planner, wherein the robot motion planner receives the control input value Vk+i from the velocity controller, and wherein the robot motion planner updates a motion plan of one or more of the robotic manipulators using the control input value Pk+i and / or the control input value V <+1.

[0306] Clause 68: The computer readable media of any one of the preceding Clauses 55-67, wherein the tissue tracker comprises a three-dimensional (3D) tissue tracker.

[0307] Clause 69: The computer readable media of any one of the preceding Clauses 55-68, wherein the camera comprises a depth sensing camera and / or a near infrared (NIR) camera.

[0308] Clause 70: The computer readable media of any one of the preceding Clauses 55-69, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: implementing one or more tissue incision planning procedures that use the imaged tissue data set, which tissue incision planning procedures are selected from the group consisting of: an NIR fluorescent landmark guidance procedure, a dynamic visual adjustment procedure, and a finite- element analysis optimization procedure.

[0309] Clause 71 : The computer readable media of any one of the preceding Clauses 55-70, wherein the depth sensing camera comprises an RGB-D camera.

[0310] Clause 72: The computer readable media of any one of the precedingClauses 55-71 , wherein the non-transitory instructions which, when executed by theprocessor, further perform operations comprising: evacuating smoke from one or more areas at least proximal to the tissue using at least one smoke evacuator.

[0311] Clause 73: The computer readable media of any one of the preceding Clauses 55-72, wherein one or more of the robotic manipulators comprise the smoke evacuator.

[0312] While the invention has been described with reference to the exemplary embodiments thereof, those skilled in the art will be able to make various modifications to the described embodiments without departing from the true spirit and scope. The terms and descriptions used herein are set forth by way of illustration only and are not meant as limitations. In particular, although the method has been described by examples, the steps of the method can be performed in a different order than illustrated or simultaneously. Those skilled in the art will recognize that these and other variations are possible within the spirit and scope as defined in the following claims and their equivalents.

Claims

What is claimed is:1 . A surgical system, comprising: at least two robotic manipulators, wherein at least a first robotic manipulator comprises at least one tissue grasping implement and at least a second robotic manipulator comprises at least one surgical cutting implement; and, at least one controller operably connected at least to the first and second robotic manipulators, wherein the controller comprises at least one processor and at least one memory communicatively coupled to the processor, the memory storing non-transitory instructions which, when executed by the processor, perform operations comprising: contacting at least a portion of the tissue grasping implement of the first robotic manipulator with at least one tissue to produce contacted tissue; grasping the contacted tissue using the tissue grasping implement of the first robotic manipulator to produce grasped tissue; and, creating at least one incision in the grasped tissue using the electrosurgical cutting implement of the second robotic manipulator to produce incised tissue.

2. The surgical system of claim 1 , wherein the tissue grasping implement comprises a robotic gripping mechanism.

3. The surgical system of claim 1 , wherein the surgical cutting implement comprises an electrosurgical cutting implement.

4. The surgical system of claim 1 , wherein the tissue grasping implement comprises a vacuum grasping mechanism.

5. The surgical system of claim 4, wherein the non-transitory instructions which, when executed by the processor, perform operations comprising: applying vacuum to the contacted tissue using the vacuum grasping mechanism of the first robotic manipulator to produce the grasped tissue.

6. The surgical system of claim 1 , wherein the system operatessubstantially autonomously.

7. The surgical system of claim 1 , wherein the electrosurgical cutting implement comprises at least one cautery device.

8. The surgical system of claim 1 , wherein the tissue comprises at least a portion of a tumor.

9. The surgical system of claim 1 , wherein the robotic manipulators are each configured to move with six degrees-of-freedom.

10. The surgical system of claim 1 , wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: removing at least a portion of the incised tissue using the first robotic manipulator.11 . The surgical system of claim 1 , wherein the system is configured to operate at least partially under human supervisory control.

12. The surgical system of claim 11 , wherein the human supervisory control comprises one or more tasks selected from the group consisting of: a realtime monitoring task, a surgical plan request task, a surgical plan approval task, and a selective intervention task.

13. The surgical system of claim 1 , wherein a subject comprises the tissue.

14. The surgical system of claim 13, wherein an oral cavity of the subject comprises the tissue.

15. The surgical system of claim 13, wherein the tissue comprises a soft tissue.

16. The surgical system of claim 1 , further comprising at least one cameraoperably connected to the controller, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: imaging at least a portion of the tissue using the camera to produce an imaged tissue data set; and, guiding movement and / or other operations of the robotic manipulators using the imaged tissue data set.

17. The surgical system of claim 16, wherein the imaged tissue data set comprises at least one video image data set.

18. The surgical system of claim 16, wherein the camera comprises a light source configured to illuminate one or more areas disposed at least proximal to the tissue.

19. The surgical system of claim 16, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: imaging at least the portion of the tissue and / or guiding movement and / or other operations of the robotic manipulators using at least one markerless tissue tracking technique and / or at least one closed-loop motion control technique.

20. The surgical system of claim 19, wherein the markerless tissue tracking technique comprises at least pixel-level tracking resolution.21 . The surgical system of claim 19, comprising at least one tissue tracker, at least one path planner, at least one velocity controller, and at least one robot motion planner, wherein the tissue tracker receives at least a portion of the imaged tissue data set from the camera and computes tracking feedback value Dk using the portion of the imaged tissue data set, wherein the path planner and the velocity controller receive the tracking feedback value DR from the tissue tracker, wherein the path planner updates a control input value p / +i using the tracking feedback value Dk, wherein the velocity controller updates a control input value Vk+i using the tracking feedback value Dk, wherein the robot motion planner receives the control input value Pk+i from the path planner, wherein the robot motion planner receives the controlinput value Vk+i from the velocity controller, and wherein the robot motion planner updates a motion plan of one or more of the robotic manipulators using the control input value p / <+i and / or the control input value V <+1.

22. The surgical system of claim 21 , wherein the tissue tracker comprises a three-dimensional (3D) tissue tracker.

23. The surgical system of claim 16, wherein the camera comprises a depth sensing camera and / or a near infrared (NIR) camera.

24. The surgical system of claim 23, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: implementing one or more tissue incision planning procedures that use the imaged tissue data set, which tissue incision planning procedures are selected from the group consisting of: an NIR fluorescent landmark guidance procedure, a dynamic visual adjustment procedure, and a finite-element analysis optimization procedure.

25. The surgical system of claim 23, wherein the depth sensing camera comprises an RGB-D camera.

26. The surgical system of claim 1 , further comprising at least one smoke evacuator operably connected to the controller, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: evacuating smoke from one or more areas at least proximal to the tissue using the smoke evacuator.

27. The surgical system of claim 26, wherein one or more of the robotic manipulators comprise the smoke evacuator.

28. A method of producing an incised tissue, the method comprising: contacting at least a portion of at least one tissue grasping implement of at least a first robotic manipulator with at least one tissue to produce contacted tissue;grasping the contacted tissue using the tissue grasping implement of the first robotic manipulator to produce grasped tissue; and, creating at least one incision in the grasped tissue using at least one surgical cutting implement of at least a second robotic manipulator to produce incised tissue, thereby producing the incised tissue.

29. The method of claim 28, wherein the tissue grasping implement comprises a robotic gripping mechanism.

30. The method of claim 28, wherein the surgical cutting implement comprises an electrosurgical cutting implement.31 . The method of claim 28, wherein the tissue grasping implement comprises a vacuum grasping mechanism.

32. The method of claim 31 , comprising applying vacuum to the contacted tissue using the vacuum grasping mechanism of the first robotic manipulator to produce the grasped tissue.

33. The method of claim 28, comprising performing the method using a system that operates substantially autonomously.

34. The method of claim 28, wherein the electrosurgical cutting implement comprises at least one cautery device.

35. The method of claim 28, wherein the tissue comprises at least a portion of a tumor.

36. The method of claim 28, wherein the first and second robotic manipulators are each configured to move with six degrees-of-freedom.

37. The method of claim 28, further comprising removing at least a portion of the incised tissue using the first robotic manipulator.

38. The method of claim 28, comprising performing the method using a system that is configured to operate at least partially under human supervisory control.

39. The method of claim 38, wherein the human supervisory control comprises one or more tasks selected from the group consisting of: a real-time monitoring task, a surgical plan request task, a surgical plan approval task, and a selective intervention task.

40. The method of claim 38, wherein a subject comprises the tissue.41 . The method of claim 40, wherein an oral cavity of the subject comprises the tissue.

42. The method of claim 40, wherein the tissue comprises a soft tissue.

43. The method of claim 38, further comprising: imaging at least a portion of the tissue using at least one camera to produce an imaged tissue data set; and, guiding movement and / or other operations of the robotic manipulators using the imaged tissue data set.

44. The method of claim 43, wherein the imaged tissue data set comprises at least one video image data set.

45. The method of claim 43, wherein the camera comprises a light source configured to illuminate one or more areas disposed at least proximal to the tissue.

46. The method of claim 43, comprising imaging at least the portion of the tissue and / or guiding movement and / or other operations of the first and second robotic manipulators using at least one markerless tissue tracking technique and / or at least one closed-loop motion control technique.

47. The method of claim 46, wherein the markerless tissue tracking technique comprises at least pixel-level tracking resolution.

48. The method of claim 46, comprising using at least one tissue tracker, at least one path planner, at least one velocity controller, and at least one robot motion planner, wherein the tissue tracker receives at least a portion of the imaged tissue data set from the camera and computes tracking feedback value Dk using the portion of the imaged tissue data set, wherein the path planner and the velocity controller receive the tracking feedback value Dk from the tissue tracker, wherein the path planner updates a control input value p / <+i using the tracking feedback value Dk, wherein the velocity controller updates a control input value Vk+i using the tracking feedback value Dk, wherein the robot motion planner receives the control input value Pk+i from the path planner, wherein the robot motion planner receives the control input value Vk+i from the velocity controller, and wherein the robot motion planner updates a motion plan of one or more of the robotic manipulators using the control input value p / <+i and / or the control input value V <+1.

49. The method of claim 48, wherein the tissue tracker comprises a three- dimensional (3D) tissue tracker.

50. The method of claim 43, wherein the camera comprises a depth sensing camera and / or a near infrared (NIR) camera.51 . The method of claim 50, further comprising implementing one or more tissue incision planning procedures that use the imaged tissue data set, which tissue incision planning procedures are selected from the group consisting of: an NIR fluorescent landmark guidance procedure, a dynamic visual adjustment procedure, and a finite-element analysis optimization procedure.

52. The method of claim 50, wherein the depth sensing camera comprises an RGB-D camera.

53. The method of claim 28, further comprising evacuating smoke from one or more areas at least proximal to the tissue using at least one smoke evacuator.

54. The method of claim 53, wherein the first and / or second robotic manipulator comprises the smoke evacuator.

55. A computer readable media, comprising non-transitory computer executable instructions which, when executed by a processor, perform operations comprising: contacting at least a portion of at least one tissue grasping implement of at least a first robotic manipulator with at least one tissue to produce contacted tissue; grasping the contacted tissue using the tissue grasping implement of the first robotic manipulator to produce grasped tissue; and, creating at least one incision in the grasped tissue using at least one surgical cutting implement of at least a second robotic manipulator to produce incised tissue, thereby producing the incised tissue.

56. The computer readable media of claim 55, wherein the tissue grasping implement comprises a robotic gripping mechanism.

57. The computer readable media of claim 55, wherein the surgical cutting implement comprises an electrosurgical cutting implement.

58. The computer readable media of claim 55, wherein the tissue grasping implement comprises a vacuum grasping mechanism.

59. The computer readable media of claim 58, wherein the non-transitory instructions which, when executed by the processor, perform operations comprising: applying vacuum to the contacted tissue using the vacuum grasping mechanism of the first robotic manipulator to produce the grasped tissue.

60. The computer readable media of claim 55, wherein the electrosurgical cutting implement comprises at least one cautery device.61 . The computer readable media of claim 55, wherein the first and second robotic manipulators are each configured to move with six degrees-of-freedom.

62. The computer readable media of claim 55, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: removing at least a portion of the incised tissue using the first robotic manipulator.

63. The computer readable media of claim 55, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: imaging at least a portion of the tissue using at least one camera to produce an imaged tissue data set; and, guiding movement and / or other operations of the robotic manipulators using the imaged tissue data set.

64. The computer readable media of claim 63, wherein the imaged tissue data set comprises at least one video image data set.

65. The computer readable media of claim 63, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: imaging at least the portion of the tissue and / or guiding movement and / or other operations of the robotic manipulators using at least one markerless tissue tracking technique and / or at least one closed-loop motion control technique.

66. The computer readable media of claim 65, wherein the markerless tissue tracking technique comprises at least pixel-level tracking resolution.

67. The computer readable media of claim 65, wherein the non-transitory instructions which, when executed by the processor, further perform operationscomprising: using at least one tissue tracker, at least one path planner, at least one velocity controller, and at least one robot motion planner, wherein the tissue tracker receives at least a portion of the imaged tissue data set from the camera and computes tracking feedback value Dk using the portion of the imaged tissue data set, wherein the path planner and the velocity controller receive the tracking feedback value Dk from the tissue tracker, wherein the path planner updates a control input value pk+i using the tracking feedback value Dk, wherein the velocity controller updates a control input value Vk+i using the tracking feedback value Dk, wherein the robot motion planner receives the control input value p / <+i from the path planner, wherein the robot motion planner receives the control input value Vk+i from the velocity controller, and wherein the robot motion planner updates a motion plan of one or more of the robotic manipulators using the control input value p / <+i and / or the control input value Vk+i .

68. The computer readable media of claim 67, wherein the tissue tracker comprises a three-dimensional (3D) tissue tracker.

69. The computer readable media of claim 63, wherein the camera comprises a depth sensing camera and / or a near infrared (NIR) camera.

70. The computer readable media of claim 69, wherein the non-transitory instructions which, when executed by the processor, further perform operations comprising: implementing one or more tissue incision planning procedures that use the imaged tissue data set, which tissue incision planning procedures are selected from the group consisting of: an NIR fluorescent landmark guidance procedure, a dynamic visual adjustment procedure, and a finite-element analysis optimization procedure.71 . The computer readable media of claim 69, wherein the depth sensing camera comprises an RGB-D camera.

72. The computer readable media of claim 55, wherein the non-transitoryinstructions which, when executed by the processor, further perform operations comprising: evacuating smoke from one or more areas at least proximal to the tissue using at least one smoke evacuator.

73. The computer readable media of claim 72, wherein one or more of the robotic manipulators comprise the smoke evacuator.

Images