Surgical performance management via triangulation adjustment

Abstract

Technical solutions for implementing surgeon-side and patient-side triangulation in robotic surgeries is provided. A processor can be configured to identify at least one of a type of task or a type of phase of a medical procedure. The processor can determine vertices associated with a first manipulator component, a second manipulator component, and an optical device that are used to perform the at least one of the type of task or the type of phase of the medical procedure. The processor can generate, via a triangulation function, a metric based on the vertices. The processor can provide a visual indication of a performance of the medical procedure based on a comparison of the metric with a threshold established for the type of task or the type of phase.

Description

SURGICAL PERFORMANCE MANAGEMENT VIA TRIANGULATIONADJUSTMENTCROSS-REFERENCES TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63 / 614,281, filed December 22, 2023, which is hereby incorporated by reference herein in its entirety.BACKGROUND

[0002] Medical procedures can be performed in an operating room. As the amount and variety of equipment in the operating room increases, or medical procedures become increasingly complex, it can be challenging to perform such medical procedures efficiently, reliably, or without incident.SUMMARY

[0003] Technical solutions disclosed herein are generally related to measuring and presenting triangulation of manipulator components and optical devices utilized in surgery. For example, when performing robotic surgeries, surgeons can use optical devices, such as cameras or displays, to control various manipulator components (e.g., robotic arms controlling medical instruments) used in completion of various tasks or phases of the surgical procedure. A surgeon can utilize a robotic medical system that handles various surgical tools to complete tasks or actions (e.g., incisions or suturing actions) based on visual feedback from one or more cameras and displays showing the surgeon the location of the tools with respect to the surgical site to facilitate surgeon’s visual guidance. However, as different surgical tasks or phases can involve different types and sizes of tools and varying control area or space within which the tasks are performed, it can be challenging to maintain the manipulator components and the visual devices appropriately positioned, focused or controlled at the most suitable scales or settings with respect to the surgical site.

[0004] The technical solutions of this disclosure can overcome these challenges by measuring and presenting objective performance metrics, based on triangulation metrics of the manipulation components and the visual devices, to facilitate the surgeon’s improved performance with respect to each of the surgical tasks or phases. For example, the technology can measure and present a level of performance of a medical phase, or a task performed via arobotic medical system, using a triangulation function with inputs that correspond to vertices established by two manipulator components and an optical component. The triangulation metric can be compared with a threshold. The threshold can be specific to a type of task, phase, medical procedure, type of robotic medical system, medical environment, or surgeon. In doing so, the technical solutions can determine, compare and facilitate adjustment for the scaling of the controls, positioning of the medical tools, improving available range of motion or field of view for various instruments in different types of tasks of phases of a procedure, thereby facilitating improving the performance of a robotic surgery.

[0005] At least one aspect of the technical solutions is directed to a system. The system can include one or more processors, coupled with memory. The one or more processors can identify at least one of a type of task or a type of phase of a medical procedure. The one or more processors can determine vertices associated with a first manipulator component, a second manipulator component, and an optical device. The first manipulator component, the second manipulator component and the optical device can be used to perform the at least one of the type of task or the type of phase of the medical procedure. The one or more processors can generate, via a triangulation function, a metric based on the vertices. The one or more processors can provide a visual indication of a performance of the medical procedure based on a comparison of the metric with a threshold established for the type of task, the type of phase, or the type of procedure.

[0006] The first manipulator component and the second manipulator component can be communicatively coupled with a robotic medical system. The robotic medical system can be configured to perform at least a portion of the medical procedure. The one or more processors can be configured to identify a time-series of the vertices associated with the first manipulator component, the second manipulator component, and the optical device. The one or more processors can generate, via the triangulation function, a time-series of the metric using the time-series of the vertices.

[0007] The one or more processors can be configured to provide, on a graphical user interface that displays the medical procedure, a real-time visual indication of the time-series of the metric. The one or more processors can be configured to calibrate the first manipulator component and the second manipulator component prior to detection of the vertices. The one or more processors can be configured to select the threshold based on an anatomical structure associated with the medical procedure. The one or more processors can be configured to selectthe threshold based on a type of the first manipulator component or a type of the second manipulator component.

[0008] The one or more processors can be configured to detect, subsequent to provision of the visual indication of the performance, a clutch operation for at least one of the first manipulator component, the second manipulator component, or the optical device. The one or more processors can be configured to generate second vertices subsequent to detection of the clutch operation. The one or more processors can be configured to provide a second visual indication of a second performance of the medical procedure based on the second vertices. The second performance can be greater than the performance.

[0009] The one or more processors can be configured to identify a swap from the first manipulator component to a third manipulator component. The third manipulator component can be used to perform the at least one of the type of task or the type of phase of the medical procedure. The one or more processors can be configured to determine second vertices associated with the third manipulator component, the second manipulator component and the optical device. The one or more processors can be configured to generate a second metric based on the second vertices. The one or more processors can be configured to provide a second visual indication of a second performance of the medical procedure based on the comparison of the second metric with one of the threshold or the metric.

[0010] The one or more processors can be configured to determine, based on the metric less than or equal to the threshold, to provide guidance to improve the performance of the medical procedure. The one or more processors can be configured to provide, for display on a display device, the guidance to improve the performance of the medical procedure. The guidance can include to perform a clutch operation on at least one of the first manipulator component, the second manipulator component or the optical device. The visual indication of the performance can include at least one of a color, a numerical score, a grade, or a binary value.

[0011] The one or more processors can be configured to provide at least one of a haptic indicator or an audio indicator based on the performance of the medical procedure. The first manipulator component, the second manipulator component, and the optical device can be coupled with a console. The console can control one or more instruments used to perform the medical procedure. The optical device can include a camera or a headset.

[0012] At least one aspect is directed to a method. The method can include one or more processors coupled with memory identifying a first manipulator component, a secondmanipulator component, and an optical device that perform a task during a medical session. The method can include the one or more processors detecting vertices associated with the first manipulator component, the second manipulator component, and the optical device during performance of the task during the medical session. The method can include the one or more processors determining, using a triangulation function and the vertices, a metric indicative of performance of at least a portion of the medical session that includes the task. The method can include the one or more processors displaying, via a display device, a visual indication of the metric.

[0013] The method can include the one or more processors identifying a type of the task and selecting a threshold based on the type of task. The method can include the one or more processors comparing the metric with the threshold to determine the performance. The method can include the one or more processors identifying a type of the task. The method can include the one or more processors normalizing the metric based on the type of task to generate a normalized metric. The method can include the one or more processors comparing the normalized metric with a threshold to determine the performance.

[0014] At least one aspect is directed to a non-transitory computer-readable medium storing processor executable instructions. The instructions, when executed by one or more processors, can cause the one or more processors to determine vertices associated with a first manipulator component, a second manipulator component, and an optical component. The first manipulator component, the second manipulator component and the optical component can be used to perform at least a portion of the medical procedure. The instructions, when executed by the one or more processors, can cause the one or more processors to determine, via a triangulation function, a performance of the at least the portion of the medical procedure based on the vertices. The instructions, when executed by the one or more processors, can cause the one or more processors to display, via a display device, a visual indication of the performance with a video of the medical procedure captured by a camera.

[0015] The instructions, when executed by the one or more processors, can cause the one or more processors to identify a type of task or a type of phase associated with the at least the portion of the medical procedure. The instructions, when executed by the one or more processors, can cause the one or more processors to determine, via the triangulation function, a metric based on the vertices. The instructions, when executed by the one or more processors, can cause the one or more processors to select a threshold based on the type of the task or thetype of the phase. The instructions, when executed by the one or more processors, can cause the one or more processors to determine the performance based on a comparison of the metric and the threshold.

[0016] These and other aspects and implementations are discussed in detail below. The foregoing information and the following detailed description include illustrative examples of various aspects and implementations and provide an overview or framework for understanding the nature and character of the claimed aspects and implementations. The drawings provide illustration and a further understanding of the various aspects and implementations and are incorporated in and constitute a part of this specification. The foregoing information and the following detailed description and drawings include illustrative examples and should not be considered as limiting.BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are not intended to be drawn to scale. Like reference numbers and designations in the various drawings indicate like elements. For purposes of clarity, not every component can be labeled in every drawing. In the drawings:

[0018] FIG. 1 depicts an example system to implement surgical performance management via triangulation adjustment.

[0019] FIG. 2 illustrates a graph of a performance of medical procedures with and without triangulation adjustment.

[0020] FIG. 3 illustrates an example system providing indications for patient and surgeon side triangulations to improve the performance on the robotic medical system.

[0021] FIG. 4 illustrates an example system for determining and adjusting the surgeon side triangulation.

[0022] FIG. 5 A illustrates a series of example presentations of patient side triangulations and the corresponding display views of tasks performed.

[0023] FIG. 5B illustrates an example in which a third manipulator component is used in the context of triangulation.

[0024] FIG. 6A illustrates examples diagrams of relations between left and right hand surgeon positionings in control space and manipulator components in a display view.

[0025] FIGs. 6B-6D illustrate examples of plots of triangulation metrics for surgeon and patient side triangulations corresponding to examples of FIG. 6 A.

[0026] FIG. 7 illustrates an example selection of types of indications that can be displayed to a user.

[0027] FIG. 8 illustrates an example view of a simulation training that can be used for providing practice or training on triangulation to the surgeons.

[0028] FIG. 9 illustrates an example flowchart outlining the operations of a method 900 for implementing triangulation is illustrated.

[0029] FIG. 10 illustrates an example of a surgical system, in accordance with some aspects of the technical solutions.

[0030] FIG. 11 illustrates an example block diagram of an example computer system is shown, in accordance with some aspects of the technical solutions.DETAILED DESCRIPTION

[0031] Following below are more detailed descriptions of various concepts related to, and implementations of, systems, methods, apparatuses for surgical performance management via triangulation adjustment. The various concepts introduced above and discussed in greater detail below can be implemented in any of numerous ways.

[0032] Although the present disclosure is discussed in the context of a surgical procedure, in some embodiments, the present disclosure can be applicable to other medical sessions or environments or activities, as well as non-medical activities where removal of irrelevant information is desired.

[0033] In the context of surgical robotics, triangulation can apply to a coordinated positioning and alignment of manipulator components (e.g., manipulator arms holding medical instruments) and optical devices (e.g., data capture devices, such as cameras) to facilitate implementation of various surgical phases or tasks. Triangulation can include determining positioning of the medical tools or manipulator components used in surgical tasks and identifying the available range of motions for such tasks to be implemented. Triangulation can be used to make adjustments to the positioning or orientation of the user’s hands or manipulator components to facilitate an improved ergonomic positioning for the user and an increased range of motions for the intended tasks. By evaluating the vertices or spatial coordinates of thecontrolled devices, triangulation can be used to improve the quality of surgical performance by increasing the available range of motions, both on the surgeon side (e.g., at the control-side of the medical robotic system) and on the patient side (e.g., in the medical environment in which the patient is undergoing surgery), thereby increasing the efficiency of the overall tasks performed during the surgery.

[0034] Triangulation can include monitoring, determining or adjusting point locations of surgeon’s arms and eyes or robotic arms and visual devices so as to form triangles that can be oriented, aligned or positioned so as to provide a suitable range of motion, control or ergonomics for improved surgical performance. Triangulation can include point locations forming triangles on the surgeon side, such as for example, a triangle of locations of surgeon’s left and right hands and surgeon’s eyes. Depending on surgeon’s positioning or movement, these locations can form triangles which can be arranged in orientations that may be ergonomically suitable or unsuitable for a sustained high level performance. Triangulation can include point locations forming triangles on the patient’s side, such as for example, a triangle of locations of a robot operated left-side and right-side manipulator components and a positioning of a camera or a display for providing visual feedback. These triangles can affect a range of available motion, control, ergonomic positioning or visibility and can be adjusted to improve the overall performance of the surgery.

[0035] Technical solutions of the present disclosure can include generating and providing triangulation metrics for robotic medical systems to improve a robotic surgery performance. In a course of a robotic surgery, a surgeon can use one or more optical devices, such as cameras or displays for visual feedback on the positioning of the manipulator components and medical tools used for performing surgical tasks at a surgical site. However, as different tasks or phases of a surgery can involve usage of different medical tools or manipulator components, it can be difficult for the surgeon to maintain a sufficient or suitable ergonomic positioning, an effective range of motion, or a desirable control level for a given surgical task or a phase. Also, as different medical tools or manipulator components can fall in or out of a field of view or focus of the images captured by the optical devices, it can be challenging for the surgeon to maintain a sufficient visibility of the surgical site and tools used, potentially adversely affecting the performance.

[0036] For example, a robotic medical system can include a clutch foot pedal for moving both manipulator controller (e.g., the hand controls) without moving the instruments handledby the hand controls, which a surgeon can use to adjust both hand positions together. The robotic medical system can also include a finger clutch in each hand control to allow the movement of the hand control without moving the corresponding instrument only, which can be used to adjust one hand position. However, during the surgery the surgeons cannot see their hands. Instead, surgeons can see the surgical site via a stereo viewer, which can cause the surgeons to lose awareness of the positioning of their hands. For example, if a surgeon performs a surgical procedure with stretched arms, collided arms, crossed arms or with arms that are open and away from each other (e.g., in ergonomically awkward positioning with respect to each other), then these arm positions can be ergonomically unsuitable for bimanual cooperative tasks. Operating in such a manner can cause fatigue during the course of the surgery, potentially adversely affecting the surgical performance in task, phase, or procedure (e.g., case) levels.

[0037] The technical solutions can overcome these challenges by providing for triangulation or readjustment of the positioning or orientation of the manipulator components, optical devices and improvement of the user ergonomics in accordance with various phases or tasks, allowing the surgeon to operate with improved efficiency across different surgical phases or tasks. For instance, the triangulation metric can measure the ergonomic states of hand controls and indicate the current state in real-time to suggest correction for surgeon’s hand positions, positioning of cameras or other tools, while providing performance review in the task, phase, and case levels with respect to the surgery. The technical solutions can measure vertices of the manipulator components and optical devices and present to the surgeon indications of objective performance metrics determined based on the triangulation metrics of such manipulator components and optical devices. These notifications can allow the surgeon to correct for any inefficiencies in the positioning or control during the course of the surgery. The technical solutions can compare triangulation metrics of the manipulator components and / or optical devices with threshold metrics of specific types of tasks or phases and continuously monitor the level of performance, providing recommendations to make adjustments. For example, the technical solutions can recommend adjusting a range of motion or control to a more ergonomically suitable position, a level or a scale of control to apply to a manipulator components, or a particular medical tool, as well as adjusting (e.g., expanding or focusing) a field of view of a camera used to monitor the surgical site during the performance of a surgical action.

[0038] In one example, the technical solutions can identify one or more tasks or phases of a medical procedure (e.g., a surgery) in which a surgeon uses one or more manipulator components (e.g., robotic arms and / or medical tools handled by the arms) and an optical device for visual guidance and control. The technical solutions can detect vertices between the manipulator components and the optical device, such as in the context of a field of view of the user’s optical device and the patient’s body. The technical solutions can generate one or more metrics based on the detected vertices and determine a performance of the medical procedure based on the metrics. The performance can be determined based on a comparison of the metric with a threshold for the given task or phase of the surgery determined or established by the system. The technical solutions can generate and display for the surgeon an indication of the performance, recommending to the surgeon any adjustments or corrections to the location of the controls for the medial tools or visual devices. The technical solutions can correspond to any type of surgery, such as an open surgery, laparoscopic surgery or a robotic surgery and can span different types of medical surgical platforms, such as robotic systems, individual machine arms, or robot agnostic systems.

[0039] The technical solutions can utilize time-series of a triangulation metrics with respect to a performance in any type of task or phase of a medical procedure. The time series of triangulation metrics can be normalized across different types of tasks or phases. The time series of triangulation metrics can account for a variety of factors including tasks, location of targeting surgical region, anatomical structures, patient body posture or types of surgical instruments and can involve a patient side triangulation metric and a surgeon side triangulation metric. Technical solutions can include a triangulation function that can provide a sum value (e.g., a weighted sum) based on multiple vertices formed from one or more of: camera (rotated / flipped), display (2d or 3d), eye, head, body or other information or data corresponding to the user (e.g., surgeon) or the robotic system or tools the surgeon uses. The technical solutions can utilize the metrics to implement the calibration, alignment and / or symmetry of the components or tools.

[0040] The technical solutions can include pre-operative functionality to recommend most suitable or desirable (e.g., the best) placement of the manipulator component or visual device in order to achieve the best triangulation in a given case (e.g., phase or task). The technical solutions can recommend adjustments or changes to patient positioning, port placement (e.g., entry or incision positions on the patient’s body) or manipulator component positioning andinsertion orientation. Such recommendations can be done based at least on the historical triangulation metrics. The technical solutions can provide alarm if suboptimal or undesirable placement of the tools or components are detected. Technical solutions can identify a user and then provide historical triangulation metrics for same user for same type of medical procedure. The technical solutions can provide the guidance based on same ergonomic or port setup, such as entry or incision positions of the manipulator components or visual device on the patient’s body.

[0041] The technical solutions can provide an intra-operative functionality to re-adjust vertices based on changes in the system, settings or parameters. For example, changes can be made to the vertices based on a finger clutch, clutch foot pedal, camera clutch, camera control pedal, motion scales, and third manipulator component. A change in surgical region or patient position, visibility or hidden target, as well as pressing of a pedal to swap instruments being controlled can each lead to adjustments and recalculations of the vertices and / or metrics. Technical solutions can guide the surgeon to maintain optimal triangulation metric during surgery and provide visual indicator of performance metric: color, score, binary, scale. Technical solutions can provide haptic or audio indicators of performance metric, overlay realtime triangulation metric on user interface, overlay hands or tools on the display, overlay clutch animations to provide user guidance and calculate the metrics at a sample rate (e.g., 50 Hz).

[0042] Technical solutions can provide post-operative functionality, such as establishing chapters in video recording with low / high triangulation performance metrics for future use and training. Technical solutions can determine best port placement for a specific type of procedure based on post-procedure analysis of triangulation metric. Time series (e.g., 500 Hz) triangulation with other objective performance metrics, system events in task, phase, and case levels, as well as correlation between triangulation metric versus other OPIs or metrics can be determined based on the gathered data. Technical solutions can provide training simulation, including for example, recommend particular simulations for user to improve triangulation performance metric per task, phase or case.

[0043] FIG. 1 depicts an example system 100 for surgical performance management via triangulation adjustment. Example system 100 can include a robotic medical system 120 which can be used by a surgeon to perform a surgery on a patient. Robotic medical system 120 can be deployed in a medical environment 102, which can include any facility for performing medical procedures, such as a surgical facility, or an operating room. Medical environment 102 caninclude various medical instruments and tools that the robotic medical system 120 can use for performing surgical patient procedures, whether invasive, non-invasive, in-patient, or outpatient procedures.

[0044] The medical environment 102 can include one or more data capture devices 110 (e.g., optical devices, such as cameras or sensors) for capturing sensor data 152 (e.g., images or videos of a surgery). The medical environment 102 can include one or more visualization tools 114 to gather the captured sensor data 152 and process it for display to the user (e.g., a surgeon) at one or more displays 116. A display 116 can present sensor data 152, such as images or video frames showing manipulator components 122 of the robotic medical system 120 handling, manipulating, holding or otherwise utilizing medical tools 112 to perform surgical tasks at the surgical site. Coupled with the robotic medical system 120, via a network 101, can be a data processing system (DPS) 130. DPS 130 can include one or more procedure monitors 140, data repositories 150, triangulation functions 170, component calibrators 162, indicators 164, clutch functions 180, interfaces 182 and manipulator controllers 184.

[0045] DPS 130 can include a procedure monitor 140 having one or more phase detectors 142 for detecting phases 144 of medical procedures and one or more task detectors 146 for detecting tasks 148 of medical procedures according to the sensor data 152 captured by data capture devices. Procedure monitor 140 can monitor the medical sessions to detect or identify individual phases 144 and tasks 148 for triangulation. A data repository 150 can store one or more sensor data 152 as data streams, along with one or more performances 154 and thresholds 156 for various phases 144 or tasks 148 to be used in performance comparisons or assessments. The manipulator controller 184 can include or provide a control space 186 within which a user (e.g., surgeon) can manipulate or control the manipulator components 122 to utilize medical tools 112 in the medical environment 102. The triangulation function 170 can include one or more vertices generators 172 for measuring or generating vertices 174 and one or more performance functions 176 for generating metrics 178 indicative of the level of performance of the surgical tasks 148 or phases 144 according to the vertices 174. The component calibrator 162 can be used for calibrating medical tools 112 or any other manipulation components, while a clutch function 180 can be provided for resetting the vertices or repositioning the manipulator controller. DPS 130 can include indicators 164 to provide indications 168, using for example, interface 182.

[0046] The system 100 can include one or more data capture devices 110 (e.g., optical devices) for collecting any data, such as videos, images, manipulator kinematics and system events, which can be used for generating or providing triangulation metrics 178. Data capture devices 110 can include, for example, optical devices, such as cameras or other image capture devices for capturing videos or images from a particular viewpoint within the medical environment 102. The data capture devices 110 can be positioned, mounted, or otherwise located to capture content from any viewpoint that facilitates the data processing system recognizing phases of a procedure. Data capture devices 110 can include any of a variety of sensors, cameras, video imaging devices, infrared imaging devices, visible light imaging devices, intensity imaging devices (e.g., black, color, grayscale imaging devices, etc.), depth imaging devices (e.g., stereoscopic imaging devices, time-of-flight imaging devices, etc.), medical imaging devices such as endoscopic imaging devices, ultrasound imaging devices, etc., non-visible light imaging devices, any combination or sub-combination of the above mentioned imaging devices, or any other type of imaging devices that can be suitable for the purposes described herein. Data capture devices 110 can include cameras that a surgeon can use to perform a surgery and observe manipulation components within a purview of field of view suitable for the given task performance.

[0047] Data capture devices 110 can capture, detect, or acquire sensor data, such as videos or images, including for example, still images, video images, vector images, bitmap images, other types of images, or combinations thereof. The data capture devices 110 can capture the images at any suitable predetermined capture rate or frequency. Settings, such as zoom settings or resolution, of each of the data capture devices 110 can vary as desired to capture suitable images from a particular viewpoint. The data capture devices 110 can have fixed viewpoints, locations, positions, or orientations. The data capture devices 110 can be portable, or otherwise configured to change orientation or telescope in various directions. The data capture devices 110 can be part of a multi-sensor architecture including multiple sensors, with each sensor being configured to detect, measure, or otherwise capture a particular parameter (e.g., sound, images, or pressure).

[0048] Data capture devices 110 can include any type and form of a sensor, such as a positioning sensor, a velocity sensor, an acceleration sensor, a vibration sensor, a motion sensor, a pressure sensor, a light sensor, a distance sensor, a current sensor, a focus sensor, a temperature or pressure sensor or any other type and form of sensor used for providing data onmedical tools 112, manipulator components 122, or data capture devices (e.g., optical devices). For example, a data capture device 110 can include a location sensor, a distance sensor or a positioning sensor providing coordinate locations of a manipulator component 122, medical tool 112 or a data capture device 110. Data capture device 110 can include a sensor providing information or data on a location, position or spatial orientation of an object (e.g., manipulator component 122, medical tool 112 or a lens of data capture device 110) with respect to a reference point. The reference point can include any fixed, defined location used as the starting point for measuring distances and positions in a specific direction, serving as the origin from which all other points or locations can be determined.

[0049] Display 116 can show, illustrate or play sensor data 152 (e.g., images or videos) in which manipulator components 122 and / or medical tools 112 and location of the surgery are presented within a control space 186. Control space 186 can include any area, surface, volume, device or system (e.g., a touch screen, a control screen or a console) within which surgeon can manipulate or control the medical tools 112 via manipulator components 122 based on the visual feedback from displayed images, videos or other sensor data 152. For example, display 116 can display a rectangular image (e.g., one or more video frames) of the surgical site along with at least a portion of medical tools 112 (e.g., instruments) controlled or manipulated by manipulator components 122 holding, managing or controlling such tools. Display 116 can provide compiled or composite images generated by the visualization tool 114 from a plurality of data capture devices 110 to provide a visual feedback from one or more points of view.

[0050] The visualization tool 114 that can be configured or designed to receive any number of different sensor data streams 152 from any number of data capture devices 110 and combine them into a single data stream displayed on a display 116. The visualization tool 114 can be configured to receive a plurality of data stream components and combine the plurality of data stream components into a single data stream. For instance, the visualization tool 114 can receive a visual sensor data 152 from one or more medical tools 112, sensors or cameras with respect to a surgical site or an area in which a surgery is performed. The visualization tool 114 can incorporate, combine or utilize multiple types of data (e.g., positioning data of a medical tool 112 along sensor readings and any number of images) to generate a single output to present on a display 116. Visualization tool 114 can present locations of medical tools 112 and / or manipulator components 122 along with locations of any reference points or surgical sites.

[0051] Medical tools 112 can be any type and form of tool or instrument used for surgery, medical procedures or a tool in an operating room or environment. Medical tool 112 can be imaged by, associated with or include an image capture device. For instance, a medical tool 112 can be a tool for making incisions, a tool for suturing a wound, an endoscope for visualizing organs or tissues, an imaging device, a needle and a thread for stitching a wound, a surgical scalpel, forceps, scissors, or any other tool or instrument to be used during a surgery. Medical tools 112 can include hemostats, trocars, surgical drills, suction devices or any instruments for use during a surgery. The medical tool 112 can include other or additional types of therapeutic or diagnostic medical imaging implements. The medical tool 112 can be configured to be installed in a robotic medical system 120 and / or to be manipulated by manipulator components 122.

[0052] Manipulator components 122 can include any device for manipulating or controlling medical tools 112. Manipulator components 122 can include electrical, mechanical or electromechanical devices, apparatuses or systems for designed or configured to hold, control, manipulate or use any one or more medical tools 112 with a level of control and precision suitable for particular surgical tasks or phases. Manipulator components 122 can include, for example, robotic manipulator arms that can be attached to and controlled by or via a robotic medical system 120. Manipulator components 122 can be remotely controlled (e.g., moved, adjusted, replaced, reset, and configured) from a data processing system 130, such as via a manipulator controller 184. Manipulator components 122 can be reconfigurable or adjustable to accommodate different medical tools 112. For instance, manipulator components 122 can include robotic arms equipped with interchangeable end-effectors, which can handle or switch between graspers, scissors, needle holders, or electrocautery devices during a surgery. For example, manipulator components 122 can include arms that are equipped with haptic feedback systems to provide tactile information. For example, manipulator components 122 can be integrated with imaging technology (e.g., data capture devices 110 and / or displays 116), allowing for real-time visualization and precise manipulation of medical tools 112 at the surgical site.

[0053] The robotic medical system 120 can be a computer-assisted system configured to perform a surgical or medical procedure or activity on a patient via or using or with the assistance of one or more robotic components or medical tools. The robotic medical system 120 can include one or more manipulator arms (e.g., manipulator components 122) for performingcomputer-assisted medical tasks using medical tool 112 installed on the manipulator arm. The images (e.g., video images) captured by the medical tool 112 can be sent to the visualization tool 114. The robotic medical system 120 can include one or more input ports to receive direct or indirect connection of one or more auxiliary devices. For example, the visualization tool 114 can be connected to the robotic medical system 120 to receive the images from the medical tool when the medical tool is installed in the robotic medical system (e.g., on a manipulator arm of the robotic medical system). The visualization tool 114 can combine the data stream components from the data capture devices 110 and the medical tool 112 into a single combined data stream for presenting on a display 116.

[0054] The system 100 can include a data processing system 130. The data processing system 130 can be deployed in or associated with the medical environment 102, or it can be provided by a remote server or be cloud-based. The data processing system 130 can include an interface 182 designed, constructed and operational to communicate with one or more component of system 100 via network 101, including, for example, the robotic medical system 120. Data processing system 130 can be implemented using instructions stored in memory locations and processed by one or more processors, controllers or integrated circuitry. Data processing system 130 can include functionalities, computer codes or programs for generating and providing triangulation metrics 178 for various tasks or phases of the robotic surgeries in order to facilitate improved performance for the tasks 148 or phases 144 using vertices 174, thresholds 156 and performances 154.

[0055] The data repository 150 can include one or more data files, data structures, arrays, values, or other information that facilitates operation of the data processing system 130. The data repository 150 can include one or more local or distributed databases and can include a database management system. The data repository 150 can include, maintain, or manage a sensor data 152. The sensor data 152 can include or be formed from one or more of a video stream, image stream, stream of sensor measurements, event stream, or kinematics stream. The sensor data 152 can include data collected by one or more data capture devices 110, such as a set of 3D sensors from a variety of angles or vantage points with respect to the procedure activity (e.g., point or area of surgery).

[0056] Sensor data 152 can include an event stream which can include a stream of event data or information, such as finger clutch, clutch foot pedal, cameral control pedal, arm swap pedal, and energy pedal events. Events can include packets, which identify or convey a state ofthe robotic medical system 120 or an event that occurred in association with the robotic medical system 120 or surgical or medical surgery being performed with the robotic medical system. Data of the event stream can be captured by the robotic medical system 120 or a data capture device 110. Event stream can include a state of the robotic medical system 120 indicating whether the medical tool 112 is calibrated, adjusted or includes a manipulator component 122 installed on a robotic medical system 120. Event stream can include data on whether a robotic medical system 120 was fully functional (e.g., without errors) during the procedure. For example, when the medical tool 112 is installed on a manipulator component 122 of the robotic medical system 120, a signal or data packet(s) can be generated indicating that the medical tool 112 has been installed on the manipulator component 122 of the robotic medical system 120. Another example state of the robotic medical system 120 can indicate whether the visualization tool 114 is connected, whether directly to the robotic medical system 120 or indirectly through another auxiliary system that is connected to the robotic medical system 120.

[0057] Sensor data 152 can include a kinematics stream data which can refer to or include data associated with one or more of the manipulator components 122 (e.g., arms) or medical tools 112 (e.g., instruments) attached to the manipulator components 122. For instance, the kinematics data can include data associated with one or more of the manipulator controller to determine surgeon side triangulation metric. Data corresponding to manipulator components 122 and / or medical tools 112 can be captured or detected by one or more displacement transducers, orientational sensors, positional sensors, or other types of sensors and devices to measure parameters or generate kinematics information. The kinematics data can include sensor data along with time stamps and an indication of the medical tool 112 or type of medical tool 112 associated with the sensor data 152.

[0058] The data repository 150 can include any storage or repository system. The data repository 150 can store performances 154 and their corresponding thresholds 156. Performances 154 can correspond to vertices or data on device or tool positioning or orientation with respect to any particular phases 144 or tasks 148 of medical procedures. Performances 154 can correspond to control or spatial configurations of medical tools 112 and / or manipulator components 122 within a control space 186 for a given task 148 or phase 144. Performances 154 can include, utilize, correspond to or refer to thresholds 156 providing a range of parameters or ranges, orientations or positions with respect to particular surgicalphases or tasks. Performances 154 can include series of thresholds 156 to indicate a range of positions, orientations, alignment or symmetries between manipulator components 122 and / or data capture devices 110 to allow a surgeon to manipulate medical tools 112 effectively and within a maximized ergonomic range of motions. Thresholds 156 can correspond to, define or indicate a space (e.g., area or volume) of vertices 174, such as in the context of a control space 186, within which a surgeon can maximize the range of motion, ergonomic positioning or the range of control over the medical tools 112 used in a particular task 148 or phase 144. As performances 154 can correspond to specific phases or tasks for any number of medical procedures, spatial configurations for the performances 154 can be task-specific, providing optimal control and utilization of tools depending on the tasks or phases.

[0059] For instance, performance 154 stored in a data repository 150 can include a desired or recommended range of tool positions or orientations for making a precise action, such as range of locations of a scalpel in a control space 186 to maximize the chances of making a correct incision in a particular surgery. For example, performance 154 can include a desired range of tool positions, orientations or scaling of controls with respect to the motion of the medical tools 112 to make accurate imaging or inspection of a cut or a precise suturing action. Performances 154 can correspond to any range of motions or actions by manipulator components 122 or data capture devices 110 (e.g., optical devices) such as locations for imaging or implementing suturing actions to increase the likelihood of a maximum alignment for secure wound closure. Performance 154 can correspond to a retraction task 148 in which a performance can focus on maintaining tissue separation, enhancing visibility, or a hemostasis task 148 in which the performance can focus on controlling bleeding with the right tool orientation.

[0060] Thresholds 156 can correspond to particular performances 154 for particular phases 144 or tasks 148. Thresholds 156 can correspond to either patient side or surgeon side triangulation aspects. For example, a threshold 156 can include the surgeon side triangulation thresholds or ranges of movement, such as surgeon's ergonomic data or ranges, or range or limit of motion of hands. The threshold 156 can refer to or include a numerical value that can be used to determine whether a metric indicative of performance 154 is satisfactory. Thresholds 156 can include any constraints, limits or specifications within the control space 186 that can be met to allow for an effective or desired performance 154. For example, a threshold 156 can include a threshold level of a performance 154 that can be considered acceptable, or desirable.Threshold 156 can include one or more limits to a range of positions, locations or orientations of manipulator components 122 and / or medical tools 112 with respect to particular phases 144 and / or tasks 148. Threshold 156 can correspond to a level of performance 154 associated with a range of motions, positions, locations, scale of control or manipulation of medical tools 112 during particular surgical phases 144 or tasks 148. Threshold 156 can establish, provide or indicate boundaries for permissible ranges of positions and orientations, safeguarding against undesirable level of performance 154 with respect to any phases 144 or tasks. For instance, in an incision task 148, the threshold 156 can specify a maximum depth of the incision to avoid damaging underlying structures and / or any locations, positioning or orientation of medical tools 112 and / or manipulator components 122 to achieve such a level of performance 154.

[0061] Phase detectors 142 can include any combination of hardware and software for identifying, determining or detecting phases 144 of a medical procedure. Phase detector 142 can include a functionality for determining or detecting a phase 144 of a medical procedure from sensor data 152, such as by analyzing visual data (e.g., images or videos of the surgery), such as images from display 116. Phase detectors 142 can include or utilize one or more machine learning models, which can be trained, established, configured, updated, or otherwise provided by a model generator to identify, predict, classify, categorize, or otherwise detect phases 144 of medical procedure. Phase detector 142 can identify a phase 144 being implemented. Phase detector 142 can include a model trained using historical data involving surgeries and including various phases of various medical procedures, allowing the model to recognize a particular phase of a particular procedure from a sensor data 152 input into a phase detector 142 (e.g., ML model). Phase detector 142 can select or identify a desired performance 154 and the corresponding thresholds 156 for the identified phase 144 to compare against using the vertices 174 and / or metrics 178 provided by the triangulation function 170 to determine the current level of performance by the surgeon.

[0062] Phases 144 can be stored in a data repository and can otherwise be detected by one or more phase detectors 142. A phase 144 can refer to or include any operative phases or nonoperative phases (e.g., operating room related phases), such as room preparation, robot setup, performance of a medical procedure, turn over, or cleaning, for example. The phases 144 can include operative phases, such as exposure, dissection, transection, reconstruction, and extraction of a surgery. Exposure phase 144 can refer to or include the process of visualizing and accessing a surgical site by creating a clear and adequate field of view. Dissection phase144 can refer to or include cutting, separating and removing tissues or anatomical structures to gain access to specific areas, identify structures, or perform surgical procedures. Transection phase 144 can refer to or include severing or cutting a structure, such as a blood vessel, nerve, or organ using a surgical instrument. Extraction phase 144 can refer to or include the removal of a tissue, organ, foreign object, or other anatomical structure from the body. Reconstruction phase 144 can refer to or include the process of restoring or rebuilding a damaged or missing tissue, organ, or body part, and can include techniques or tasks such as grafting, suturing, or using prosthetic materials to recreate the structure and restore form and function.

[0063] Task detectors 146 can include any combination of hardware and software for identifying, determining or detecting tasks 148 of a medical procedure. Task detector 146 can include a functionality for determining or detecting a task 148 from sensor data 152, such as by analyzing visual data (e.g., images or videos of the surgery) from display 116. Task detectors 146 can include or utilize one or more machine learning models, which can be trained, established, configured, updated, or otherwise provided by a model generator to identify, predict, classify, categorize, or otherwise detect tasks 148. Task detector 146 can identify a task 148 of a medical procedure being implemented. Task detector 146 can include a model trained using historical data involving surgeries and including various phases of various medical procedures, allowing the model to recognize a particular task 148 from sensor data 152 input into a task detector 146 (e.g., ML model). Task detector 146 can select or identify a desired performance 154 and the corresponding thresholds 156 for the identified task 148 to compare against using the vertices 174 and / or metrics 178 provided by the triangulation function 170 to determine the current level of performance by the surgeon.

[0064] Tasks 148 can include any action or intervention implemented during a medical treatment, such as a surgery. Task 148 can include a series of actions and interventions conducted to diagnose, treat, or alleviate a patient. Tasks 148 can include or involve usage of medical tools 112. Task 148 can include, for example, making an incision, using a scalpel to make an opening. Task 148 can include a dissection task using scissors and forceps to manipulate tissues of a patient. Task 148 can include a hemostasis task in which hemostats can be used to control bleeding. Task 148 can include a suturing tasks in which a needle and a needle holder can be controlled by a manipulator component 122 to close incisions with surgical sutures. Tasks 148 can include any series of actions using manipulator components 122 and / or medical tools 112, such as using a biopsy forceps to extract tissue samples forexamination, or endoscopies. Tasks 148 can vary across phases 144 and can be specific to particular medical operations implemented.

[0065] Manipulator controller 184 can include a combination of hardware and software for using, moving and controlling manipulator components 122 (e.g., robotic arms) to hold and manipulate medical tools 112 during a medical procedure. Manipulator controller 184 can include or utilize inputs from a user interface 182 having for example knobs, joysticks, computer mice or touch screens in which a surgeon can select, move, adjust or otherwise manipulate medical tools 112 using manipulator components 122. Manipulator controller 184 can use the inputs to control the manipulator components according to the surgeon’s movement, indications or instructions. Manipulator controller 184 can include or provide an interface 182 or control space 186 in which the user can provide commands for adjusting, calibrating, selecting, adding or removing various manipulator components 122. Manipulator controller 184 can include a system or an interface 182 for controlling, moving, adjusting or manipulating medical tools 112 within a control space 186, or vice versa.

[0066] A control space 186 can include or use any interface 186, whether in 2D or 3D, where the user can manipulate manipulator components 122 (e.g., holding medical tools 112) within specified coordinates or boundaries. A control space 186 can include any include any one-dimensional (e.g., a line), a two dimensional (e.g., area) or three-dimensional (e.g., volume) space within which manipulator components 122 can be moved, adjusted or manipulated to perform medical tasks 148 or phases 144. Control space 186 can include a coordinate system (e.g., vertices 174) that can scale, map or otherwise correspond to the locations, positions or movements of manipulator components 122 within a medical environment 102. Control space 186 can include or define vertices 174 of various parts of manipulator components 122 or medical tools 112 within a confined space of a particular line, area or volume. For example, a control space 186 can include, correspond to, or utilize an area (e.g., a 2D area of a touch screen) corresponding to a scaled version of a physical space in a medical environment 102 (e.g., next to the patient) in which manipulator components 122 are being manipulated to use medical tools 112 for an ongoing surgery. For example, a control space 186 can include, correspond to, or utilize a volumetric space (e.g., 3D space) corresponding to a scaled version of a physical space in a medical environment 102 in which manipulator components 122 are being manipulated.

[0067] Control space 186 can allow for scaling of movement to be set or adjusted to any ratio between movement in the control space 186 and physical movement in the medical environment 102. Control space 186 scaling can be set, for example to, to a ratio of 1 : 1 (e.g., equal scaling in movement in a control space 186 and in medical environment 102). Control space 186 can be set to scaling of 1 :2, 1 :3, 1 :4, 1 :5, 1 : 10, 1 :20, 1 :50, 1 : 100, 1 :500, or one to more than 500. Similarly, scaling of movement can be adjusted to 1 :1 / 2, 1 : 1 / 3, 1 : 1 / 4, 1 : 1 / 5, 1 : 1 / 10, 1 : 1 / 20, 1 : 1 / 50, 1 : 1 / 100, 1 : 1 / 500 or one to less than 1 / 500. Scaling can impact the range of motion that a user (e.g., surgeon) may have within a control space 186 for any given manipulator component 122 or medical tool 112 and so adjustments (e.g., clutching) or recalibrations can be implemented to reset the range of motion or correspondence of vertices 174 in a control space 186 to locations or space of manipulator components 122 or medical tools 112 within medical environment 102.

[0068] Triangulation function 170 can include any combination of hardware and software for determining and adjusting triangulation of manipulator components and optical devices used in robotic surgeries. Triangulation function 170 can also include any functionality for determining the performance of the components or devices in the context of coordinated positioning, symmetry and alignment for both patient side triangulation (e.g., 310) and surgeon side triangulation (e.g., 312). For example, triangulation function 170 can include the functionality to determine the positioning or performance of the manipulator components 122, medical tools 112 and / or data capture devices 110 (e.g., cameras or displays used for visual feedback and control) and identify or detect a desired performance for the given task 148 or phase 144 based on the stored performances 154 and thresholds 156 (e.g., vertices 174) desired for the components and devices for the given tasks or phases. In some surgeries, such as for example, laparoscopic surgeries in which robotic system can be omitted, triangulation metrics can be used to triangulate only a surgeon side hand and site positioning using cameras.

[0069] In some examples, laparoscopic surgeries can have both surgeon and patient side triangulation. In such implementations, a surgeon can manipulate two lap instruments while looking at monitor that is away from the surgeon and the patient. The display may be placed further from surgeon’s eyes and the surgeon’s head direction may be not forward depending on the location / angle of the display. The display 116 used can include a 2D display without depth unlike 3D display 116 that can be used with a robotic system 120. The lap instrument tip canhave an inverted motion, such that moving of the handle towards a left side of the surgeon can move the tip on the display towards a right side.

[0070] Triangulation function 170 can include the functionality (e.g., vertices generator 174) to determine the desired range of motions, orientations, symmetries or spatial configurations of the manipulator components 122 and / or optical devices (e.g., data capture devices 110) to improve the ergonomic performance, spatial coordination and control for the given tasks 148 or phases 144. For example, the triangulation function 170 can measure or determine vertices 174 of manipulator components 122, medical tools 112 and / or data capture devices 110, as they are being currently used and can determine the metrics 178 indicative of performance for the currently performed tasks 148 or phases 144 of the medical procedure.

[0071] Triangulation function 170 can include coordinating the placement and alignment of manipulator components 122 and their corresponding medical tools 112, as well as any data capture devices 110 (e.g., cameras) utilized in surgical tasks 148 or phases 144. Triangulation function 170 can utilize vertices 174 to generate metrics 178 corresponding to or indicative of the performance associated with the available or maximized range of motions for controlling the devices (e.g., 122, 112 or 110), efficient or available ergonomic positioning for the user to efficiently manage the devices, or effective view for observation of actions performed. By assessing the vertices 174 of the controlled devices, triangulation function 170 can determine the metrics 178 indicative of the quality of performance (e.g., motion range, user ergonomics, efficiency in executing specific movements during surgery, effective field of view or vision).

[0072] Metrics 178 of the triangulation function can include any values or parameters that correspond to any measurement or indication of the quality or shape of a triangulation, including a surgeon-side triangulation or patient side triangulation. For example, metrics 178 can indicate or correspond to an area of a surgeon side or patient side triangle or a length of sides of a triangle. Metrics 178 can include, indicate or correspond to a ratio of sides of a triangle, such as a ratio of a bottom side of a triangle with respect to a left side or a right side of the triangle, such as in a triangle in which the bottom side is the longest side of a triangle. Metrics 178 can include, indicate or correspond to a ratio of the left or right side of the triangle with respect to the bottom side. In an inverted triangle (e.g., where longest side is the top side), the same or similar ratios can be used. Metrics 178 can include, indicate or corresponds to a size of a triangle (e.g., area of the triangle) whether on a flat surface, Euclidean space or a curved space, such as a surface of a patient’s body. For example, metrics 178 can indicate orcorrespond to a triangle projected on a curved surface or area. Metrics 178 can indicate performance or value of the current triangulation (e.g., surgeon-side or patient-side) and can be compared with threshold values of desired performances.

[0073] Triangulation function 170 can include, correspond to, or determine a surgeon side triangulation. The surgeon side triangulation can correspond to the usage of vertices 174 to address, determine or improve the performance or effectiveness of the surgeon’s ability to control, move or adjust the devices (e.g., 122, 112 or 110) in the control space 186 via which the surgeon controls the devices. The surgeon side triangulation can correspond to the range of motion, effectiveness of movements, scale of control or fields of view corresponding to the manipulator components 122, medical tools 112, data capture devices 110 and / or displays 116 from the surgeon’s point of view. For instance, the surgeon side triangulation can concern the surgeon’s range of motion or granularity of control of movement of the devices within the control space 186.

[0074] Triangulation function 170 can include, correspond to, or determine a patient side triangulation. The patient side triangulation can correspond to the usage of vertices 174 to address, determine or improve the performance or effectiveness of the movement, adjustment or control of the devices (e.g., 122, 112 and / or 110) in the space of the medical environment 102 in which the patient is being operated on. The patient side triangulation can correspond to the range of motion, effectiveness of movements, scale of control or fields of view corresponding to the manipulator components 122, medical tools 112, data capture devices 110 and / or displays 116 within the medical environment 102 (e.g., within a surgical room in which the surgery is performed on the patient). For instance, the patient side triangulation can concern the range of motion or granularity of control of movement of the devices within the space of the surgical site (e.g., at the body of the patient or within the body of the patient). Triangulation function 170 can include, correspond to, determine or implement any combination of the surgeon side triangulation and the patient side triangulation.

[0075] Triangulation function 170 can use a vertices generator 172 to measure, detect or determine vertices 174 for any number of manipulator components 122 and data capture devices 110 (e.g., optical devices). Vertices 174 can include any specific, fixed points within a defined space (e.g., control space 186 or medical environment 102). Vertices 174 can include coordinates that can indicate or serve as reference locations for precise positioning, control, or manipulating of medical tools 112, manipulator components 122 or data capture devices 110 ina medical environment 102 using a robotic medical system 120. Vertices generator 172 of the triangulation function 170 can capture vertices 174 from control coordinates, such as X-axis, Y-axis and / or Z-axis coordinates or reference points from a control space 186. Vertices generator 172 can capture the coordinates, or reference points, distances or locations from a medical environment 102 (e.g., using sensor measurements from data capture devices 110.

[0076] Triangulation function 170 can include or use vertices 174 indicative of locations or positions of various points (e.g., parts of manipulator components 122, data capture devices 110 or medical tools 112) with respect to a reference point. The reference point could be a point on a coordinate system, such as a coordinate system of the control space 186 or a reference point (e.g., location) within a medical environment 102, such as a corner of a room or a sensor location. Vertices 174 can include coordinate locations of one or more ends, parts, or components of manipulator components 122, medical tools 112 or data capture devices 110, allowing the triangulation function 170 to determine the orientation, location, position and / or movement of such devices or components.

[0077] Triangulation function 170 can include or use a performance function 176 to determine the performance of the tasks 148 or phases 144 using vertices 174. Performance can be indicated or defined using metrics 178 determined by the performance function 176. Performance function 176 can include any combination of hardware and software for determining performance of the task or phase being performed by the user (e.g., surgeon). Performance function 176 can include functionality for comparing vertices 174 of the manipulator components 122 or medical tools 112 (e.g., in a control space 186) with vertices 174 of other examples of the same tasks 148 or phrases 144. For example, a performance function 176 can compare the vertices 174 of the task 148 or phase 144 currently performed (e.g., as detected by the phase detectors 142 or task detectors 146) against the vertices 174 of the control space 186 of the detected tasks 148 or phases 144 to determine the level of performance of the current task 148 or phase 144. In some implementations, vertices 174 can be used to determine metrics 178 (e.g., triangulation) that can be compared against other triangulation (e.g., threshold) values.

[0078] Triangulation function 170 can determine the performance in the context of locations, orientations or positions of manipulator components 122, medical tools 112 and / or data capture devices 110, whether on the control space 186 in which the user controls these devices or in the medical environment 102 in which the surgery on the patient is provided.Triangulation function 170 can also determine instruments to be used for particular tasks 148 or phases 144 of a surgical procedure. For example, a particular triangulation 170 can be suitable for a particular surgical instrument rather than other instruments, based on the instrument’s geometry (e.g., size or shape) affecting the triangulation for the given instrument and the given task 148 or phase 144. Performance function 176 can determine, based on the vertices 174, that a range of motion or an available room for control or use of manipulator components 122, medical tools 112 and / or optical devices is outside of a desired range. For example, performance function 176 can determine that positioning of the user’s arms on the control space 186 is outside of ergonomically suitable position, field or range, limiting the surgeon’s movements and ability to control the devices. For example, performance function 176 can determine that a level or a scale of control to apply to manipulator components 122, or a particular medical tool 112 or a data capture device 110 is outside of the desired scaling range. For example, for improved performance a more granular ratio of pixels in the control space 186 to a distance (e.g., centimeters) of movement in the medical environment 102 space can be desired or preferred. For example, performance function 176 can determine that a field of view of a data capture device 110 (e.g., a camera) is outside of the field of view, outside of a zoom range than a desired field of view or zoom range, adversely affecting the performance of a surgical action.

[0079] Triangulation function 170 can operate in various modes. For example, in a preoperative mode (e.g., prior to a surgical procedure) triangulation function 170 can utilize any information about the phases 144 or tasks 148 to be performed to recommend best port placement to achieve best triangulation in a given phase 144 or task 148. For instance, triangulation function 170 can utilize information on prior performed procedures, including stored prior performances 154 and the corresponding thresholds 156 for those performances to recommend port placements, tools or various settings. In doing so, the triangulation function 170 can improve, expand or adjust the range of movement of the tools or devices, range of motion available to the surgeon during the operation, the field of view from optical devices (e.g., cameras or displays), or various ergonomic settings, such as positioning of arms or fingers on the interface to maximize the range of motion for controlling the devices (e.g., 122, 112 and / or 110).

[0080] Triangulation function 170 can operate in an intraoperative mode. In the intraoperative mode, the triangulation function 170 can provide indications 168 of thetriangulation (e.g., metrics 178 responsive to the vertices 174) in the real-time (e.g., during the ongoing operation). Indications 168 can be provided on a display (e.g., 116) in the form of overlaying indications 168 in a user interface 182 (e.g., at the side of the display), making the user aware of the current surgical conditions and performance.

[0081] Triangulation function 170 can operate in a postoperative mode. In the postoperative mode, the triangulation function 170 can provide procedure review, indicating a time-series triangulation measure along with any other objective performance indicators (e.g., metrics 178), along with any system events corresponding to the tasks 148 or phases 144 or surgery overall.

[0082] DPS 130 can include a component calibrator 162 for calibrating manipulator components 122, medical tools 112 and / or data capture devices 110. Calibration can include a process of aligning, configuring, and adjusting settings and parameters (e.g., location and scaling of movement) to allow for accurate and consistent, precise and reliable operation within a robotic medical system 120. DPS 130 can include and utilize a clutch function 180 to reset the vertices 174 of a device (e.g., manipulator component 122, data capture device 110 and / or medical tool 112). For instance, a foot pedal can be used by the surgeon to move both hand controls simultaneously, adjusting their positions without affecting the instruments. Each hand control's finger clutch can allow independent movement, adjusting one hand's position without affecting its corresponding instrument. Simultaneously using both finger clutches adjusts both hand positions without impacting the instruments. In one example, clutch function 180 can allow the user to reset (e.g., set to zero) vertices 174 of devices in a control space 186, allowing the user to change the ergonomics (e.g., location of hands, fingers, arms or any other body part) to allow for more accurate, efficient, precise and reliable operation, thereby improving performance of the surgical tasks 148 and / or phases 144.

[0083] DPS 130 can include an interface 182 designed, constructed and operational to communicate with one or more component of system 100 via network 101, including, for example, the robotic medical system 120 or another device, such as a client’s personal computer. The interface 182 can include a network interface. The interface 182 can include or provide a user interface, such as a graphical user interface. Interface 182 can provide data for presentation via a display, such as a display 116, and can depict, illustrate, render, present, or otherwise provide indications 168 generated by an indicator 164.

[0084] Indications 168 can include messages, indications or notifications relating the performance based on the metrics 178 and / or vertices 174. Indications 168 can be verbal or illustrative, can include images or graphs and can provide real-time feedback regarding the current level of performance. Indication 168 can include for example, illustrations, images, pictures, diagrams, graphical elements, charts or figures corresponding to patient side triangulation 310 and / or surgeon side triangulation 312. For example, indication 168 can include a patient side triangulation 310 including an image of a triangle whose shape (e.g., corner angles) are indicative of patient-side triangulation visibility 322, left-hand manipulator triangulation 326 and / or right-hand manipulator triangulation 324. For example, indication 168 can include a surgeon side triangulation 312 including an image of a triangle whose shape (e.g., corner angles) are indicative of surgeon-side visibility 332, left-hand surgeon positioning 336 and / or right-hand surgeon positioning 334.

[0085] Indication 168 generated by the indicator 164 functionality can include or reflect, with respect to surgeon side triangulation 312, performance with respect to ergonomic positioning of the user, limitations to the user’s range of movement or device control, positioning of controls or hands on the control space 186 that may limit the surgeon’s ability to control devices or any inefficiencies relating the positioning or orientation of the devices manipulated or controlled. Indication 168, with respect to the patient side triangulation 310, can include range or efficiency of available movement by the left and right manipulator components 122 and / or medical tools 112 at the patient side, as well as visibility of the surgical site 302 from the standpoint of the user (e.g., surgeon).

[0086] Indicator 164 can include the functionality to produce or generate indications 168 (e.g., via interface or GUI 182) to illustrate or indicate patient side triangulation 310 and surgeon side triangulation 312, as well as recommend specific corrective actions to improve the performance, such as performing a clutching maneuver via a clutch function 180 to reset the vertices 174. Indicator 164 can generate indications 168 to recommend recalibrating the devices (e.g.., 122, 112 and / or 110) using the component calibrator 162. Indications 168 can suggest to the user to adjust the control or positioning of devices to maintain or improve the effectiveness of a task 148 and / or phase 144 of the medical operation.

[0087] The data processing system 130 can interface with, communicate with, or otherwise receive or provide information with one or more component of system 100 via network 101, including, for example, the robotic medical system 120. The data processing system 130,robotic medical system 120 and devices in the medical environment 102 can each include at least one logic device such as a computing device having a processor to communicate via the network 101. The data processing system 130 or robotic medical system 120 can include at least one computation resource, server, processor or memory. For example, the data processing system 130 can include a plurality of computation resources or processors coupled with memory.

[0088] The data processing system 130 can be part of or include a cloud computing environment. The data processing system 130 can include multiple, logically grouped servers and facilitate distributed computing techniques. The logical group of servers may be referred to as a data center, server farm or a machine farm. The servers can also be geographically dispersed. A data center or machine farm may be administered as a single entity, or the machine farm can include a plurality of machine farms. The servers within each machine farm can be heterogeneous - one or more of the servers or machines can operate according to one or more type of operating system platform.

[0089] The data processing system 130, or components thereof can include a physical or virtual computer system operatively coupled, or associated with, the medical environment 102. In some embodiments, the data processing system 130, or components thereof can be coupled, or associated with, the medical environment 102 via a network 101, either directly or directly through an intermediate computing device or system. The network 101 can be any type or form of network. The geographical scope of the network can vary widely and can include a body area network (BAN), a personal area network (PAN), a local-area network (LAN) (e.g., Intranet), a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The topology of the network 101 can assume any form such as point-to-point, bus, star, ring, mesh, tree, etc. The network 101 can utilize different techniques and layers or stacks of protocols, including, for example, the Ethernet protocol, the internet protocol suite (TCP / IP), the ATM (Asynchronous Transfer Mode) technique, the SONET (Synchronous Optical Networking) protocol, the SDH (Synchronous Digital Hierarchy) protocol, etc. The TCP / IP internet protocol suite can include application layer, transport layer, internet layer (including, e.g., IPv6), or the link layer. The network 101 can be a type of a broadcast network, a telecommunications network, a data communication network, a computer network, a Bluetooth network, or other types of wired and wireless networks.

[0090] The data processing system 130, or components thereof, can be located at least partially at the location of the surgical facility associated with the medical environment 102 or remotely therefrom. Elements of the data processing system 130, or components thereof can be accessible via portable devices such as laptops, mobile devices, wearable smart devices, etc. The data processing system 130, or components thereof, can include other or additional elements that can be considered desirable to have in performing the functions described herein. The data processing system 130, or components thereof, can include, or be associated with, one or more components or functionality of a computing including, for example, one or more processors coupled with memory that can store instructions, data or commands for implementing the functionalities of the DPS 130 discussed herein.

[0091] System 100 can include one or more processors (e.g., 1110) coupled with memory (e.g., 1115) that can include instructions for implementing the functionalities of the system 100. For example, one or more processors 1110 can be configured by instructions, data or computer code stored in memory 1115 to identify at least one of a type of task 148 or a type of phase 144 of a medical procedure. The type of a task 148 can include any medical or surgical task, such as making an incision or a dissection, performing hemostasis, suturing a wound, removing a tissue, performing tissue repair, performing a biopsy or surgical imaging. The type of phases 144 can include any phase of a medical procedure or a surgery, such as a preoperative phase in preparation for a surgery or intraoperative phase involving various surgical tasks 148. Phase 144 can include or involve any number of surgical tasks 148.

[0092] One or more processors 1110 can be configured to determine vertices 174 of a plurality of manipulator components 122 and data capture devices 110 used to perform surgical tasks 148 and / or phases 144. For instance, one or more processors 1110 can be configured to determine vertices 174 associated with a first manipulator component 122 (e.g., a right-hand robotic arm for handling medical tools 112 or instruments) and a second manipulator component (e.g., a left-hand robotic arm for handing medical tools 112 or instruments). One or more processors 1110 can be configured do determine vertices 174 of a data capture device 110 (e.g., an optical device, such as a camera used to monitor the movement of the manipulator components 122 and medical tools at a surgical site 302). The first manipulator component 122, the second manipulator component 122 and the optical device (e.g., 110) are used to perform the at least one of the type of task 148 or the type of phase 144 of the medical procedure. For example, the one or more processors 1110 can utilize the triangulation function 170 todetermine a type of a task 148 associated a desired task performance 154 and thresholds 156 for the given task 148.

[0093] One or more processors 1110 can be configured to generate, via a triangulation function 170, a metric 178 based on the vertices 174. The metric 178 generated can be based on the current vertices 174 for the manipulator components 122 and / or data capture devices 110. The metric 178 can include a value indicative of the positioning, orientation, alignment or symmetry of components or devices. For instance, a triangulation function 170 can utilize the vertices 174 (e.g., coordinates) indicating the current locations, orientations or positioning of the manipulator components 122 and / or data capture devices 110 to compare them against corresponding vertices 174 of the manipulator components 122 and / or data capture devices 110 of the desirable or acceptable levels of performance 154. Based on the thresholds 156 for the vertices 174 corresponding to the desired performances 154 for the given tasks 148 or phases 144, triangulation function 170 can determine metrics 178 indicative of the current level of performance of the system 100.

[0094] One or more processors 1110 can be configured to provide a visual indication 168 of a performance of the medical procedure based on a comparison of the metric 178 with a threshold 146 established for the type of task 148 or the type of phase 144. For example, types of tasks 148 and / or phases 144 can include thresholds 156 for an acceptable range or threshold of metrics 178 for a given task 148 or phase 144. Triangulation function 170 can determine a level of performance by comparing one or more metrics 178 of the current state of vertices 174 of the manipulator components 122, medical tools 112 and / or data capture devices (e.g., optical devices) against thresholds 156 of the corresponding task or phase performances 154.

[0095] For example, a first manipulator component 122 and the second manipulator component 122 can be communicatively coupled with a robotic medical system 120. The robotic medical system 120 can include any number of manipulator components 122 configured or designed for handling or manipulating any number of medical tools 112. The robotic medical system 120 can be configured to perform at least a portion of the medical procedure, including a portion of a medical task 148 or phase 144. One or more processors 1110 can be configured to identify a time-series of the vertices 174 that can be associated with the first manipulator component 122, the second manipulator component 122, and the optical device (e.g., 110). For example, time-series of vertices 174 can include measurements or values of coordinates, or identifiers of positioning or location of the components or devices arrangedin accordance with time stamps or distributed over a period of time. For example, a time-series of vertices 174 can include measurements of vertices 174 over a time interval, where measurements are taken at various periods within the time interval, thereby indicating a path of movement (e.g., positions or orientations) of the components or devices. The triangulation function 170 can generate a time-series of one or more metrics 178 using the time-series of the vertices 174.

[0096] One or more processors 1110 can be configured to provide, on a graphical user interface 182 that displays the medical procedure (e.g., via a display view 204), a real-time visual indication 168 of the time-series of the metric 178. For example, the visual indication 168 can include an illustration, animation or image of a triangle, rectangle or other visual feature indicative of the present state of surgeon side triangulation or patient side triangulation.

[0097] One or more processors 1110 can be configured to calibrate, using a component calibrator 162, the first manipulator component 122 and the second manipulator component 122 prior to detection of the vertices 174. In some instances, calibration can be implemented during the procedure or with respect to adjusted, replaced or used medical tools 112 and / or manipulator components 122.

[0098] One or more processors 1110 can be configured to select one or more thresholds 156 based on an anatomical structure associated with the medical procedure. For example, a phase detector 142 can determine or detect a phase based on anatomical feature of a patient in a display view 204. For example, a task detector 146 can determine a task 148 based on the anatomy identified in the display view 204. Thresholds 156 associated with the desired performances 154 for the given task 148 or phase 144 can be selected based on the anatomy identified. For example, a processor 1110 can be configured to select the threshold 156 based on a type of the first manipulator component 122 or a type of the second manipulator component 122.

[0099] One or more processors 1110 can be configured to detect, subsequent to provision of the visual indication 168 of the performance, a clutch operation (e.g., via a clutch function 180) for at least one of the first manipulator component 122, the second manipulator component 122, or the optical device (e.g., 110). For instance, in response to a presented or displayed indication 168 identifying or indicating the level of patient side triangulation 310 or surgeon side triangulation 312, the surgeon can perform a clutching action to reset, adjust or improve the positioning, location, orientation, symmetry or visibility of the component ordevice. One or more processors 1110 can be configured to generate second vertices 174 subsequent to detection of the clutch operation and provide a second visual indication 168 of a second performance of the medical procedure based on the second vertices 174. The second performance indicated by the indication 168 can be greater than the performance prior to the clutching action.

[0100] For example, one or more processors 1110 can be configured to identify a swap from the first manipulator component 122 to a third manipulator component 122. The third manipulator component can be used to perform the at least one of the type of task 148 or the type of phase 144 of the medical procedure. One or more processors 1110 can be configured to determine second vertices 174 associated with the third manipulator component 122, the second manipulator component 122 and the optical device (e.g., 110). One or more processors 1110 can be configured to generate a second metric 178 based on the second vertices and provide a second visual indication 168 of a second performance of the medical procedure based on the comparison of the second metric 178 with one of the threshold 156 or the metric 178.

[0101] One or more processors 1110 can be configured to determine, based on the metric 178 less than or equal to the threshold 156, to provide guidance to improve the performance of the medical procedure. For example, the guidance can include a prompt, a suggestion or a recommendation to the surgeon to perform a clutching action via a clutching function 180 or a calibration via a component calibrator 162. The one or more processors 1110 can provide, for display on a display device (e.g., 1130), the guidance to improve the performance of the medical procedure. The guidance can include a statement to perform a clutch operation on at least one of the first manipulator component 122, the second manipulator component 122 or the optical device.

[0102] The visual indication 168 of the performance can include least one of a color, a numerical score, a grade, or a binary value. One or more processors 1110 can be configured to provide at least one of a haptic indicator or an audio indicator based on the performance of the medical procedure. The first manipulator component, the second manipulator component, and the optical device can be coupled with a console. For example, the console can be configured to control one or more instruments (e.g., medical tools 112) used to perform the medical procedure. The optical device can include a camera or a headset, such as a heads up display (HUD).

[0103] FIG. 2 illustrates a graph 200 comparing a performance between a medical procedure in which triangulation adjustment is applied against a medical procedure in which no triangulation adjustment is applied. Graph 200 can include a procedure time or duration 206 on one axis, for example the X-axis (e.g., horizontal axis), and performance 208 on another axis, for example the Y-axis (e.g., vertical axis). Procedure duration 206 can correspond to a time duration of a medical procedure, which can include any number of phases 144 and tasks 148. Performance 208 can correspond to performance metrics 178 measured throughout the procedure.

[0104] Second procedure 216, marked with a solid line, can indicate or correspond to a procedure in which no triangular performance corrective actions are taken. As a result, the second procedure 216 continues to decrease in performance 208 as procedure duration 206 progresses. In contrast, the first procedure 202, marked with a dashed line, can indicate the performance 208 over the procedure duration 206 in which several performance corrective actions were taken to improve the performance 208. Each of the actions (e.g., 210, 212 and 214) can be implemented by the surgeon in response to an indication 168, issued following a performance function 176 providing metrics 178 in response to the vertices 174 of the devices controlled.

[0105] The first corrective action can include adding a manipulator component action 210, following which performance can increase (e.g., graph moves upward). The second action can include a camera clutch action 212, in which the user can use the clutch function 180 to reset the coordinates or location of the data capture device 110 (e.g., camera). As a result, hand position can change from the center and performance can slightly decrease. The third corrective action can be a finger clutch action 214, in which the user (e.g., surgeon) can utilize a clutch function 180 to reset the vertices 174 (e.g., coordinates or location) of the hands at the control space 186. The user can adjust hand position to be ergonomically improved and the performance can increase. Each of the actions 210, 212 or 214 can be implemented responsive to indications 168 suggesting these actions to the user and prompting the user to take the suggested action.

[0106] FIG. 3 illustrates an example system 300 providing indications for patient and surgeon side triangulations to improve the performance on the robotic medical system (RMS) 120. The robotic medical system 120 can include a plurality of manipulator components 122(e.g., four manipulator arms) that can be used to hold, control, maneuver or utilize medical tools 112 or instruments in the course of a surgery on a surgical site 302.

[0107] In example system 300, display view 204 includes a field of view from a data capture device 110 (e.g., camera) showing a surgical site 302 that includes an incision or a cut of a patient to be sutured (e.g., stitched). A first of manipulator component 122A (e.g., a first manipulator arm of the robotic medical system 120) can hold and utilize a medical tool 112A (e.g., a needle), while a second manipulator component 122B (e.g., a second manipulator arm of the RMS 120) can hold and utilize a thread (e.g., medical tool 112B) in order to complete stitching or suturing of a patient’s wound. Display view 204 can be positioned to observe the surgical site 302 along with the medical tools 112A and 112B (e.g., the needle and the thread) and parts of the manipulator components 122 A and 122B performing the task 148.

[0108] Display view 204 can include one or more indications 168 including illustrations, images, pictures, diagrams, graphical elements, charts or figures indicative of the triangulation performance based on the metrics 178 provided by the triangulation function 170. For example, an indication 168 can include an image, illustration or a sign of a patient side triangulation 310, a surgeon side triangulation 312 or a combination of the patient side triangulation 310 and surgeon side triangulation 312. Indication 168 can include an illustration of a patient side or a surgeon side triangle, having sides and angles (e.g., shape) indicative of the triangulation score or quality.

[0109] Indication 168 of the patient side triangulation 310, which can also be referred to as a patient side triangle 310, can include a triangle whose shape (e.g., angles of the three corners and / or triangle side lengths) can indicate the state of the triangulation performance. For example, a first corner (e.g., a top comer) of the patient side triangulation 310 can corresponds to a patient side visibility 322, such as a visibility of the target object (e.g., surgical site 302) and ability to navigate the operation at the patient-side (e.g., medical environment 102). The second comer (e.g., left hand side comer) can be indicative of the left-side manipulator triangulation 326 (e.g., corresponding to the manipulator component 122B) and can be indicative of alignment and symmetry of manipulator component 122B with respect to the target and / or the range of available motion of manipulator component 122B with respect to the target site (e.g., the location of the incision to be stitched), given the positioning and orientation. The third corner (e.g., right hand side comer) can be indicative of the right-side manipulator triangulation 324 (e.g., with respect to the manipulator component 122A),indicative of its alignment and symmetry with respect to the target and / or its range of available motion with respect to the target site (e.g., the location of the incision to be stitched), given its positioning and orientation. The shape of the patient side triangle 310 can include any type of triangle shape (e.g., right triangle, obtuse triangle, acute triangle or any other) as defined by the angles at corners 322, 324 and 326 and / or by the lengths of triangle sides. For example, a patient side triangle 310 whose side has a length between a comer 322 and a corner 326 that is the same as a side length between the comer 322 and a comer 324 (e.g., the corner 322 forming a 90 degree angle at the point 322 and 45 degree angles at the 324 and 326) can be indicative of a triangulation performance that is maximized (e.g., having a perfect triangulation performance for the patient side triangle 310). Likewise, a triangle having angles that are offset from such a setup, can indicate a sub-optimal or diminished performance in terms of the triangulation at the patient side triangle 310.

[0110] Indication 168 of the surgeon side triangulation 312, which can also be referred to as a surgeon side triangle 312, can include a triangle whose shape (e.g., angles of the three corners and / or triangle side lengths) can indicate the state of the triangulation performance for the surgeon side triangle 312. For example, a first corner (e.g., a bottom corner) of the surgeon side triangle 312 can corresponds to a surgeon side visibility 332, such as a visibility of the target object (e.g., surgical site 302), surgeon’s ergonomics with respect to the direction to look at (e.g., head or body angles) and display performance. The second comer (e.g., left hand side corner) can be indicative of the left hand triangulation 336 of the surgeon controlling the manipulator component 122B. Left hand triangulation 336 can be indicative of the ergonomics of the surgeon, alignment and symmetry of the surgeon’s hand, finger or arm with respect to the manipulator component 122B, hand to eye coordination and alignment, and the teleoperation efficiency while controlling the manipulator component 122B. The third comer (e.g., right hand side corner) can be indicative of the right hand triangulation 334 while controlling the manipulator component triangulation 324 (e.g., with respect to the manipulator component 122A). Right hand triangulation 334 can be indicative of the ergonomics of the surgeon, alignment and symmetry of the surgeon’s hand, finger or arm with respect to the manipulator component 122 A, hand to eye coordination and alignment, and the teleoperation efficiency while controlling the manipulator component 122 A. The shape of the surgeon side triangle 312 can include any type of triangle shape (e.g., right triangle, obtuse triangle, acute triangle or any other) as defined by the angles at corners 332, 334 and 336 and / or by the lengths of triangle sides. For example, a surgeon side triangle 312 whose side has a length between acorner 332 and a corner 336 that is the same as a side length between the corner 332 and a corner 334 (e.g., the corner 332 forming a 90 degree angle at the point 322 and 45 degree angles at the 334 and 336) can be indicative of a triangulation performance that is maximized (e.g., having a perfect triangulation performance for the patient side triangle 312). Likewise, a triangle having angles that are offset from such a setup, can indicate a sub-optimal or diminished performance in terms of the triangulation at the surgeon side triangle 312.

[0111] FIG. 4 illustrates an example system 400 for determining and adjusting the surgeon side triangulation 312. Example system 400 can use various offset angles and positions of the surgeon’s body parts to determine the surgeon side visibility 332 of the surgeon side triangle 312, hand triangulations 336 and 334 affect the control of movement of medical tools 112, as well as hand-eye coordination and bimanual dexterity in the medical environment 102. Example system 400 can include a surgeon side triangulation 312 having a left hand triangulation 336, a right hand triangulation 334 and surgeon side visibility triangulation 332. The surgeon side visibility 332 can be defined, adjusted, modified or established based on, or using surgeon side visibility components 402.

[0112] Surgeon side visibility components 402 can include a body position 410, a head position 412, an eye position 414, a display position 416 and an image transformation 418. There can be a series of offset angles or tilts that can be added to define the surgeon side visibility 332. For example, there can be a first offset angle between the body position 410 and the head position 412, such as due to a tilted head with respect to the user’s body. For example, there can be a second offset angle between the head position 412 and the eye position 414, such as due to an angle of the user’s eye direction or view with respect to the user’s head. For example, there can be a third offset angle between the eye position 414 and display position 416, and there can be a fourth offset angle between the display position 416 and the image transformation 418. These offset angles or titles can be added or combined to define, sum up or establish a surgeon side visibility 332. The total offset from the various offset angles can be determined using a function, such as: E (A) =where w corresponds to weighted values and A corresponds to individual offset angles.

[0113] Control space 186 can include points, coordinates or vertices that correspond to the surgeon’s left hand and right hand positions or their corresponding triangulations. For example, a control space 186 can include a point or location corresponding to left hand triangulation 336 and right hand triangulation 334 values. In the display view 204, a medical tool 112 (e.g.,thread) corresponding to the left hand of the surgeon (e.g., controlling a left-side manipulator component 122) can be affected by the left hand triangulation 336 at the control space 186. Likewise, a medical tool 112 (e.g., needle) corresponding to the right hand of the surgeon (e.g., controlling a right-side manipulator component 122) at the display view 204 can be affected by the right hand triangulation 334 at the control space 186.

[0114] FIG. 5A illustrates a series of example presentations 500 of patient side triangulations 310 and the corresponding display views 204 of tasks performed. In example view 502, display view 204A can depict or show a pair of medical tools 112 (e.g., handled by a corresponding pair of manipulator components 122) in a particular orientation that is suitable for efficient use. Example view 502 shows a patient side triangulation 310A or patient side triangle having corners 322, 324 and 326 of the patient side triangle 310A. Corners 326 and 324 are equally spaced from the comer 322 which forms a right (e.g., 90 degree) angle, resulting the corners 324 and 326 being at around 45 degrees and producing a triangulation performance metrics 178 value of T=l, indicative of a maximized triangulation performance for example view 502.

[0115] In example view 504, display view 204B can depict or show the same pair of medical tools 112 as in 204A (e.g., handled by the same corresponding pair of manipulator components 122) in which the target can be movable, and the visibility or accessibility of a surgical site can be reduced. Patient side triangulation 310B corresponding to the display view 204B includes corners 322, 324 and 326 of the patient side triangle 310B where the corner 322 is an obtuse angle, making angles 326 and 324 more acute than 45 degree angles at view 502. As a result, the triangulation performance metrics 178 for example view 504 is a value T=0.7, indicative of a sub-optimal triangulation performance due to reduced visibility at the patient side visibility 322.

[0116] In example view 506, display view 204C can depict or show the same pair of medical tools 112 as in 204A and 204B (e.g., handled by the same corresponding pair of manipulator components 122) in which the visibility or accessibility of a surgical site is even further reduced than in example 504. Patient side triangulation 310C corresponding to the display view 204C includes comers 322, 324 and 326 of the patient side triangle 310C where the comer 322 is approaching an obtuse angle of almost 180 degrees, making angles 326 and 324 very acute (e.g., less than 10 degrees each), indicating of a serious deficiency of patient side visibility or accessibility. As a result, the performance metrics 178 for example view 506 isa value T=E, indicative of a triangulation performance that is below an acceptable threshold (e.g., no effective patient side visibility 322).

[0117] In example view 508, display view 204D can depict or show the same pair of medical tools 112 (e.g., handled by the same corresponding pair of manipulator components 122) in which the ergonomics, positioning or orientation of the manipulator components 122 or their corresponding medical tools 112 is ineffective, hindering the triangulation performance. Patient side triangulation 310D corresponding to the display view 204D includes corners 322, 324 and 326 of the patient side triangle 310D where the corners 326 and 324 are offset, reduced or limited, with an imperfect patient side visibility 322. Patient side triangulation 310D can correspond to a reduced patient side visibility 322 due to the camera or display being off- centered, oriented to an angle, flipped, or the point of interest (e.g., surgical site) is obscured or hidden). Patient side triangulation 310D can include a limited left-side manipulator triangulation 326 and a limited right-side manipulator triangulation, indicative of inefficient orientation of the tools which may reduce the available range of movement or accessibility. For instance, the performance metrics 178 for example view 508 is a value T = 0.1, indicative of a poor triangulation for each of the corners 322, 324 and 326. In such instances, the technical solutions can send an indication 168 with a prompt to perform a clutching action or recalibration.

[0118] FIG. 5B illustrates example presentations 510 with views 512 and 514 in which a third manipulator component is used in the context of triangulation. For example, a third arm instrument can be used in a robotic medical system for implementing various part of robotic surgeries. The third arm instrument can be activated and deactivated by tapping the arm swap pedal or clutch. As shown in views 512 and 514, the left hand control can be swapped from the lower left instrument (e.g., manipulator component 122B active in view 512) to the upper left instrument (e.g., manipulator component 122C active in view 514). In doing so, the technical solutions can deactivate the lower left instrument (e.g., 122B) and activate the upper left instrument (e.g., 122C) to then continue to be used with the right-side instrument (e.g., 122A) on further surgical tasks.

[0119] Before swapping of the component 122B with component 122C, as shown in view 512, patient side triangulation 310E can be evaluated at Tt=0.9, which can be sufficient or suitable for bimanual tasks with the active lower instrument 122B and active right instrument 122A in view 512. In some implementations, the inactive third arm instrument in the upper leftcorner of views 512 and 514 (e.g., 122C) can limit the workspace of the lower left instrument (e.g., 122B), which can adversely affect the triangulation of the lower left instrument.

[0120] After swapping, the lower left instrument 122B can become inactive, while the upper left instrument 122C can become active. The third instrument in this example is not located in a suitable location for a bimanual manipulation task 148 of this kind (as each task 148 can have its own parameters for the triangulation), and its workspace can be limited by the lower left instrument 122B. This can result in a change in the patient side triangulation (from 0.9 in view 512 for patient side triangulation 310E to 0.6 in view 514 for patient side triangulation 310F). In a worse case, if the upper left instrument was out of the view, then sudden activation and movement might damage tissue or organ that are not visible. For example, there can be different strategies to manage a patient side triangulation during the arm swap, depending on the task or the current instrument positions. For example, a user can leave the lower left instrument in a place that restricts the upper left instrument, maneuver camera if necessary to visualize the upper left instrument inside the view, and then activate the upper left instrument.

[0121] Further referring to FIGs. 5 A and 5B, as shown in views 502-514, there can be a variety of instruments for different types of tasks, phases, and cases. For instance, selecting a most suitable instrument for a particular task can improve the surgical performance. The triangulation function can be extended to quantify the suitable choice of instrument, as well as different types or configurations of triangulations for different instruments. For example, a triangulation function can include the left and right corner vertices to determine desirable instrument installation (e.g., for a given task 148, phase 144 or a medical procedure), as it can include the vertices 174 for the particular task 148, phase 144 or instrument 112 that can be used to determine the desirable instrument choice, as well as the instrument’s desirable position and orientation (e.g., triangulation).

[0122] For example, when instrument is used in bimanual surgical task, then the triangulation metric can have a low value and performance can be poor. For example, when two scissors are used in suture knot tying task (needle driver and grasper can be preferred), then the result may not be preferred or acceptable and the performance may be reduced. For example, an advanced energy instruments, such as vessel sealers and staplers, can be used to effectively control bleeding in cutting tissue, then the patient outcome would be significantly improved. For example, optimal pair of instruments (such as a large needle driver or aMaryland bipolar grasper, and monopolar cut scissors for dissection) for a particular task and phase, can improve the triangulation metric and improve the surgical performance. For instance, different instruments can have different triangulation values and parameters, therefore allowing for a different range of motion and different levels of ergonomic positioning. The triangulation improvement by suitable selection of instrument can be determined using information on prior performed procedures, including stored prior performances 154. The corresponding thresholds 156 can also be determined in a similar way.

[0123] FIG. 6A illustrates examples diagrams 600 of relations between left and right hand surgeon positionings 334 and 336 in control space 186 and manipulator components 122 in a display view 204 during a task performed at a surgical site 302. The series of examples 600 can include a first example 602 in which a user can perform clutching (e.g., using a clutch function 180) to adjust a triangulation of the manipulator components to adjust the locations or orientations of the manipulator components 122 with respect to the surgical site 302, a second example 604 in which the user can perform clutching of the optical device with respect to the surgical site 302, and a third example 606 in which the user can perform clutching to both readjust the triangulation of the optical device and of the manipulator components.

[0124] At example 602, a user can utilize left hand surgeon positioning 336 and right hand surgeon positioning 334 in the control space 186 to control a left-hand manipulator component 122 and a right-hand manipulator component 122 within a field of view 204 and near a surgical site 302. The left hand surgeon positioning 336 and right hand surgeon positioning 334 can be located or positioned a distance apart below or away from the surgical site 302. For example, the location of the surgical site 302 can be used as a reference point for the coordinates or positional references for any of the manipulator components 122, medical tools 112 and / or left hand surgeon positioning 336 and / or right hand surgeon positioning 334 (e.g., in either the control space 186 or field of view 204). As shown in illustrations 602(a), 602(b), 602(c) and 602(d), to adjust the triangulation during the surgery, the user can reach with the manipulator components 122 to get to the surgical site 302 and perform a clutching action via the clutching function 180 to reset the vertices (e.g., locations or coordinates) of the manipulator components 122 with respect to the location of the surgical site 302 (e.g., at 602(b) or 602(c)). The clutching action can be implemented on the manipulator components 122 or on the optical device (e.g., camera) in order to reset the vertices of either the camera or the manipulator components. As a result, at 602(d) the manipulator components 122 can now have improvedtriangulation (e.g., improved ergonomics and range of movement) with respect to the surgical site 302. This can allow the surgeon to now have an improved or increased range of motion, effectiveness of movement, ergonomics of the surgeon, visibility of the surgical site 302, thereby resulting in improved performance and control during the surgery.

[0125] At example 604, as shown in illustrations 604(a), 604(b), 604(c), 604(d) and 604(e), a user can maintain usage of the left hand surgeon positioning 336 and right hand surgeon positioning 334 in the control space 186 to control the left-hand manipulator component 122 and the right-hand manipulator component 122 within the field of view 204 and near the surgical site 302 (e.g., as in example 602). The left hand surgeon positioning 336 and right hand surgeon positioning 334 can be located or positioned a distance apart below or away from the surgical site 302. The user can then reach closer to the surgical site 302 with the manipulator components 122 and perform one or more clutching actions via the clutching function 180 to reset the vertices (e.g., locations or coordinates) of the manipulator components 122 and / or optical device with respect to the location of the surgical site 302. The clutching actions can be implemented to reset vertices of to the manipulator components 122 and / or medical tools 112 or of the optical device (e.g., camera or any other data capture device 110). As a result, the manipulator components 122 and / or the optical device (e.g., 110) can have improved triangulation (e.g., improved ergonomics, field of view and range of movement) with respect to the surgical site 302, resulting in the improved visibility and / or increased range of motion, effectiveness or ease of movement, ergonomics or overall performance and control during the surgery.

[0126] At example 606, as shown in illustrations 606(a), 606(b), 606(c) and 606(d), in the example with similar left hand surgeon positioning 336 and right hand surgeon positioning 334 in the control space 186 controlling the manipulator components 122 within the field of view 204 and near the surgical site 302 (e.g., as in the examples 602 and 604), the surgeon can perform a combination of clutching of the optical device and clutching the manipulator components. For example, the surgeon can reach with the manipulator components 122 to the surgical site 302 and perform a clutching action with respect to the optical device and then another clutching action with respect to the manipulator components, as a result of which the triangulation can be improved, resulting in the improved visibility and / or increased range of motion or movement, effectiveness or ease of movement, ergonomics or overall performance and control during the surgery.

[0127] FIG. 6A can represent or demonstrate how a user (e.g., a surgeon) can manage both surgeon and patient triangulations in a camera targeting task. The task can be to visualize the target centered in a screen. The first states shown as 602(a), 604(a) and 606(a) can be the same, as well as the last states 602(d), 604(e) and 606(d). The differences can be in how to maneuver camera using clutching and camera pedals.

[0128] FIG. 6B shows an example 610 graph of plots of triangulation metrics for surgeon side triangulation (Ts) 312 and patient side triangulation (Tp) 310 corresponding to example 602 of FIG. 6 A in which manipulation components are clutched to improve triangulation. At initial state, shown as 602(a), two arms 336 and 334 of 602(a) are in the ergonomically suitable position in the control space 186. Two instruments shown as 122 within display 204 can be positioned in an offset (e.g., downward) from the center, whereas the preferred position is closer to the center (e.g., upward). At 602(b), fully stretch the arms can move the instruments close to the target. At 602(c), the user can press the camera pedal, pull the arms to move the camera upward to center the target on the screen and then release the camera pedal. At 602(d), the user can press apply the clutch to pull the arms to be in the ergonomically suitable position. In the illustrated example, the surgeon side triangulation metric is zero in 602(b) because the two arms are fully stretched in a position that is ergonomically undesirable, and the hands reaches the boundary of the control space. The surgeon side triangulation metric can capture this and provide an alarm or indication to correct the positions.

[0129] FIG. 6C shows an example 620 graph of plots of triangulation metrics for surgeon side triangulation (Ts) 312 and patient side triangulation (Tp) 310 corresponding to example 604 of FIG. 6 A in which manipulation components are clutched to improve triangulation. At initial state, shown as 604(a), two arms 336 and 334 of 602(a) are in the ergonomically suitable position in the control space 186 and two instruments shown as 122 are in an offset (e.g., downward) from the center, whereas the preferred position is closer to the center (e.g., upward). At 604(b), the user can push the hand control but not fully stretch the arms to move the instrument to the target but not too close to the center (e.g., a portion of the way from its initial position to the center). At 604(c), the user can press the camera pedal / clutch, pull the arms to move the camera upward to center the target on the screen and then release the camera pedal. At 604(d), the user can push the hand controls to move the instruments close to the target. At 604(e), the user can apply the clutch to pull the arms to be in the ergonomically suitable position. In the illustrated example, both surgeon and patient side triangulation metrics can be well managed without dropping to lower values unlike in 602 and 606 examples. Thiscan occur because the hand motions and instrument motions may not be large at a time. Instead, the motions can be split into one or more smaller motions to manage the triangulation metrics above the thresholds. This can result in keeping the hands in improved ergonomic positions for bimanual manipulation tasks and keep the instrument in the safe operational space and close to the target for better manipulation.

[0130] FIG. 6D shows an example 630 graph of plots of triangulation metrics for surgeon side triangulation (Ts) 312 and patient side triangulation (Tp) 310 corresponding to example 606 of FIG. 6 A in which manipulation components are clutched to improve triangulation. At initial state, shown as 606(a), two arms 336 and 334 of 602(a) are in the ergonomically suitable position in the control space 186 and two instruments shown as 122 are in an offset (e.g., downward) from the center, while the preferred position is closer to the center (e.g., upward). At 606(b), the user can press the camera pedal, pull the arms to move the camera upward to center the target on the screen and then release the camera pedal. At 606(c), the user can push the hand controls to move the instruments closer (e.g., in the vicinity of or adjacent) to the target. At 606(d), the user can apply the clutch to pull the arms to be in the ergonomically suitable position. In the illustrated example, the patient side triangulation metric can be zero in 606(b) as the two instruments are out of view. The patient side triangulation metric can capture this and provide an alarm or indication to correct the positions.

[0131] FIG. 7 illustrates an example selection 700 of types of indications 168 that can be displayed to a user. Example selection 700 can include indications 168, such as those illustrated in example views 702, 704, 706 or 708. Indications 168 can be displayed in a display view 204 showing a pair of manipulator components 122 that utilize or manipulate a pair of medical tools 112, while performing a task 148 at a surgical site 302. A first manipulator arm (e.g., manipulator component 122 A) can hold a needle (e.g., a medical tool 112 A) for stitching a wound or a cut, a second manipulator arm (e.g., manipulator component 122B) can hold a thread attached to the needle. Display view 204 can show or present a visual indication 168 including an image, an illustration, an animation, a graph, an alphanumeric statement or an icon that can indicate the state, level or characteristics of the patient side triangulation 310 and / or surgeon side triangulation 312.

[0132] As shown in example view 702, indication 168 of the display view 204 can include a line or a rectangular shaped icon whose height indicates the current level of triangulation,such as a surgeon side triangulation 312 or patient side triangulation 310. Shown in example view 704, indication 168 can include a triangle which can be shaped to have the angles (e.g., shape of the triangle) indicate the surgeon side triangulation 312 or patient side triangulation 310. For example, a top or bottom comer of the triangle can correspond to the surgeon side visibility 332 or patient side visibility 322, and the comers on the left and the right can correspond to the left-side or right-side manipulators or surgeon’s hand, mousejoystick or finger locations. As shown in example view 706, indication 168 of the display view 204 can include a line or a rectangular shaped icon, similar to that of view 702, but include indications of the current level of triangulation with respect to both the surgeon side triangulation 312 and patient side triangulation 310. For instance, one part (e.g., up to a half) of the rectangle can correspond to the surgeon side triangulation 312 and another part (e.g., up to the other half) can correspond to the patient side triangulation 310. Shown in example view 708, indication 168 can include two triangles. One triangle can correspond to the surgeon side triangulation 312 and the remaining triangle can correspond to the patient side triangulation 310. The shapes of the two triangles can be indicative of the triangulation of each of the manipulator components 122 and / or data capture devices (e.g., visibilities) for either of the triangles. Using such indications 168 displayed in the display view 204, the surgeon can be continuously aware of the level of the triangulation performance during the course of the procedure.

[0133] FIG. 8 illustrates an example view of a simulation training 800 that can be used for providing practice or training on triangulation to the surgeons. Example simulation training 800 can be implemented in a postoperative phase in which surgeon can simulate various phases 144 or tasks 148 of a surgery to improve the triangulation performance. Example simulation training 800 can include an example view 802 providing a surgeon side triangulation 312 visual feedback to the user, such as in the form of an indication 168. At example view 804, simulation training 800 can provide a three-dimensional animation, simulation or illustration of a space showing locations of the user’s hands or controls, such as, for example in a control space 186. At example view 806, simulation training 800 can include illustrated or simulated objects 810 that can be presented for the purpose of an exercise in which the user can move the manipulator components 122 A and 122B to manipulate the illustrated or simulated objects 810 with respect to one or more target locations. Such actions can be implemented, for example, during the course of triangulation user training or adjusting vertices or positioning of the manipulator arms with respect to reference points (e.g., clutching).

[0134] Turning now to FIG. 9, an example flowchart outlining the operations of a method 900 for implementing triangulation is illustrated. The method 900 can be performed by a system having one or more processors executing computer-readable instructions stored on a memory. The method 900 can be performed, for example, by system 100 and in accordance with any features or techniques discussed in connection with FIGS. 1-8 and 10-11. For instance, the method 900 can be implemented one or more processors 1110 of a computing system 1100 executing non-transitory computer-readable instructions stored on a memory (e.g., the memory 1115) and using data from a data repository 150 (e.g., storage device 1125).

[0135] The method 900 can be used to monitor and adjust patient side and surgeon side triangulation in robotic surgeries. Thus, at operation 905, a system can identify a task or a phase of a procedure. At operation 910, the system can detect vertices of components or devices used to implement the task or the phase. At operation 915, the system can determine a triangulation metric. At operation 920, the system can determine if the metric exceeds a threshold. At 925, if the metric exceeds the threshold, a visual indication can be updated. At 930, if the metric does not exceed the threshold, the system can provide the indication to correct triangulation. At 935, the system can perform corrective action.

[0136] At 905, a system can identify a task or a phase of a procedure. The method one or more processors coupled with memory utilizing a phase detector function to identify the type of a phase or a task detector function to identify the type of the task. For instance, the method can include a task detector identifying or detecting a type of a task or the task being performed. For example, a task detector can utilize a ML model trained to recognize or detect the type of task, or the task using sensor data from the data capture devices. For instance, the task detector can use the ML model to detect or identify the type of a task or the task itself based on one or more images or video fragments of the task being performed using a robotic medical system input into the ML model.

[0137] The method can include a phase detector detecting or identifying a type of a phase or the phase being performed. For example, a phase detector can utilize a ML model trained to recognize or detect the type of phase, or the phase itself, using sensor data from the data capture devices. For instance, the phase detector can use the ML model to detect or identify the type of a phase or the phase based on one or more images or video fragments of the phase being performed using a robotic medical system input into the ML model.

[0138] The method can include the one or more processors identifying one or more manipulator components and one or more data capture devices used to implement a task or a phase of a procedure (e.g., surgery). The method can include the one or more processors identifying a first manipulator component of the robotic medical system, a second manipulator component of the robotic medical system, and an optical device (e.g., data capture device) which can perform a phase or a task during a medical session. The manipulator components and / or data capture systems and their settings, positioning and location can be indicative of the task, type of task, phase or type of phase being performed.

[0139] At 910, the system can detect vertices of components or devices used to implement the task or the phase. The method can include the one or more processors detecting vertices associated with the first manipulator component, the second manipulator component, and the optical device during performance of the task during the medical session. For example, triangulation function can detect or monitor vertices of any number of manipulator components, medical tools manipulated or controlled by the manipulator components and data capture devices.

[0140] Vertices can include, for example coordinates or data on positioning, orientation or location of any of the manipulator components, medical tools or data capture devices. Vertices can include coordinates or indications of locations or positioning established with respect to a reference point in the control space on the surgeon or control side of the robotic medical system. Vertices can include coordinates or indications of locations or positioning established with respect to a reference point (e.g., a location of a corner of a room or device) in a medical environment in which the surgery is performed.

[0141] At 915, the system can determine a triangulation metric. The method can include determining a metric indicative of performance of at least a portion of the medical session that includes a task. The method can determine the metric using a triangulation function and the vertices. The method can include the one or more processors utilizing a triangulation function and the vertices to determine a metric. The metric can be a triangulation performance metric and can be indicative of performance of at least a portion of the medical session that includes the task. For example, the metric can correspond to, or can be, a matric for a task, such as a task being performed by the robotic medical system. For example, the metric can correspond to, or can be, a metric for a phase, such as a phase being performed.

[0142] The method can include the one or more processors identifying a time-series of the vertices associated with the first manipulator component, the second manipulator component, and the optical device. The method can include the one or more processors using the triangulation function to generate a time-series of the metric using the time-series of the vertices.

[0143] At 920, the system can determine if the metric exceeds a threshold. The method can include the one or more processors determining (e.g., based on a comparison of the metric with the threshold) whether the metric determined based on the vertices exceeds a threshold. For example, the one or more processors can select a threshold based on the type of task, or a type of a phase being performed. The threshold can correspond to a desired or recommended level of performance for the task, or the phase being performed.

[0144] The method can include comparing the metric with the threshold to determine the performance of the task, the phase or a portion of the medical session. The method can include comparing the metric of the task or phase being performed with the threshold for the given task or the phase responsive to detecting the phase or the task. For example, the one or more processors can use a threshold for a detected or identified task to compare against the metrics. For example, the one or more processors can use a threshold for a detected or identified phase to compare against the metric.

[0145] The one or more processors can normalize the metric based on the type of task to generate a normalized metric. The one or more processors can compare the normalized metric with a threshold corresponding to the given task, phase or portion of the medical session, in order to determine the performance of the current portion of the session (e.g., task or phase). The one or more processors can select the threshold based on an anatomical structure associated with the medical procedure. For example, the threshold can be detected responsive to detection of the anatomy of the patient at which the task is to be performed. The one or more processors can select the threshold based on a type of the first manipulator component, a type of the second manipulator component, a type of the data capture device being used, or a type of medical tool being utilized.

[0146] At 925, if the metric exceeds the threshold, a visual indication can be updated. The method can include the one or more processors displaying, via a display device, a visual indication of the metric. For example, the one or more processors can display an image, an illustration, animation, or an icon indicating, identifying or illustrating the triangulationperformance associated with the task, phase or a portion of the session. For example, the visual indication can include at least one of a color, a numerical score, a grade, or a binary value. For example, the one or more processors can provide at least one of a haptic indicator or an audio indicator based on the performance of the medical procedure. The one or more processors can provide, on a graphical user interface that displays the medical procedure, a real-time visual indication of the time-series of the metric. The real-time visual indication can include a continuously or periodically updated state or status of the level of performance.

[0147] The visual indication can include, for example, an image or icon whose shape, size, color or features can indicate the level of surgeon side triangulation and / or patient side triangulation. For example, visual indication can indicate or identify one or more triangles indicative of the surgeon side triangulation (e.g., the quality or level of a surgeon-side visibility of the surgical site or the target of surgery), and left-hand and right-hand triangulation (e.g., positioning, alignment, symmetry or ergonomics) with respect to a desired or recommended performance level. For example, a visual indication can indicate or identify a quality of the point of view (e.g., angle of view) of the surgeon with respect to the display, configuration of hand and eye alignment or symmetry and / or teleoperation performance.

[0148] At 930, if the metric does not exceed the threshold, the system can provide the indication to correct triangulation. The method can include the one or more processors displaying, via a display device, a visual indication of the metric, including any indication discussed in connection with 925 above. The visual indication can include or identify at least one of a color, a numerical score, a grade, or a binary value and can include an image, icon or animation indicative of the triangulation with respect to at least a portion of the medical session (e.g., task or phase) being performed.

[0149] The method can include the one or more processors determining, based on the metric less than or equal to the threshold, to provide guidance to improve the performance of the medical procedure. For example, in response to the metric not exceeding the threshold at 920 above, the one or more processors can generate an indication providing a recommendation to adjust the triangulation (e.g., positioning, location, orientation, ergonomics or scaling) with respect to any manipulator component, data capture device, a surgeon’s hand or finger location or positioning at the control space, or a positioning of the surgeon’s head or view with respect to a display.

[0150] The one or more processors can provide, for display on a display device, the guidance to improve the performance of the medical procedure. The guidance can include recommendation or instruction to the surgeon to perform a clutch operation on at least one of the first manipulator component, the second manipulator component or the optical device. The guidance can include a recommendation for the surgeon to reposition or adjust a portion of the surgeon’s body (e.g., hand, finger, eye or eyes, position of the head, posture, location of the leg or any other body part).

[0151] At 935, the system can perform corrective action. The method can include the one or more processors detecting, subsequent to provision of the visual indication of the performance, a clutch operation for at least one of the first manipulator component, the second manipulator component, or the optical device. For example, the surgeon may, responsive to the recommendation at 930, initiate a corrective action, such as a clutching action or a recalibration of a component or a device. The one or more processors can implement the clutching operation or the calibration, responsive to the surgeon’s input, selection or request.

[0152] The method can include the one or more processors generating a second one or more vertices subsequent to detection of the clutch operation. The one or more processors can provide a second visual indication of a second performance of the medical procedure based on the second vertices. The second performance can be greater than the prior performance (e.g., performance preceding the clutching action). The one or more processors can calibrate the first manipulator component and the second manipulator component prior to detection of the vertices. The one or more processors can identify a swap from the first manipulator component to a third manipulator component. The third manipulator component can be used to perform the at least one of the type of task or the type of phase of the medical procedure. The one or more processors can determine second vertices associated with the third manipulator component, the second manipulator component and the optical device. The one or more processors can generate a second metric based on the second vertices and can provide a second visual indication of a second performance of the medical procedure based on the comparison of the second metric with one of the threshold or the metric.

[0153] For example, a surgeon can perform a camera maneuver in which the surgeon can press the camera control or clutch pedal and disconnect the manipulator controller (hand control) from the instruments. This can allow movement of the camera with respect to the surgeon’s instrument location or with respect to the patient’s body. The patient sidetriangulation in this state may result in maneuvering camera. A surgeon can then maneuver two manipulator controllers to move the camera. In doing so, the hand positions of the surgeon can change from prior positions, as a result of which the surgeon side triangulation changes. The surgeon can then release the camera pedal when achieving the centered target position, which can connect the manipulator controller with the manipulator components (e.g., instruments) again. A better view to the target improves patient side triangulation in this state. The surgeon can then apply the clutch, either with the clutch foot pedal or finger clutch, which can disconnect the manipulator controller from the manipulator components as the camera pedal does. The surgeon can move the manipulator controller positions (hand positions) to new suitable ergonomic positions to improve surgeon side triangulation. The surgeon can then release the clutch to continue to manipulate the instruments and perform a surgical task. As a result of these actions, both the patient side triangulation and surgeon side triangulation can be improved, allowing for a more efficient and reliable surgical performance.

[0154] FIG. 10 depicts a surgical system 1000, in accordance with some embodiments. The surgical system 1000 may be an example of the medical environment 102. The surgical system 1000 may include a robotic medical system 1005 (e.g., the robotic medical system 120), a user control system 1010, and an auxiliary system 1015 communicatively coupled one to another. A visualization tool 1020 (e.g., the visualization tool 115) may be connected to the auxiliary system 1015, which in turn may be connected to the robotic medical system 1005. Thus, when the visualization tool 1020 is connected to the auxiliary system 1015 and this auxiliary system is connected to the robotic medical system 1005, the visualization tool may be considered connected to the robotic medical system. In some embodiments, the visualization tool 1020 may additionally or alternatively be directly connected to the robotic medical system 1005.

[0155] The surgical system 1000 may be used to perform a computer-assisted medical procedure on a patient 1025. In some embodiments, surgical team may include a surgeon 1030A and additional medical personnel 1030B-1030D such as a medical assistant, nurse, and anesthesiologist, and other suitable team members who may assist with the surgical procedure or medical session. The medical session may include the surgical procedure being performed on the patient 1025, as well as any pre-operative (e.g., which may include setup of the surgical system 1000, including preparation of the patient 1025 for the procedure), and post-operative (e.g., which may include clean up or post care of the patient), and / or other processes during themedical session. Although described in the context of a surgical procedure, the surgical system 1000 may be implemented in a non-surgical procedure, or other types of medical procedures or diagnostics that may benefit from the accuracy and convenience of the surgical system.

[0156] The robotic medical system 1005 can include a plurality of manipulator arms 1035A-1035D to which a plurality of medical tools (e.g., the medical tool 112) can be coupled or installed. Each medical tool can be any suitable surgical tool (e.g., a tool having tissueinteraction functions), imaging device (e.g., an endoscope, an ultrasound tool, etc.), sensing instrument (e.g., a force-sensing surgical instrument), diagnostic instrument, or other suitable instrument that can be used for a computer-assisted surgical procedure on the patient 1025 (e.g., by being at least partially inserted into the patient and manipulated to perform a computer-assisted surgical procedure on the patient). Although the robotic medical system 1005 is shown as including four manipulator arms (e.g., the manipulator arms 1035A-1035D), in other embodiments, the robotic medical system can include greater than or fewer than four manipulator arms. Further, not all manipulator arms can have a medical tool installed thereto at all times of the medical session. Moreover, in some embodiments, a medical tool installed on a manipulator arm can be replaced with another medical tool as suitable.

[0157] One or more of the manipulator arms 1035A-1035D and / or the medical tools attached to manipulator arms can include one or more displacement transducers, orientational sensors, positional sensors, and / or other types of sensors and devices to measure parameters and / or generate kinematics information. One or more components of the surgical system 1000 can be configured to use the measured parameters and / or the kinematics information to track (e.g., determine poses of) and / or control the medical tools, as well as anything connected to the medical tools and / or the manipulator arms 1035A-1035D.

[0158] The user control system 1010 can be used by the surgeon 1030 A to control (e.g., move) one or more of the manipulator arms 1035A-1035D and / or the medical tools connected to the manipulator arms. To facilitate control of the manipulator arms 1035A-1035D and track progression of the medical session, the user control system 1010 can include a display (e.g., the display 116 or 1130) that can provide the surgeon 1030A with imagery (e.g., high-definition 3D imagery) of a surgical site associated with the patient 1025 as captured by a medical tool (e.g., the medical tool 112, which can be an endoscope) installed to one of the manipulator arms 1035A-1035D. The user control system 1010 can include a stereo viewer having two or more displays where stereoscopic images of a surgical site associated with the patient 1025 andgenerated by a stereoscopic imaging system can be viewed by the surgeon 1030 A. In some embodiments, the user control system 1010 can also receive images from the auxiliary system 1015 and the visualization tool 1020.

[0159] The surgeon 1030A can use the imagery displayed by the user control system 1010 to perform one or more procedures with one or more medical tools attached to the manipulator arms 1035A-1035D. To facilitate control of the manipulator arms 1035A-1035D and / or the medical tools installed thereto, the user control system 1010 can include a set of controls. These controls can be manipulated by the surgeon 1030A to control movement of the manipulator arms 1035A-1035D and / or the medical tools installed thereto. The controls can be configured to detect a wide variety of hand, wrist, and finger movements by the surgeon 1030A to allow the surgeon to intuitively perform a procedure on the patient 1025 using one or more medical tools installed to the manipulator arms 1035A-1035D.

[0160] The auxiliary system 1015 can include one or more computing devices configured to perform processing operations within the surgical system 1000. For example, the one or more computing devices can control and / or coordinate operations performed by various other components (e.g., the robotic medical system 1005, the user control system 1010) of the surgical system 1000. A computing device included in the user control system 1010 can transmit instructions to the robotic medical system 1005 by way of the one or more computing devices of the auxiliary system 1015. The auxiliary system 1015 can receive and process image data representative of imagery captured by one or more imaging devices (e.g., medical tools) attached to the robotic medical system 1005, as well as other data stream sources received from the visualization tool. For example, one or more image capture devices (e.g., the image capture devices 110) can be located within the surgical system 1000. These image capture devices can capture images from various viewpoints within the surgical system 1000. These images (e.g., video streams) can be transmitted to the visualization tool 1020, which can then passthrough those images to the auxiliary system 1015 as a single combined data stream. The auxiliary system 1015 can then transmit the single video stream (including any data stream received from the medical tool(s) of the robotic medical system 1005) to present on a display (e.g., the display 130) of the user control system 1010.

[0161] In some embodiments, the auxiliary system 1015 can be configured to present visual content (e.g., the single combined data stream) to other team members (e.g., the medical personnel 1030B-1030D) who might not have access to the user control system 1010. Thus,the auxiliary system 1015 can include a display 1040 configured to display one or more user interfaces, such as images of the surgical site, information associated with the patient 1025 and / or the surgical procedure, and / or any other visual content (e.g., the single combined data stream). In some embodiments, display 1040 can be a touchscreen display and / or include other features to allow the medical personnel 1030A-1030D to interact with the auxiliary system 1015.

[0162] The robotic medical system 1005, the user control system 1010, and the auxiliary system 1015 can be communicatively coupled one to another in any suitable manner. For example, in some embodiments, the robotic medical system 1005, the user control system 1010, and the auxiliary system 1015 can be communicatively coupled by way of control lines 1045, which can represent any wired or wireless communication link that can serve a particular implementation. Thus, the robotic medical system 1005, the user control system 1010, and the auxiliary system 1015 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc. It is to be understood that the surgical system 1000 can include other or additional components or elements that can be needed or considered desirable to have for the medical session for which the surgical system is being used.

[0163] FIG. 11 depicts an example block diagram of an example computer system 1100 is shown, in accordance with some embodiments. The computer system 1100 can be any computing device used herein and can include or be used to implement a data processing system or its components. The computer system 1100 includes at least one bus 1105 or other communication component or interface for communicating information between various elements of the computer system. The computer system further includes at least one processor 1110 or processing circuit coupled to the bus 1105 for processing information. The computer system 1100 also includes at least one main memory 1115, such as a random-access memory (RAM) or other dynamic storage device, coupled to the bus 1105 for storing information, and instructions to be executed by the processor 1110. The main memory 1115 can be used for storing information during execution of instructions by the processor 1110. The computer system 1100 can further include at least one read only memory (ROM) 1120 or other static storage device coupled to the bus 1105 for storing static information and instructions for the processor 1110. A storage device 1125, such as a solid-state device, magnetic disk or optical disk, can be coupled to the bus 1105 to persistently store information and instructions.

[0164] The computer system 1100 can be coupled via the bus 1105 to a display 1130, such as a liquid crystal display, or active-matrix display, for displaying information. An input device 1135, such as a keyboard or voice interface can be coupled to the bus 1105 for communicating information and commands to the processor 1110. The input device 1135 can include a touch screen display (e.g., the display 1130). The input device 1135 can also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor 1110 and for controlling cursor movement on the display 1130.

[0165] The processes, systems and methods described herein can be implemented by the computer system 1100 in response to the processor 1110 executing an arrangement of instructions contained in the main memory 1115. Such instructions can be read into the main memory 1115 from another computer-readable medium, such as the storage device 1125. Execution of the arrangement of instructions contained in the main memory 1115 causes the computer system 1100 to perform the illustrative processes described herein. One or more processors in a multi-processing arrangement can also be employed to execute the instructions contained in the main memory 1115. Hard-wired circuitry can be used in place of or in combination with software instructions together with the systems and methods described herein. Systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

[0166] Although an example computing system has been described in FIG. 11, the subject matter including the operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0167] The herein described subject matter sometimes illustrates different components contained within, or connected with, different other components. It is to be understood that such depicted architectures are illustrative, and that in fact many other architectures can be implemented which achieve the same functionality. In a conceptual sense, any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality can be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any twocomponents so associated can also be viewed as being “operably connected,” or “operably coupled,” to each other to achieve the desired functionality, and any two components capable of being so associated can also be viewed as being “operably couplable,” to each other to achieve the desired functionality. Specific examples of operably couplable include but are not limited to physically mateable or physically interacting components or wirelessly interactable or wirelessly interacting components or logically interacting or logically interactable components.

[0168] With respect to the use of plural or singular terms herein, those having skill in the art can translate from the plural to the singular or from the singular to the plural as is appropriate to the context or application. The various singular / plural permutations can be expressly set forth herein for sake of clarity.

[0169] It will be understood by those within the art that, in general, terms used herein, and especially in the appended claims (e.g., bodies of the appended claims) are generally intended as “open” terms (e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “includes but is not limited to,” etc.).

[0170] Although the figures and description can illustrate a specific order of method steps, the order of such steps can differ from what is depicted and described, unless specified differently above. Also, two or more steps can be performed concurrently or with partial concurrence, unless specified differently above. Such variation can depend, for example, on the software and hardware systems chosen and on designer choice. All such variations are within the scope of the disclosure. Likewise, software implementations of the described methods can be accomplished with standard programming techniques with rule-based logic and other logic to accomplish the various connection steps, processing steps, comparison steps, and decision steps.

[0171] It will be further understood by those within the art that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation, no such intent is present. For example, as an aid to understanding, the following appended claims can contain usage of the introductory phrases “at least one” and “one or more” to introduce claim recitations. However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles “a” or “an” limits any particular claim containing such introduced claim recitation toinventions containing only one such recitation, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an” (e.g., “a” or “an” should typically be interpreted to mean “at least one” or “one or more”); the same holds true for the use of definite articles used to introduce claim recitations. In addition, even if a specific number of an introduced claim recitation is explicitly recited, those skilled in the art will recognize that such recitation should typically be interpreted to mean at least the recited number (e.g., the bare recitation of “two recitations,” without other modifiers, typically means at least two recitations, or two or more recitations).

[0172] Furthermore, in those instances where a convention analogous to “at least one of A, B, and C, etc.” is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “a system having at least one of A, B, and C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc.). In those instances where a convention analogous to “at least one of A, B, or C, etc.” is used, in general, such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “a system having at least one of A, B, or C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc.). It will be further understood by those within the art that virtually any disjunctive word or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase “A or B” will be understood to include the possibilities of “A” or “B” or “A and B.”

[0173] Further, unless otherwise noted, the use of the words “approximate,” “about,” “around,” “substantially,” etc., mean plus or minus ten percent.

[0174] The foregoing description of illustrative implementations has been presented for purposes of illustration and of description. It is not intended to be exhaustive or limiting with respect to the precise form disclosed, and modifications and variations are possible in light of the above teachings or can be acquired from practice of the disclosed implementations. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.

Claims

CLAIMSWhat is claimed is:

1. A system, comprising: one or more processors, coupled with memory, to: identify at least one of a type of task or a type of phase of a medical procedure; determine vertices associated with a first manipulator component, a second manipulator component, and an optical device, wherein the first manipulator component, the second manipulator component and the optical device are used to perform the at least one of the type of task or the type of phase of the medical procedure; generate, via a triangulation function, a metric based on the vertices; and provide a visual indication of a performance of the medical procedure based on a comparison of the metric with a threshold established for the type of task or the type of phase.

2. The system of claim 1, wherein the first manipulator component and the second manipulator component are communicatively coupled with a robotic medical system, wherein the robotic medical system configured to perform at least a portion of the medical procedure.

3. The system of claim 1 or 2, wherein the one or more processors are further configured to: identify a time-series of the vertices associated with the first manipulator component, the second manipulator component, and the optical device; and generate, via the triangulation function, a time-series of the metric using the time-series of the vertices.

4. The system of claim 3, wherein the one or more processors are further configured to: provide, on a graphical user interface that displays the medical procedure, a real-time visual indication of the time-series of the metric.

5. The system of claim 1 or 2, wherein the one or more processors are further configured to: calibrate the first manipulator component and the second manipulator component prior to detection of the vertices.

6. The system of claim 1, wherein the one or more processors are further configured to: select the threshold based on an anatomical structure associated with the medical procedure.

7. The system of claim 1, wherein the one or more processors are further configured to: select the threshold based on a type of the first manipulator component or a type of the second manipulator component.

8. The system of claim 1, 6 or 7, wherein the one or more processors are further configured to: detect, subsequent to provision of the visual indication of the performance, a clutch operation for at least one of the first manipulator component, the second manipulator component, or the optical device; generate second vertices subsequent to detection of the clutch operation; and provide a second visual indication of a second performance of the medical procedure based on the second vertices, wherein the second performance is greater than the performance.

9. The system of claim 1, 6 or 7, wherein the one or more processors are further configured to: identify a swap from the first manipulator component to a third manipulator component, wherein the third manipulator component is used to perform the at least one of the type of task or the type of phase of the medical procedure; determine second vertices associated with the third manipulator component, the second manipulator component and the optical device; generate a second metric based on the second vertices; and provide a second visual indication of a second performance of the medical procedure based on the comparison of the second metric with one of the threshold or the metric.

10. The system of claim 1, wherein the one or more processors are further configured to: determine, based on the metric less than or equal to the threshold, to provide guidance to improve the performance of the medical procedure; and provide, for display on a display device, the guidance to improve the performance of the medical procedure.

11. The system of claim 10, wherein the guidance comprises to perform a clutch operation on at least one of the first manipulator component, the second manipulator component or the optical device.

12. The system of claim 1, wherein the visual indication of the performance comprises at least one of a color, a numerical score, a grade, or a binary value.

13. The system of claim 1, wherein the one or more processors are further configured to: provide at least one of a haptic indicator or an audio indicator based on the performance of the medical procedure.

14. The system of claim 1, wherein the first manipulator component, the second manipulator component, and the optical device are coupled with a console, wherein the console controls one or more instruments used to perform the medical procedure.

15. The system of claim 1, wherein the optical device comprises a camera or a headset.

16. A method, comprising: identifying, by one or more processors coupled with memory, a first manipulator component, a second manipulator component, and an optical device that perform a task during a medical session; detecting, by the one or more processors, vertices associated with the first manipulator component, the second manipulator component, and the optical device during performance of the task during the medical session; determining, by the one or more processors using a triangulation function and the vertices, a metric indicative of performance of at least a portion of the medical session that includes the task; displaying, by the one or more processors via a display device, a visual indication of the metric.

17. The method of claim 16, comprising: identifying, by the one or more processors, a type of the task; selecting, by the one or more processors, a threshold based on the type of task; andcomparing, by the one or more processors, the metric with the threshold to determine the performance.

18. The method of claim 16, comprising: identifying, by the one or more processors, a type of the task; normalizing, by the one or more processors, the metric based on the type of task to generate a normalized metric; and comparing, by the one or more processors, the normalized metric with a threshold to determine the performance.

19. A non-transitory computer-readable medium storing processor executable instructions that, when executed by one or more processors, cause the one or more processors to: determine vertices associated with a first manipulator component, a second manipulator component, and an optical component, wherein the first manipulator component, the second manipulator component and the optical component are used to perform at least a portion of a medical procedure; determine, via a triangulation function, a performance of the at least the portion of the medical procedure based on the vertices; and display, via a display device, a visual indication of the performance with a video of the medical procedure captured by a camera.

20. The non-transitory computer-readable medium of claim 19, wherein the instructions further comprise instructions to cause the one or more processors to: identify a type of task or a type of phase associated with the at least the portion of the medical procedure; determine, via the triangulation function, a metric based on the vertices; select a threshold based on the type of the task or the type of the phase; and determine the performance based on a comparison of the metric and the threshold.

Images