Anatomy-based force feedback and instrument guidance

Abstract

Systems and methods for anatomy-based force feedback and instrument guidance are provided. A system receives a data stream of a medical procedure performed with a robotic medical system. The system identifies a type of an anatomical structure on which the medical procedure is performed. The system determines an amount of force applied to the anatomical structure. The system determines a metric indicative of performance of the medical procedure based at least on a comparison of the amount of force applied to the anatomical structure to a force threshold established for the type of anatomical structure. The system provides an indication of the metric for controlling performance of the medical procedure.

Description

[0001] Cross-references to related applications This application claims the priority of U.S. Provisional Patent Application No. 63 / 559,783, filed February 29, 2024, pursuant to 35 USC § 119, the entire contents of which are incorporated herein by reference. Background Technology

[0002] Remote operation of robotic systems offers several advantages to remote operators. For example, remote operators can operate such systems with greater control and precision compared to traditional techniques. Furthermore, when addressing operator fatigue, remote operators can pause the movement caused by the machine and resume operation once fatigue subsides. However, unlike traditional machines that apply forces to various structures without continuous feedback, remote operators often struggle to accurately perceive the state of one or more structures during remote operation, especially when these structures may be sensitive to repeated forces. Summary of the Invention

[0003] The technical solutions disclosed herein generally relate to systems and methods for anatomically based force feedback and instrument guidance. These solutions can determine the amount of force applied by an instrument (e.g., a medical device) during remote operation. For example, this document describes specific techniques for updating a user interface during remote operation to establish a particular remote operation experience. The user interface may be generated, at least in part, based on video data received from a sensing system during a medical procedure. The user interface may include one or more indicators that direct the execution of the medical procedure. These indicators may be in the form of numbers, letters, etc., or may be, for example, in the form of updating an image represented by the video data according to the techniques described herein.

[0004] This technical solution relates to a system. The system may include one or more processors coupled to a memory. The one or more processors may receive a data stream of a medical procedure executed by a robotic medical system. The one or more processors may use the data stream and one or more models trained with machine learning to identify the type of anatomical structure on which the medical procedure is performed. The one or more processors may use the data stream to determine the amount of force applied to the anatomical structure. The one or more processors may determine a metric instructing the execution of the medical procedure, at least based on a comparison of the amount of force applied to the anatomical structure with a force threshold established for the type of anatomical structure. The one or more processors may provide indications of the metric for controlling the execution of the medical procedure.

[0005] In some aspects, one or more processors can receive a data stream of a medical procedure, which includes data associated with force vectors representing the direction and magnitude of force interactions between instruments and anatomical structures involved in the medical procedure, the force vectors being captured at multiple points in time. In other aspects, one or more processors can determine the amount of force applied to the anatomical structure based at least on the force vectors.

[0006] In some respects, one or more processors can determine the amount of force applied to an anatomical structure, which represents the interaction between the instruments involved in the medical procedure and the anatomical structure at a certain point in time.

[0007] In some aspects, one or more processors can determine the amount of cumulative force applied to an anatomical structure over a period of time, the amount of cumulative force being determined at least based on the amount of force applied to the anatomical structure by instruments involved in the medical procedure at one or more time points within that period. In other aspects, one or more processors can determine the amount of force applied to the anatomical structure based at least on the amount of cumulative force applied to the anatomical structure.

[0008] In some aspects, one or more processors may determine the amount of force applied based at least on the contact between at least one instrument and the anatomical structure. In other aspects, one or more processors may determine the three-dimensional orientation associated with the contact between at least one instrument and the anatomical structure.

[0009] In some respects, one or more processors can determine the amount of force applied to an anatomical structure based at least on one or more sensor signals generated by one or more torque sensors. In other respects, the one or more torque sensors are configured to measure torque at one or more joints of one or more arms of a surgical robot, the one or more arms supporting the instrument that applies the amount of force to the anatomical structure.

[0010] In some aspects, one or more processors may update the amount of force applied to an anatomical structure based at least on the type of instrument associated with the instrument that applies the force to the anatomical structure. In other aspects, one or more processors may determine a metric instructing the execution of a medical procedure based at least on the update of the amount of force applied to the anatomical structure.

[0011] In some respects, one or more processors may determine a metric based at least on a comparison of the amount of force with a force threshold established for the type of anatomical structure, and one or more of the following: the type of instrument involved in the medical procedure, the orientation of the instrument involved in the medical procedure, the manipulation associated with the timing of the amount of force applied to the anatomical structure, or the amount of time of patient recovery associated with the amount of force applied to the anatomical structure.

[0012] In some respects, one or more processors may generate a user interface based at least on the data stream of the medical procedure and metrics instructing the execution of the medical procedure. In other respects, scanning by one or more processors causes a display device to display the user interface, the display of which occurs during or after a point in time when the medical procedure is performed by the robotic medical system.

[0013] In some aspects, one or more processors can generate visual representations of anatomical structures involved in a medical procedure. In other aspects, one or more processors can update the visual representations of the anatomical structures based at least on metrics instructing the execution of the medical procedure. In other aspects, one or more processors can cause a display device to display a user interface that includes at least a portion of the visual representation of the anatomical structures.

[0014] In some respects, one or more processors may update the visual representation of an anatomical structure based at least on a heatmap associated with the anatomical structure. In various respects, the heatmap includes a visual representation overlaid on the anatomical structure that indicates the execution of a medical procedure.

[0015] In some respects, one or more processors may determine indications of metrics that direct the execution of a medical procedure. In other respects, one or more processors may update the user interface based at least on one or more indications to include one or more indications at a certain location on the user interface.

[0016] In some respects, one or more processors can determine indications of metrics used to control the execution of medical procedures. In various respects, this indication represents the amount of force applied to the anatomical structure at the point in time when force is applied to the anatomical structure.

[0017] In some respects, one or more processors can determine indications of metrics controlling the execution of a medical procedure. In various respects, this indication represents the amount of force applied to the anatomical structure over a period of time during which force is applied to the anatomical structure.

[0018] In some respects, one or more processors can determine indications of metrics used to control the execution of a medical procedure. In various respects, these indications represent the three-dimensional orientation of the instrument in contact with the anatomical structures.

[0019] In some respects, one or more processors may determine the indications for controlling the execution of a medical procedure based at least on the metrics that instruct the execution of the medical procedure.

[0020] This technical solution relates to a method. The method may include one or more processors that receive a data stream of a medical procedure performed by a robotic medical system. The method may include one or more processors that use the data stream and one or more models trained with machine learning to identify the type of anatomical structure on which the medical procedure is performed. The method may include one or more processors that use the data stream to determine the amount of force applied to the anatomical structure. The method may include one or more processors that determine a metric instructing the execution of the medical procedure, at least based on a comparison of the amount of force applied to the anatomical structure with a force threshold established for the type of anatomical structure. The method may include one or more processors that provide indications of the metric for controlling the execution of the medical procedure.

[0021] In some aspects, the method may include one or more processors receiving a data stream of a medical procedure. In other aspects, the data stream includes data associated with force vectors, representing the direction and magnitude of force interactions between instruments and anatomical structures involved in the medical procedure, the force vectors being captured at multiple points in time. In other aspects, the method may include one or more processors determining the amount of force applied to the anatomical structure based at least on the force vectors.

[0022] In some aspects, the method may include one or more processors determining the amount of force applied to the anatomical structure, the amount of force representing the interaction between the instruments involved in the medical procedure and the anatomical structure at a certain point in time.

[0023] This technical solution relates to a non-transient computer-readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to receive a data stream of a medical procedure performed by a robotic medical system. The instructions may include instructions to use the data stream and one or more models trained with machine learning to identify the type of anatomical structure on which the medical procedure is performed. The instructions may include instructions to use the data stream to determine the amount of force applied to the anatomical structure. The instructions may include instructions to determine a metric instructing the execution of the medical procedure based at least on a comparison of the amount of force applied to the anatomical structure with a force threshold established for the type of anatomical structure. The instructions may include indications for providing a metric for controlling the execution of the medical procedure. Attached Figure Description

[0024] Figure 1A An example system for determining the force applied by the instrument during remote operation is described.

[0025] Figure 1B A schematic block diagram of an example environment for determining the amount of force applied by a device during remote operation, according to some embodiments, is shown. Figure 2A flowchart is shown of an example method for determining the amount of force applied by an instrument during remote operation, according to some embodiments. Figure 3 Images of example user interfaces according to some implementations are shown; Figure 4 A diagram illustrating example force limitations according to some implementation methods is shown; Figure 5 A schematic diagram of a medical environment according to some embodiments is shown; and Figure 6 A block diagram depicting the architecture of a computer system that can be used to implement the systems and methods described and illustrated herein is shown. Detailed Implementation

[0026] The following sections describe in more detail the various concepts and implementation schemes related to anatomical-based force feedback and instrument-guided methods, devices, and systems. The various concepts introduced above and discussed in more detail below can be implemented in any of a variety of ways.

[0027] Although this disclosure is discussed in the context of surgical procedures, in some embodiments, this disclosure may be applicable to other medical sessions or environments or activities, as well as non-medical activities where it is necessary to determine the force applied.

[0028] Systems, methods, apparatuses, and non-transient computer-readable media are provided for anatomical-based force feedback and instrument guidance. For example, the technique can determine the amount of force applied by an instrument (e.g., a medical device) during remote operation. In some embodiments, the method described herein includes receiving a data stream of a medical procedure performed by a robotic medical system; using the data stream and one or more models trained with machine learning to identify the type of anatomical structure on which the medical procedure is performed; using the data stream to determine the amount of force applied to the anatomical structure; and determining a metric indicative of the execution of the medical procedure, at least based on a comparison of the amount of force applied to the anatomical structure with a force threshold established for the type of anatomical structure. The arrangement also relates to providing indications of the metrics for controlling the execution of a medical procedure using a robotic medical system.

[0029] Because medical procedures such as surgery can be performed using the robotic system described herein, at any given point in time, the surgeon controlling such a robotic surgical system typically needs to estimate the forces applied by the medical instruments involved in the surgery, as well as the forces that may be applied in the future. This estimation is almost entirely based on images displayed to the surgeon via a user input system (described herein), at least during the surgical procedure. To address the inefficiencies associated with estimating the forces applied by medical instruments in this manner, this disclosure includes systems and methods for quantifying the amount of forces applied to one or more anatomical structures involved in surgery, determining metrics that indicate the execution of the amount of forces that can be applied, and providing the surgeon with indications of these metrics (e.g., via a user interface). The described techniques can improve perception of how the robotic system interacts with the patient during a medical procedure, reduce the chance of applying forces to anatomical structures that unnecessarily adversely affect the patient's short-term and long-term health outcomes, and improve overall patient outcomes. By understanding the state of one or more anatomical structures within the context of the metrics described herein, the surgeon can better understand possible future interactions and the downstream effects of such interactions. This, in turn, enables the surgeon to avoid prolonged surgical time or unintended damage to the anatomical structures involved in the medical procedure.

[0030] Figure 1A An example system 100 is depicted for determining the forces exerted by instruments during remote operation of a robotic system, such as a robotic medical system for robot-assisted surgery. Example system 100 may include a combination of hardware and software for generating indications of the amount of force during operation of the robotic system. For example, example system 100 may include a network 101, a medical environment 102, and a data processing system 130 as described herein.

[0031] Example system 100 may include medical environment 102 (e.g., with Figure 5 The example medical environment 500 is the same as or similar to the medical environment 102. Medical environment 102 includes one or more data capture devices 110, medical instruments 112, visualization tools 114, displays 116, and robotic medical systems (RMS) 120. RMS 120 may include or generate various types of data streams 158 as described herein, and may be operated using system configuration 122. One or more RMS 120 may be communicatively coupled to one or more data processing systems 130.

[0032] The RMS 120 can be deployed in any medical environment 102. The medical environment 102 may include any space or facility used to perform medical procedures, such as a surgical facility or operating room. The medical environment 102 may include medical devices 112 (e.g., surgical instruments for specific tasks) that the RMS 120 can use to perform procedures such as surgical patient procedures, whether invasive, non-invasive, or any inpatient or outpatient procedure. The RMS 120 may be centralized or distributed across multiple computing devices or systems, such as computing device 600 (e.g., used on servers, network devices, or cloud computing products), to implement various functions of the RMS 120, including communicating or processing data streams 158 across various devices via network 101.

[0033] The medical environment 102 may include one or more data capture devices 110 (e.g., optical devices, such as cameras or sensors, or other types of sensors or detectors) for capturing data stream 158. Data stream 158 may include any sensor data (such as images or videos of surgical procedures), kinematic data of any movement on the medical device 112, or any event data (such as installation, configuration, or selection events corresponding to the medical device 112). The medical environment 102 may include one or more visualization tools 114 to collect and process the captured data stream 158 for display to a user (e.g., a surgeon, medical professional, or engineer or technician configuring the RMS) via one or more (e.g., touchscreen) displays 116. The displays 116 may present data stream 158 (e.g., images or video frames) of medical procedures (e.g., surgical procedures) performed using the RMS 120, while processing, manipulating, holding, or otherwise utilizing the medical device 112 to perform surgical tasks at the surgical site. The RMS 120 may include a system configuration 122 based at least on the operability of the RMS 120, and its functionality may influence the direction of data flow 158.

[0034] System 100 may include one or more data capture devices 110 (e.g., cameras, sensors, or detectors) for collecting any data stream 158 that can be used for machine learning, including detecting objects from sensor data (e.g., video frames or force or feedback data), detecting specific events (e.g., user interface selection or surgeon engagement of medical device 112), or detecting kinematics (e.g., movement of medical device 112). Data capture devices 110 may include cameras or other image capture devices for capturing video or images from specific perspectives within the medical environment 102. Data capture devices 110 may be positioned, mounted, or otherwise positioned to capture content from any perspective, which facilitates the data processing system in capturing a variety of surgical tasks or actions.

[0035] Data capture device 110 may include various detectors, sensors, cameras, video imaging devices, infrared imaging devices, visible light imaging devices, light intensity imaging devices (e.g., black and white, color, grayscale imaging devices, etc.), hyperspectral imaging devices (e.g., hyperspectral cameras, etc.), depth imaging devices (e.g., stereo imaging devices, time-of-flight imaging devices, etc.), medical imaging devices (such as endoscopic imaging devices, ultrasound imaging devices, etc.), non-visible light imaging devices, any combination or sub-combination of the above imaging devices, or any other type of imaging device suitable for the purposes described herein. Data capture device 110 may include a camera that a surgeon can use to perform surgical procedures and observe manipulating parts within a field of view suitable for performing a given task. Data capture device may output any type of data stream 158, including data streams of kinematic data 158 (e.g., kinematic data streams), data streams of event data 158 (e.g., event data streams), and data streams of sensor data 158 (e.g., sensor data streams).

[0036] For example, data capture device 110 can capture, detect, or acquire sensor data, such as video or images, including, for example, still images, video images, vector images, bitmap images, other types of images (e.g., Raman hyperspectral images), or combinations thereof. Data capture device 110 can capture images at any suitable predetermined capture rate or frequency. Settings for each data capture device 110, such as scaling settings or resolution, can be varied as needed to capture suitable images from any perspective. For example, data capture device 110 may have a fixed perspective, position, orientation, or orientation. Data capture device 110 may be portable or otherwise configured to change orientation or scale in various directions. Data capture device 110 may be part of a multi-sensor architecture comprising multiple sensors, each configured to detect, measure, or otherwise capture a specific parameter (e.g., sound, image, or pressure).

[0037] Data capture device 110 can generate sensor data from any type and form of sensor, such as a positioning sensor, a biometric sensor, a velocity sensor, an acceleration sensor, a vibration sensor, a motion sensor, a pressure sensor, a light sensor, a distance sensor, a current sensor, a focus sensor, a temperature or pressure sensor, or any other type and form of sensor used to provide data for medical device 112 or data capture device (e.g., optical device). For example, data capture device 110 may include a position sensor, distance sensor, or positioning sensor that provides the coordinate position (e.g., kinematic data) of medical device 112. Data capture device 110 may include a sensor that provides information or data about the position, orientation, or spatial orientation of an object (e.g., a lens of medical device 112 or data capture device 110) relative to a reference point of kinematic data. The reference point may include any fixed, defined location that serves as the starting point for measuring distance and orientation in a particular direction, and as the origin for determining all other points or positions.

[0038] Display 116 can display, show, or play data stream 158, such as a video stream, in which a surgical site or nearby medical device 112 is shown. For example, display 116 can display a rectangular image of the surgical site and at least a portion of the medical device 112 used to perform surgical tasks. Display 116 can provide compiled or synthesized images generated by visualization tools 114 from multiple data capture devices 110 to provide visual feedback from one or more perspectives.

[0039] Visualization tool 114 can be configured or designed to receive any number of different data streams 158 from any number of data capture devices 110 and combine them into a single data stream displayed on display 116. Visualization tool 114 can be configured to receive multiple data stream components and combine them into a single data stream 158. For example, visualization tool 114 can receive visual sensor data about a surgical site or area where surgery is performed from one or more medical devices 112, sensors, or cameras. Visualization tool 114 can merge, combine, or utilize multiple types of data (e.g., positioning data of medical device 112 along sensor readings of pressure, temperature, vibration, or any other data) to generate output for presentation on display 116. Visualization tool 114 can present the position of medical device 112 as well as the position of any reference points or surgical sites, including the position of anatomical parts of the patient (e.g., organs, glands, or bones).

[0040] Medical device 112 can be any type and form of tool or medical device used in surgical procedures, medical procedures, or operating rooms or environments. Medical device 112 can be imaged by, associated with, or include an image capturing device. For example, medical device 112 can be a tool for forming an incision, a tool for suturing a wound, an endoscope for visualizing an organ or tissue, an imaging device, needles and sutures for suturing a wound, a scalpel, forceps, scissors, a retractor, a grasper, or any other tool or medical device used during surgery. Medical device 112 may include hemostatic agents, cannulas, surgical drills, aspiration devices, or any medical device used during surgery. Medical device 112 may include other or additional types of therapeutic or diagnostic medical imaging instruments. Medical device 112 can be configured to be mounted in, coupled to, or manipulated by RMS 120, such as via a manipulator arm or other components for holding, using, and manipulating the medical device. In some embodiments, medical device 112 can be associated with... Figure 5 The medical devices discussed are the same or similar.

[0041] RMS 120 can be a computer-aided system configured to perform surgical or medical procedures or activities on a patient via or using one or more robotic components or medical devices 112 or with their assistance. RMS 120 may include any number of manipulator arms for grasping, holding, or manipulating various medical devices 112 and performing computer-aided medical tasks using the medical devices 112 controlled by the manipulator arms.

[0042] Data stream 158 may be generated by RMS 120. For example, sensor data associated with data stream 158 may include images (e.g., video images) captured by medical device 112, which may be sent to visualization tool 114. For example, a surgeon may use display 116 (e.g., a touchscreen) to select, engage, or configure a specific medical device 112, thereby triggering events that may be indicated or included in data packets of data stream 158. RMS 120 may include one or more input ports to receive direct or indirect connections to one or more assistive devices. For example, when a medical device is mounted in RMS 120 (e.g., on a manipulator arm used to support medical device 112), visualization tool 114 may connect to RMS 120 to receive images from the medical device. For example, data stream 158 may include data indicating the positioning and movement of medical device 112, which may be captured or identified via kinematic data packets. Visualization tool 114 may combine data stream components from data capture device 110 and medical device 112 into a single combined data stream 158, which may be indicated or presented on display 116. RMS 120 can provide data streams 158 to data processing system 130 periodically, continuously, or in real time.

[0043] Data packets may include data units in data stream 158. Data packets may include actual information being transmitted and metadata, such as source and destination addresses, port identifiers, or any other information used for data transmission. Data packets may include data (e.g., payload) corresponding to an event (e.g., installation, unloading, engagement, or setup of medical device 112). Data packets may include data corresponding to sensor information (e.g., video frames captured by a camera) or data regarding the movement of medical device 112. Data packets may be transmitted in data stream 158, which can be separated or combined. For example, data stream 158 for kinematic data (e.g., kinematic data stream) may include multiple data packets indicating the movement of robot system components or features.

[0044] Data packets may include one or more timestamps, which can indicate a specific time when a particular event occurred. Timestamps can include time indications expressed in any combination of nanoseconds, microseconds, milliseconds, seconds, hours, days, months, or years. Timestamps may be included in the payload or metadata of the data packet and can indicate the time the data packet was generated, the time the data packet was transmitted from the device that generated it, the time the data packet was received by another device (e.g., a system within RMS 120 or another device on the network), or the time the data packet was stored in data repository 132.

[0045] Data repository 132 may include one or more data files, data structures, arrays, values, or other information that facilitates the operation of data processing system 130. Data repository 132 may include one or more local or distributed databases and may include a database management system. Data repository 132 may include, maintain, or manage one or more data streams 158. Data stream 158 may include, or be formed from, one or more of video streams, image streams, sensor measurement streams, event streams, or kinematic streams. Data stream 158 may include data collected by one or more data acquisition devices 110 (such as a set of 3D sensors) from various angles or observation points relative to procedural activities (e.g., surgical sites or areas).

[0046] Data stream 158 may include any data stream. Data stream 158 may include a video stream, comprising a series of video frames or video segments organized into segments, such as video segments of approximately 1, 2, 3, 4, 5, 10, or 15 seconds. Each second of video may include, for example, 30, 45, 60, 90, or 120 video frames per second. Data stream 158 may include an event stream, which may include event data or information streams, such as packets, that identify or convey the status of RMS 120 or events that occur in association with RMS 120. For example, data stream 158 may include any portion of system configuration 122, including information about operations on data stream 158, data regarding installation, uninstallation, calibration, setup, attachment, disassembly, or any other action performed by RMS 120 relative to medical device 112 or performed on RMS 120.

[0047] Data stream 158 may include data about events, such as indications of whether medical device 112 has been calibrated, adjusted, or the status of the RMS 120, including the manipulator arm mounted on the RMS 120. Data stream 158 representing event data (e.g., event data stream) may include data about whether the RMS 120 is functioning correctly during the procedure (e.g., without errors). For example, when medical device 112 is mounted on the manipulator arm of the RMS 120, a signal or (one or more) data packet may be generated indicating that medical device 112 has been mounted on the manipulator arm of the RMS.

[0048] Data stream 158 may include a kinematic data stream, which may refer to or include data related to one or more manipulator arms or a medical device 112 attached to one or more manipulator arms, such as arm position or orientation. Data corresponding to medical device 112 may be captured or detected by one or more displacement transducers, orientation sensors, position sensors, or other types of sensors and devices to measure parameters or generate kinematic information. Kinematic data may include sensor data as well as timestamps associated with data stream 158 and an indication of medical device 112 or the type of medical device 112.

[0049] Data repository 132 can store sensor data with video frames, which may include one or more still images or frames extracted from an image sequence of a video file. Video frames can represent a specific moment in time and can be identified by metadata including a timestamp. Video frames can display the visual content of a video of a medical procedure analyzed by data processing system 130 (e.g., by anatomical detector 146) to form a composite video and performance metrics indicating the surgeon performing the procedure. For example, in a video file capturing a robotic surgical procedure, video frames can depict snapshots of the surgical task, showing the movement or use of medical instruments 112, such as a robotic arm manipulating surgical tools inside a patient.

[0050] The data stream 158 corresponding to sensor data (e.g., video), events, and kinematics can include relevant, corresponding, or repetitive information that can be used for cross-data comparison and verification of consistency across all three data sources. For example, the detection function can check the consistency between different data types and data sources by mapping and comparing timestamps between different data types to assess whether they have evolved over time, such as based on the expected flow and correlation of events, video stream details, and kinematic values.

[0051] For example, the installation of medical device 112 can be recorded as a system event and provided in data stream 158 of the event data. At the same or similar expected time frame, the installed medical device 112 can be displayed in sensor data (e.g., in video), which can be detected by data processing system 130, which may include a computer vision model. Kinematic data can confirm movement of medical device 112 based on movement detected by data processing system 130. Using these cross-data stream correlation techniques, data processing system 130 can verify time synchronization between three data sources (e.g., three data streams 158).

[0052] Continue to refer to Figure 1A The data processing system 130 may include any combination of hardware or software performing one or more of the functions described herein. For example, the data processing system 130 may include any combination of hardware and software for determining the force applied by a medical device during remote operation. The data processing system 130 may include any computing device (e.g., with…) Figure 6The computing device 600 is the same or similar computing device and may include one or more servers, virtual machines, or may be part of or include a cloud computing environment. The data processing system 130 may be provided via centralized computing devices or via distributed computing components, such as servers comprising multiple logical groups and supporting distributed computing technologies. The logical group of servers may be referred to as a data center, server farm, or machine cluster. Servers may include virtual machines and may be geographically distributed. A data center or machine cluster may be managed as a single entity, or a machine cluster may include multiple machine clusters. Servers within each machine cluster may be heterogeneous—one or more servers or machines may run on one or more types of operating system platforms.

[0053] Data processing system 130 or its components may include physical or virtual computer systems operatively connected to or associated with medical environment 102. Data processing system 130 or its components may be connected to or associated with medical environment 102 directly or directly through intermediate computing devices or systems via network 101. Network 101 may be any type or form of network. The geographical scope of the network can vary widely and may include body area networks (BANs), personal area networks (PANs), local area networks (LANs, such as intranets), metropolitan area networks (MANs), wide area networks (WANs), or the Internet. The topology of network 101 can take any form, such as point-to-point, bus, star, ring, mesh, tree, etc. Network 101 may utilize different technologies and protocol layers or stacks, including, for example, Ethernet protocol, Internet Protocol Suite (TCP / IP), ATM (Asynchronous Transfer Mode) technology, SONET (Synchronous Fiber Networking) protocol, SDH (Synchronous Digital Hierarchy) protocol, etc. The TCP / IP Internet Protocol Suite may include application layer, transport layer, Internet layer (including, for example, IPv6), or link layer. Network 101 can be a broadcast network, telecommunications network, data communication network, computer network, Bluetooth network, or other types of wired and wireless networks.

[0054] Data processing system 130 or its components may be located, at least partially, at or away from, a surgical facility associated with medical environment 102. Elements of data processing system 130 or its components may be accessible via portable devices such as laptops, mobile devices, wearable smart devices, etc. Data processing system 130 or its components may include additional or supplementary elements that may be necessary to perform the functions described herein. Data processing system 130 or its components may include or be associated with one or more components or functions of a computing device, including, for example, one or more processors coupled to a memory that may store instructions, data, or commands for implementing the functions of data processing system 130 as described herein.

[0055] Data processing system 130 may include one or more of a data collector 144, an anatomical detector 146, a force predictor 148, a measurement predictor 150, an execution controller 152, or a data repository 132. Execution controller 152 may include a timer 154 or a user interface 156. Data processing system 130 may be communicatively coupled to one or more other data processing systems 130. For example, data processing system 130 may be communicatively coupled to one or more other data processing systems 130 that cooperate to perform one or more of the operations described herein.

[0056] Data processing system 130 may be implemented by one or more components of medical environment 102. For example, data processing system 130 may be implemented by one or more components of RMS 120. Data processing system 103 may receive one or more data streams 158 as described herein and may use system configuration 122 to monitor the operation of RMS 120. One or more RMS 120 may be communicatively coupled to one or more data processing systems 130. Data repository 132 may be configured to receive, store, and provide data streams 158 (e.g., one or more data packets associated with data streams 158) before, during, or after medical procedures. In some embodiments, data repository 132 stores data associated with one or more of the following: machine learning (ML) model 134, historical data 136 associated with one or more previously performed medical procedures involving RMS 120, type 138 (e.g., one or more force types), threshold 140 (e.g., a threshold representing a force limitation), or table 142 (e.g., a table representing one or more sets of force limitations).

[0057] Data collector 144 can be implemented by data processing system 130, or it can be integrated with... Figure 6 The computing device 600 is the same as or similar to the data stream 158. Data collector 144 can receive data stream 158. For example, data collector 144 can receive data streams via network 101. In an example, data collector 144 can receive data stream 158 from data processing system 130. Data collector 144 can receive data stream 158 from a medical procedure performed using RMS 120. In some embodiments, one or more packets associated with data stream 158 can represent one or more images during a medical procedure. One or more images can be captured by visualization tool 114 or otherwise obtained. As described herein, one or more images can represent one or more anatomical structures or one or more medical devices. In some embodiments, data collector 144 can provide data stream 158 (e.g., one or more packets of data stream 158) to anatomical detector 146, force predictor 148, or measurement predictor 150.

[0058] The anatomical detector 146 can be implemented by the data processing system 130, or it can be integrated with... Figure 6 The computing device 600 is the same as or similar to the computing device 600. The anatomical detector 146 can receive data stream 158. For example, the anatomical detector 146 can receive data stream 158 from the data collector 144. The anatomical detector 146 can identify the type of anatomical structure on which a medical procedure is performed. For example, the anatomical detector 146 can identify the type of anatomical structure on which a medical procedure is performed based at least on data stream 158. In some embodiments, the anatomical detector 146 can identify the type of anatomical structure on which a medical procedure is performed based at least on data stream 158 and an ML model. For example, the anatomical detector 146 can provide data stream 158 to an ML model so that the ML model provides an output representing the type of anatomical structure on which a medical procedure is performed. The anatomical detector 146 can provide data related to the anatomical structure type to a force predictor 148 or an execution controller 152.

[0059] Force predictor 148 can be implemented by data processing system 130, or it can be integrated with... Figure 6 The computing device 600 is the same as or similar to the computing device 600. Force predictor 148 can receive data stream 158. For example, force predictor 148 can receive data stream 158 from data collector 144. In some embodiments, force predictor 148 can receive data associated with the type of anatomical structure from anatomical detector 146, or force predictor 148 can receive data associated with the interaction type. Force predictor 148 can determine the amount of force applied to the anatomical structure. For example, force predictor 148 can determine the amount of force applied to the anatomical structure based at least on the type of anatomical structure or the interaction type with the anatomical structure. Force predictor 148 can provide data related to the amount of force applied to the anatomical structure to execution controller 152.

[0060] The metric predictor 150 can be implemented by the data processing system 130, or it can be integrated with... Figure 6 The computing device 600 is the same as or similar to the computing device 150. The metric predictor 150 can receive data stream 158. For example, the force predictor 148 can receive data stream 158 from the data collector 144. The metric predictor 150 can identify the type of interaction involving anatomical structures. For example, the metric predictor 150 can identify the type of interaction involving anatomical structures based at least on data stream 158. The metric predictor 150 can provide data related to the interaction type to the force predictor 148 or the execution controller 152. The anatomy detector 146 can identify the type of anatomical structure on which medical procedures are performed and provide data associated with the anatomical structure type to the metric predictor 150.

[0061] The execution controller 152 can be implemented by the data processing system 130, or it can be integrated with... Figure 6 The computing device 600 is the same as or similar to the device described herein. The execution controller 152 may receive data from the force predictor 148 associated with the amount of force applied to the anatomical structure. In some embodiments, the execution controller 152 may receive data from the measurement predictor 150 associated with one or more measurements described herein. In some embodiments, the execution controller 152 may determine indications of one or more measurements as described herein to control the execution of medical procedures utilizing the robotic medical system.

[0062] In the example, execution controller 152 may determine user interface 156. In the example, execution controller 152 determines user interface 156 periodically (e.g., every 1 second, every 2 seconds, etc.). In some examples, execution controller 152 determines user interface 156 continuously. In some implementations, execution controller 152 provides an indication of a measurement, wherein the indication is represented or is represented by user interface 156. In some implementations, execution controller 152 may provide data associated with the indication of a measurement to cause the device to display an indication of the amount of force. For example, execution controller 152 may provide data associated with the indication of a measurement to cause display 116 to display the indication of a measurement. In this example, the data associated with the indication of a measurement may be configured to cause display 116 to display the indication.

[0063] Data repository 132 can be implemented by data processing system 130, or it can be integrated with... Figure 6 The computing device 600 is the same as or similar to the device. The data repository can be directly or indirectly (e.g., via data processing system 130) from... Figure 1A Data is received by any device. The data may include ML model 134, historical data 136, type 138, threshold 140, or table 142. In one example, the data stored in data repository 132 is associated with a previously performed medical procedure. In some examples, the data stored in data repository 132 is associated with the current medical procedure. In some implementations, data repository 132 may receive data stream 158 or system configuration 122 and store both. Data repository 132 may then provide data stream 158 or system configuration 122 (e.g., one or more data packets thereof) to one or more components of data processing system 130.

[0064] Figure 1BThis is a schematic block diagram illustrating an example environment 160 according to some embodiments, in which the devices, systems, methods, or products described herein may be implemented. As shown, environment 160 includes a measurement prediction system 162, a sensing system 164, and a user input system 168. In some embodiments, the measurement prediction system 162 and... Figure 1A The data processing system 130 is the same as or similar to the data processing system 130. In some embodiments, the sensing system 164 is the same as... Figure 1A One or more data capture devices 110 are the same as or similar to each other.

[0065] The measurement prediction system 162 can receive video data 166 from the sensing system 164. Furthermore, the measurement prediction system 162 can receive robot system data 170 and medical device data 172. In this example, the measurement prediction system 162 can receive video data 166, robot system data 170, or medical device data 172 as part of a data stream. The data stream can be received from... Figure 1A The robotic medical system 120 is the same as or similar to the robotic medical system receiving the data stream (e.g., one or more packets included in the data stream). Figure 1A The data stream 158 is the same or similar.

[0066] The measurement prediction system 162 can also communicate with the user input system 168 (e.g., establish a communication connection to exchange data). The measurement prediction system 162, sensing system 164, or user input system 168 may include one or more suitable computing systems, such as… Figure 6 The computing device 600 may be implemented by one or more suitable computer systems. For example, the measurement prediction system 162, sensing system 164, or user input system 168 may include one or more components that are the same as or similar to one or more components of the computing device 600. In some embodiments, the measurement prediction system 162, sensing system 164, or user input system 168 may be configured to communicate (e.g., establish a communication connection to exchange data). In some embodiments, system 100 may include components related to... Figure 5 The example medical environment 500 refers to one or more devices or systems that are the same as or similar to the one or more devices or systems discussed.

[0067] In some examples, the processes described herein, such as generating one or more user interfaces, can be implemented by the metric prediction system 162. Some or all of the processes implemented by the metric prediction system 162 can be implemented by one or more other devices (alone or in cooperation with the metric prediction system 162), such as the sensing system 164 or the user input system 168, which can be integrated with... Figure 6The computing device 600 is the same as or similar to the user input system 168. Although the metric prediction system 162 is shown as a system separate from the user input system 168, in this example, the metric prediction system 160 may be included in (e.g., implemented by) the user input system 168. Therefore, one or more functions performed by the metric prediction system 162 as described herein can be similarly performed by the user input system 168.

[0068] In some implementations, the metric prediction system 162 can be coupled with... Figure 6 The computing device 600 is the same as or similar to the computing device 600. In some embodiments, the user input system 168 may be similar to... Figure 5 User control system 510 or Figure 6 The computing device 600 is the same as or similar to the computing device 600. In some embodiments, the sensing system 164 may be the same as... Figure 6 The computing device 600 is the same as or similar to the computing device 600. In some embodiments, the measurement prediction system 162 can receive robot system data 170 from the sensing system 164, wherein the sensing system includes a device that is connected to the manipulator arm (e.g., with...). Figure 5 The manipulator arms 535A-535D (which are the same or similar to the manipulator arms) support one or more medical devices that are the same or similar to those described herein, such as imaging devices (e.g., endoscopes, ultrasound tools, etc.) or sensing devices (e.g., force-sensing surgical instruments). In some embodiments, the imaging device or sensing device is associated with (e.g., included in or implemented by) the sensing system 164.

[0069] As used herein, a medical procedure refers to a procedure performed by one or more medical personnel, robotic systems, or medical devices in a medical setting (e.g., with...). Figure 5 Surgical procedures or operations performed in 500 or more identical or similar medical or surgical operating rooms. Examples of medical personnel include surgeons, nurses, support staff, etc. (e.g., who may work with...) Figure 5The surgeon 530A or other medical personnel 530B-530D are the same or similar individuals. Examples of robotic systems include the robotic medical systems or robotic surgical systems described herein, such as one or more devices in medical environment 500 (e.g., robotic medical system 524). Examples of medical devices include medical devices supported by manipulator arms 535A-535D. Medical procedures can have various modes, including robotic (e.g., using at least one robotic system), non-robotic laparoscopy, non-robotic incision, etc. Robotic system data 170 and medical device data 172 collected during a medical procedure also refer to or include robotic system data 170 and medical device data 172 collected by one or more devices in a medical environment (e.g., medical environment 500) in which the medical procedure is performed, as well as medical device data for one or more medical personnel, robotic systems, or medical devices used in performing or performing medical procedures.

[0070] The measurement and prediction system 162 can receive and process data sources or data streams, including one or more of video data 166, robotic system data 170, and medical device data 172 collected for training or medical procedures. For example, the measurement and prediction system 162 can acquire data streams of video data 166, robotic system data 170, and medical device data 172 in real time. In some examples, when generating one or more user interfaces (UIs) as described herein, the measurement and prediction system 162 can utilize all types of robotic system data 170 and medical device data 172 collected, acquired, determined, or calculated for medical procedures.

[0071] In some embodiments, the measurement prediction system 162 receives video data 166, robot system data 170, or medical device data 172 during operation of the robotic system. For example, during operation of the robotic system, the measurement prediction system 162 may receive video data 166 from the sensing system 164. The video data 166 may be associated with one or more images captured individually or sequentially by an imaging device included in the sensing system 164. In some embodiments, the imaging device includes a visual imaging endoscope, laparoscopic ultrasound, a camera, etc. Other suitable imaging devices are also contemplated. In some embodiments, the sensing system 164 includes a repositionable component comprising one or more links supported by the robotic system. For example, the sensing system 164 may include a repositionable component comprising one or more links that can be hinged by the robotic system at least based on input provided by a surgeon via the user input system 168 described herein.

[0072] In the example, the measurement prediction system 162 can receive robot system data 170 from a robotic system (e.g., from one or more components of the robotic system). In some implementations, the robot system data 170 includes a system event stream, which also includes data associated with one or more system events (e.g., the status of one or more devices, such as whether one or more devices or medical instruments are connected to the robotic system, whether one or more devices are operating as expected, error messages, or the like). The robotic system may include one or more devices or components of a robotic medical system (e.g., with...). Figure 5 The robotic medical system 524 is the same as or similar to the robotic medical system on which one or more tools are supported (e.g., similar to those provided by...). Figure 5 The manipulator arms 535A-535D support one or more tools that are the same as or similar to the medical device. In the example, the measurement prediction system 162 may receive medical device data 172, wherein the medical device data 172 is associated with the state of one or more tools supported by the robotic medical system. In some embodiments, the robotic system may include a user input system 168 (e.g., with...). Figure 5 The user control system 510 is the same as or similar to the first surgeon's console.

[0073] One or more images captured by an imaging device included in sensing system 164 can show at least a portion of at least one medical device (tool, surgical instrument, or the like) within the field of view of the imaging device. For example, sensing system 164 may include an imaging device supported along a distal portion of a tool (e.g., supported by a robotic medical system 524, e.g., mounted on the robotic medical system). Sensing system 164 may be operated by a medical professional during a training session where the professional familiarizes themselves with the robotic system or practices certain actions using the robotic system. Sensing system 164 may be operated by a medical professional during a surgical procedure performed by a surgeon using user input system 168. In these examples, sensing system 164 may be operated such that the imaging device of sensing system 164 is positioned to capture and generate an image of at least a distal portion of at least one medical device within the field of view of the imaging device included in sensing system 164, wherein the field of view is directed to at least a portion of the patient's tissue. Images may be captured when the robotic system controls one or more medical devices based at least on input received by user input system 168.

[0074] Robot system data 170 can be associated with the control state of one or more devices of the robot system, at least based on input received by user input system 168. For example, when user input system 168 communicates with the robot system to control at least one medical device, the robot system can generate robot system data 170 and provide it to measurement prediction system 162. Robot system data 170 can indicate whether user input system 168 is controlling one or more medical devices, whether user input system 168 is generating control signals configured to manipulate one or more medical devices within the field of view of the sensing system, apply torque at one or more joints involved in supporting the one or more medical devices, or the like. In some embodiments, robot system data 170 can be associated with forces applied by the robot system to one or more anatomical structures. For example, robot system data 170 can be generated by the robot system at least based on the movement of one or more links or one or more components of the medical devices of the robot system. In an illustrative example, where one or more links of the robot system are configured to support the positioning and repositioning of a medical device, one or more sensors corresponding to one or more links can generate sensor signals representing the forces applied by the links during the repositioning of the medical device. Sensor signals can be included in robot system data 170, which in turn is included in the data stream. One or more different sensors (e.g., encoders or the like) can be used to generate sensor data indicating the position of the links relative to each other and the robot system. This sensor data can later be used to deduce the position of a medical device supported by the links.

[0075] Medical device data 172 can be associated with one or more states of one or more medical devices of the robotic system. For example, medical device data 172 can be associated with one or more states of one or more medical devices controlled by user input system 168 during remote operation of the robotic system. One or more states can indicate whether one or more medical devices of the robotic system are performing one or more functions. As an example, functions may include tool activation, movement of medical devices, or similar operations as described herein. One or more states can indicate whether one or more medical devices are controlled by the robotic system, at least based on input received by user input system 168.

[0076] Continue to refer to Figure 1BIn some embodiments, the measurement prediction system 162 receives a data stream of a medical procedure. For example, the measurement prediction system 162 may receive a data stream including data associated with force vectors, where the force vectors represent the direction and magnitude of force interactions between the medical device and the anatomical structures involved in the medical procedure. In some embodiments, the measurement prediction system 162 may receive a data stream including data associated with force vectors, where the force vectors include gripping force vectors. For example, the measurement prediction system 162 may receive a data stream including data associated with gripping force vectors, where the gripping force vectors represent the direction and magnitude of force interactions between the various parts of the device and the anatomical structures gripped by the various parts of the device during the medical procedure. In an example, the measurement prediction system 162 may receive data associated with force vectors at multiple points in time during the medical procedure. In this example, the multiple points in time may be instantaneous points in time (e.g., the procedure may occur in real time). In some embodiments, the measurement prediction system 162 may receive data associated with force vectors at multiple points in time prior to the instantaneous point in time. For example, when measurement prediction system 162 determines the cumulative force associated with the interaction between one or more medical devices and one or more anatomical structures, measurement prediction system 160 may receive data associated with the force vector at multiple time points prior to an instantaneous time step. In this example, the data may be generated during the current medical procedure or a previous medical procedure (e.g., a previous medical procedure associated with a patient or surgeon). In some implementations, measurement prediction system 162 determines the amount of force applied to or to be applied to the anatomical structure based at least in part on force vectors captured at multiple time points when the medical device interacts with the anatomical structure involved in the medical procedure.

[0077] In some implementations, the measurement prediction system 162 receives a data stream of a medical procedure, which includes data associated with the skill levels of one or more surgeons involved in the procedure. For example, the measurement prediction system 162 may receive data associated with the skill levels of one or more surgeons involved in the medical procedure, based at least in part on one or more interactions between the medical devices of the robotic system and the anatomical structures involved in the medical procedure. Skill levels may be represented, for example, the amount of previous interactions involving similar medical devices and anatomical structures during previous medical procedures performed by the surgeon, a score representing patient outcomes specifically involving interactions involving similar medical devices and anatomical structures during previous medical procedures performed by the surgeon, a force vector associated with previous performance interactions involving the surgeon, or other historical information that can be used to determine force limitations.

[0078] The measurement and prediction system 162 can receive a data stream of a medical procedure, wherein the data stream includes robot system data 170 associated with kinematic information or system event information corresponding to the operation of the robot system. In one embodiment, the measurement and prediction system 162 receives a data stream of a medical procedure, wherein the data stream includes data associated with one or more aspects of the medical procedure (e.g., the type of medical procedure, the level of complexity associated with the medical procedure, fragments of the medical procedure or the like associated with stages, tasks or steps). In some embodiments, the measurement and prediction system 162 receives patient data associated with information about the patient, such as the patient's age, demographics, whether the patient has a compromised immune system (or is ill at the time of the medical procedure), the tensile strength of one or more anatomical structures involved in the medical procedure, and any other such information.

[0079] The metric prediction system 162 can identify the type of anatomical structure involved in a medical procedure. For example, the metric prediction system 162 can identify the type of anatomical structure involved in a medical procedure based at least in part on a data stream (e.g., one or more aspects of data represented by the data stream). The metric prediction system 162 can identify the type of anatomical structure involved based at least in part on the type of medical procedure. The metric prediction system 162 can identify the type of anatomical structure based on image interpretation techniques using one or more models trained using machine learning. For example, in the case where the medical procedure involves addressing a patient's abdominal hernia, the metric prediction system 162 can determine the type of anatomical structure involved based at least in part on one or more anatomical structures accessible to the robotic system during the medical procedure.

[0080] In some implementations, the metric prediction system 162 establishes reference frames. For example, the metric prediction system 162 may establish reference frames based at least on one or more interactions involved in a medical procedure. Reference frames may include, for example, portions of a data stream associated with a given interaction (e.g., images of video data 166). In some implementations, the metric prediction system 162 establishes reference frames such that the reference frames correspond to (e.g., focus on) one or more interactions.

[0081] In some implementations, the metric prediction system 162 identifies the type of anatomical structure involved based at least in part on one or more models trained with machine learning. For example, the metric prediction system 162 may receive video data 166 from the sensing system 164 during a medical procedure. In this example, the metric prediction system 162 may feed the video data 166 to one or more models to generate outputs. The outputs may represent one or more classifications, each corresponding to an identifier of one or more anatomical structures represented by the video data 166. In some implementations, one or more classifications may be performed on a frame or pixel basis. In some implementations, one or more classifications may be performed at least in part on a set or multiple sets of pixels. For example, one or more classifications may be associated with one or more bounding boxes or segmentation masks generated by one or more models, each corresponding to a group of pixels representing one or more anatomical structures.

[0082] In some implementations, the metric prediction system 162 uses one or more models trained using machine learning to identify the type of anatomical structure on which a medical procedure is performed and the type of interaction with the anatomical structure. For example, the metric prediction system 162 may receive video data 166 from the sensing system 164 during a medical procedure, and the metric prediction system 162 may provide the video data 166 to one or more models to generate outputs. In this example, the output may represent one or more classifications corresponding to identifiers of one or more anatomical structures or one or more medical devices represented by the video data 166. The one or more classifications may correspond to identifiers of one or more interactions between one or more anatomical structures or one or more medical devices represented by the video data 166. In an illustrative example, when a medical device is used to move an anatomical structure during a medical procedure, one or more images associated with the video data 166 may be provided to one or more models to generate outputs representing the movement of the anatomical structure by one or more medical devices. In some implementations, the output may be further represented as an indication of the type of interaction, wherein the type includes one or more of the following: grasping interaction (e.g., grasping at least a portion of an anatomical structure using the jaws of an end effector supported by a medical device), retraction interaction (e.g., holding back or separating tissue associated with one or more anatomical structures), cutting interaction (e.g., cutting at least a portion of an anatomical structure), or cauterization interaction (e.g., cauterizing tissue associated with one or more anatomical systems using, for example, an electrocautery system, a chemical cauterization system, etc.).

[0083] In some implementations, the metric prediction system 162 may provide robotic system data 170 or medical device data 172 to one or more models to enable the models to identify the type of anatomical structure on which a medical procedure is performed, and the type of interaction with that anatomical structure. For example, during training (discussed below), one or more models may be trained on training data, including video data 166 from previous medical procedures, robotic system data 170 from previous medical procedures, or medical device data 172 from previous medical procedures. In these examples, the metric prediction system 162 may provide data from a data stream received during a surgical procedure to one or more models to enable the models to generate the aforementioned output.

[0084] In some implementations, the metric prediction system 162 may provide one or more models with previously generated video data 166 from a previous medical procedure, robotic system data 170 from a previous medical procedure, or medical device data 172 from a previous medical procedure, to enable one or more models to generate the output described herein. For example, the metric prediction system 162 may provide previously generated video data 166 from a previous medical procedure, robotic system data 170 from a previous medical procedure, or medical device data 172 from a previous medical procedure, wherein the previously generated data corresponds to one or more earlier time points and one or more labels. One or more labels may be determined based at least on input received by an individual annotating the previously generated data. For example, annotations may correspond to the type of anatomical structure involved at a given time point, the location of the anatomical structure involved at a given time point, the medical device involved in the interaction at a given time point, or the type of interaction involved at a given time point. The metric prediction system 162 may compare the output of one or more models with corresponding inputs to one or more models (e.g., classifications generated by one or more models versus labels corresponding to inputs to one or more models) and determine the differences between the output and the input. The metric prediction system 162 can update one or more models by changing one or more weights associated with one or more models, and repeat the training process until one or more models converge.

[0085] In some implementations, the measurement prediction system 162 determines one or more force vectors based at least on the data stream of the medical procedure. For example, the measurement prediction system 162 may determine one or more force vectors, wherein the one or more force vectors represent the direction and magnitude associated with the operation of the medical device during interaction (e.g., contact) with the anatomical structure. In some implementations, when the medical device begins to contact the anatomical structure, the measurement prediction system 162 may determine the force applied to the anatomical structure based at least in part on the force and direction of the medical device at one or more time points.

[0086] In some implementations, the measurement prediction system 162 determines one or more force vectors based at least on the context in which one or more interactions occur. For example, the measurement prediction system 162 may determine one or more anatomical structures and one or more interactions involving a medical device and one or more anatomical structures based at least on the data stream as described herein. The measurement prediction system 162 may then determine one or more force vectors based at least on the context associated with the interaction between the medical device and one or more anatomical structures. For example, in the case where the interaction involves the medical device moving an anatomical structure, the measurement prediction system 162 may determine that the movement is associated with a predetermined context, and the measurement prediction system 162 may determine the force vectors based at least on that context. In the example, the measurement prediction system 162 may update aspects of the force vectors (e.g., increase or decrease the measured force or direction, etc.) at least based on the context.

[0087] In some implementations, the measurement prediction system 162 determines the magnitude of force interactions between anatomical structures and medical devices involved in a medical procedure based at least on data streams. For example, the measurement prediction system 162 may determine the magnitude of force interactions based at least in part on robot system data 170. In such an example, the measurement prediction system 162 may determine the magnitude of forces based at least in part on sensor signals (e.g., torque measurements as described herein) representing forces applied by links when repositioning the medical device or maintaining contact with an anatomical structure (e.g., when grasping, moving, or holding the anatomical structure).

[0088] In some implementations, different sensors can generate sensor data indicating the orientation of the links relative to each other or the robotic system. This sensor data can be included in a data stream. The sensor data can later be used to deduce the orientation of a medical device supported by the links. For example, when a robotic system (e.g., one or more components of the robotic system) is registered relative to the orientation or posture of a patient, sensor data indicating the orientation of the links relative to each other and the robotic system can be used to determine the relative orientation of the medical device and the patient's anatomy.

[0089] In some implementations, the measurement prediction system 162 determines the amount of force applied to the anatomical structure. For example, the measurement prediction system 162 may determine the amount of force applied to the anatomical structure based at least on a force vector associated with a given interaction. In some implementations, one or more sensor signals generated by one or more torque sensors may be used to determine the amount of force. In this example, one or more sensor signals may be generated by one or more manipulator arms (e.g., with one or more manipulator arms). Figure 5One or more torque sensors or links of one or more manipulator arms (identical to or similar to manipulator arms 535A-535D) are associated with (e.g., mounted or included therein) these manipulator arms. In some embodiments, the torque sensors are configured to measure torque at one or more joints of one or more manipulator arms or links. For example, when a force is applied to an anatomical structure based at least on the movement of one or more manipulator arms, links, or medical devices, the measurement prediction system 162 can receive and analyze sensor signals (via a data stream) to determine the amount of force applied to the anatomical structure based at least on the torque measured at each joint. In some embodiments, the measurement prediction system 162 can take into account (e.g., subtract) the torque required to move one or more manipulator arms, links, or medical devices associated with inertia or the weight of a component to determine the actual torque applied to the anatomical structure during interaction. In this way, the measurement prediction system 162 can determine the amount of force applied to a given anatomical structure during a medical procedure (e.g., a medical device supported by a surgical robot).

[0090] The measurement prediction system 162 can determine the amount of force applied to an anatomical structure based at least on one or more force vectors. For example, the measurement prediction system 162 can determine one or more force vectors that represent the direction and magnitude associated with the operation of a medical device during a medical procedure. In some embodiments, the amount of force applied is associated with (e.g., corresponds to) one or more interactions between the medical device and the anatomical structure involved in the surgical procedure. In some embodiments, the measurement prediction system 162 determines one or more force vectors based at least in part on images generated during the medical procedure and torque measured at the joint by a torque sensor.

[0091] The measurement prediction system 162 can determine the amount of force applied to an anatomical structure based at least on one or more forces not applied by the medical device. For example, the measurement prediction system 162 can determine the amount of applied force based at least on the application of force vectors and other forces (e.g., gravity) determined during the interaction. In an illustrative example, when the external jaws of the anastomosis device are used to hold a portion of the patient's intestine or a moving organ, the measurement prediction system 162 can first determine the force applied to the portion of the patient's intestine or the moving organ based at least on the force vector associated with the interaction. The measurement prediction system 162 can then update the forces applied to the intestine and organ based at least on the effect of gravity on the intestine and organ during the interaction. Thus, even in cases where the interaction between the medical device and the anatomical structure is relatively small, as illustrated by the measured torque involved in the interaction, the measurement prediction system 162 can determine the amount of force applied to an anatomical structure (e.g., the intestine), which can move fluidly, based at least on the tensile strength of the anatomical structure.

[0092] The measurement prediction system 162 can determine the amount of force applied to the anatomical structure based at least on the properties of the anatomical structure involved in the interaction. For example, the measurement prediction system 162 can determine that the anatomical structure involved in the interaction is associated with a predetermined tensile strength. The measurement prediction system 162 can then determine the amount of applied force based at least on the force vector of the interaction and the amount of expected resistance corresponding to the tensile strength associated with the anatomical structure. In this way, the measurement prediction system 162 can more accurately determine the amount of force applied to different types of anatomical structures, at least based on the expected resistance of the anatomical structure.

[0093] The metric prediction system 162 can determine the context of the interaction. For example, the metric prediction system 162 can determine the context of the interaction based at least on the force vectors associated with the interaction. In an illustrative example, when the context is associated with normal operation (e.g., the expected execution of movement), the metric prediction system 162 can receive video data 166, robot system data 170, or medical device data 172 and analyze the data to confirm that the context is associated with normal operation. In this example, the metric prediction system 162 can analyze the data and determine that the data indicates that the percentage of forces applied throughout the interaction is less than a threshold amount (e.g., less than 2 Newtons), and that different percentages of the applied forces are equal to or greater than the threshold amount (e.g., equal to or greater than 2 Newtons). The metric prediction system 162 can then determine that, for a given interaction, operation of the medical device at or above the threshold amount is expected (e.g., forces applied during cutting, stapling, etc.). The metric prediction system 162 can determine or update the expected amount based at least on the analysis of similar medical procedures (e.g., historical procedures) involving similar anatomical structures.

[0094] In some implementations, the measurement prediction system 162 determines one or more interactions (e.g., interaction types) or a surgeon's force signature or set of force signatures. For example, the measurement prediction system 162 may determine the force signature based at least on one or more force vectors associated with a medical device involved in interacting with anatomical structures. The measurement prediction system 162 may determine the force signature based at least on force vectors associated with similar interactions in multiple surgical procedures.

[0095] In some implementations, the measurement prediction system 162 determines force characteristics of one or more stacks. For example, the measurement prediction system 162 may determine force characteristics of one or more stacks (e.g., based at least on a combination of one or more distinct force characteristics). In this example, the measurement prediction system 162 may determine force characteristics of one or more stacks based at least on one or more force characteristics involved in the interaction between the medical device and the anatomical features. In some implementations, the force characteristics or the force characteristics of the stacks may be associated with a surgeon. In some implementations, one or more force characteristics or the force characteristics of one or more stacks may be associated with a medical procedure (e.g., a type of medical procedure).

[0096] In some implementations, the metric prediction system 162 trains a model to determine one or more force features as described herein. For example, the metric prediction system 162 may provide data associated with the force features determined by the metric prediction system 162 or data associated with a given surgical procedure (e.g., data included in a data stream) as input to the model, causing the model to generate an output. The metric prediction system 162 may then compare the output with an expected output (e.g., corresponding to a force feature, one or more forces applied in association with the force feature, etc.) to determine the difference between the output and the expected output. The metric prediction system 162 may then update one or more weights of the model and iteratively repeat the process until the model converges.

[0097] In some implementations, the metric prediction system 162 compares one or more force features to identify one or more examples of interactions involving unwanted contact between a medical device and an anatomical structure during the interaction. In one illustrative example, a force feature represents the amount of average force exerted by the medical device when moving an anatomical structure, and the amount of other force features (e.g., force features associated with a surgeon moving a similar anatomical structure) can be compared to the amount of average force. If the difference in the amount of force meets a threshold amount, the metric prediction system 162 can determine that the other force feature is an outlier. In some implementations, as discussed below, the metric prediction system 162 determines the metric as described herein based at least on whether the other force feature is an outlier.

[0098] In some implementations, the measurement prediction system 162 determines force characteristics or groups of force characteristics of a surgeon. For example, the measurement prediction system 162 may determine the surgeon's force characteristics based at least on one or more force vectors associated with medical instruments involved in interactions with anatomical structures performed by the surgeon. The measurement prediction system 162 may determine force characteristics based at least on force vectors associated with similar interactions in multiple surgical procedures involving the surgeon. In some implementations, the measurement prediction system 162 may determine stacked force characteristics for the surgeon (e.g., based at least on a combination of one or more distinct force characteristics). In this example, the measurement prediction system 162 may determine one or more stacked force characteristics based at least on one or more force characteristics involved in interactions between medical instruments and anatomical features performed by the surgeon. In some implementations, the measurement prediction system 162 may generate profiles of one or more surgeons based at least on one or more force characteristics or stacked force characteristics associated with one or more surgeons.

[0099] In some implementations, the measurement prediction system 162 determines the amount of force applied based at least on the contact between the medical device and the anatomical structure during the surgical procedure. For example, the measurement prediction system 162 may determine the amount of force applied based at least on the contact between the medical device and the anatomical structure during the surgical procedure, wherein the contact is associated with a three-dimensional orientation of the medical device contacting the anatomical structure. In these examples, the measurement prediction system 162 may determine the three-dimensional orientation of the medical device contacting the anatomical structure based at least on the type of interaction involved.

[0100] In some implementations, the amount of force applied represents the amount of force involved in the interaction between the medical device and the anatomical structure at a given point in time. In examples, the amount of force applied represents the amount of force involved in the interaction between the medical device and the anatomical structure over a period of time (e.g., the cumulative amount of force). In these examples, the amount of force applied over time can be determined at least based on the amount of force involved in one or more interactions described herein, added at one or more points in time associated with that time period.

[0101] In some implementations, the measurement prediction system 162 determines the amount of force applied based at least on one or more models (referred to herein as "models") trained using machine learning. For example, the measurement prediction system 162 may receive a data stream including video data 166, robotic system data 170, or medical device data 172, and the measurement prediction system 162 may provide data associated with the data stream to one or more models to cause the one or more models to generate one or more outputs. In some implementations, the one or more outputs may represent the amount of force applied to an anatomical structure during interaction with the device. For example, the one or more outputs may represent the amount of force applied to an anatomical structure during interaction with the device at one or more time points.

[0102] In some implementations, the data provided to one or more models to enable the models to output quantities representing forces may include raw data (e.g., unprocessed data). For example, one or more sensor signals included in video data 166, robotic system data 170, or medical device data 172 may be generated from one or more sensors as described herein without processing (e.g., updating) before being provided to one or more models.

[0103] In some implementations, one or more of the outputs of one or more models can be scored. For example, during a surgical procedure, the metric prediction system 162 can store the data stream in association with the output (e.g., in a data repository 132). The metric prediction system 162 can then be delivered via a client device (e.g., associated with an individual and with...). Figure 6 The data stream is played back using a computing device (identical or similar to a computing device 600). During playback, the client device may receive input from an individual indicating whether part or all of a surgical procedure involves appropriate or inappropriate handling of tissue by the surgeon. The data processing system 130 may then receive the input and store it in further association with the data stream. The measurement prediction system 162 may then provide an indication of whether the data stream and part or all of the surgical procedure are appropriate as input to a model to train the model to determine whether part or all of a future surgical procedure involves appropriate or inappropriate tissue handling. Where the model trained to provide such output is the same as the model used to determine the forces applied to anatomical structures, the measurement described herein may be determined at least based on the determination of whether the tissue handling was appropriate or inappropriate during the surgical procedure. In some embodiments, the indication of the measurement may be determined at least based on output indicating whether the handling of tissue during the surgical procedure was appropriate or inappropriate.

[0104] In some implementations, the measurement prediction system 162 updates the amount of force applied to the anatomical structure. For example, the measurement prediction system 162 may update the amount of force applied to the anatomical structure based at least on the type of medical device that interacts with the anatomical structure (e.g., contacts, grasps, cuts). In one illustrative example, the measurement prediction system 162 may update the amount of force applied by a stapler, where the force applied in the stapler is intended to cause tissue deformation (e.g., during the stapling process). In another illustrative example, the measurement prediction system 162 may update the amount of force applied by the stapler, where the applied force is not intended to cause tissue deformation (e.g., when the anatomical structure is moved using the non-cutting portion of the stapler). Thus, the overall force applied to the anatomical structure can be updated (e.g., increased or decreased) in cases where certain tissue interactions are anticipated. In some implementations, the measurement prediction system 162 may update the amount of applied force based at least on one or more predetermined amounts corresponding to a given interaction.

[0105] In some embodiments, the metric prediction system 162 determines a metric (referred to as a "metric") that instructs the execution of a medical procedure. For example, the metric prediction system 162 may determine the metric based at least on a comparison of the amount of force applied during interactions involving medical devices and anatomical structures with a force threshold. In examples, the force threshold may be the same as or similar to the thresholds described herein, such as threshold 140. In some embodiments, the metric prediction system 162 determines the metric based at least on updates made by the metric prediction system 162 to the amount of force, type of medical device, or anatomical structure involved in the medical procedure as described herein.

[0106] In some implementations, the measurement prediction system 162 determines the measurement based at least on a comparison of the amount of force applied with a force threshold established for the anatomical structure. For example, the measurement prediction system 162 may compare the amount of force applied with a force threshold, where the force threshold is associated with the degree to which the anatomical structure can be manipulated at one or more time points. In an illustrative example, the force threshold for an organ such as the liver may be relatively low compared to the force threshold for an organ such as the kidney. In this example, a higher force threshold corresponding to the kidney may be established compared to a portion of the liver that may not have similar sensitivity, to account for the sensitivity of the kidney portion (e.g., when manipulated, the adrenal glands can cause hormone release that affects other parts of the patient). In these examples, the measurement prediction system 162 may determine the measurement based at least on whether the amount of force applied meets the force threshold established for the anatomical structure. In examples, the measurement prediction system 162 may compare the amount of force applied with one or more force thresholds associated with corresponding short-term or long-term health outcomes. For example, each force threshold may be associated with a short-term or long-term health outcome indicating the time period of tissue recovery, the likelihood of tissue recovery (e.g., a tear will heal, etc.), etc. In this example, the measurement prediction system 162 may determine the measurement based at least on a comparison of the amount of force applied with one or more force thresholds associated with corresponding short-term or long-term health outcomes. In some implementations, the force thresholds may be updated based at least on information associated with one or more surgical procedures. For example, the measurement prediction system 162 may receive input from a user after a surgical procedure indicating one or more of the following: the type of surgical procedure, the short-term or long-term health outcome of the surgical procedure, etc., and the measurement prediction system 162 may cross-correlate the amount of force applied to the tissue with the short-term or long-term health outcome. In this example, the measurement prediction system 162 may update the force thresholds at least based on the correlation between the amount of force applied to the tissue and the short-term or long-term health outcome.

[0107] In some implementations, the measurement prediction system 162 determines the measurement based at least on a comparison of the amount of force applied with force thresholds established for interactions involving anatomical structures and types of medical devices. For example, as described above, one or more medical devices involved in a medical procedure may be associated with different force thresholds. In these examples, the measurement prediction system 162 may compare the amount of force applied with force thresholds established for interactions between anatomical structures and types of medical devices.

[0108] In some implementations, the measurement prediction system 162 determines the measurement based at least on a comparison of the amount of the applied force with a force threshold established for interactions involving the orientation of the anatomical structure and the medical device. In an example, the measurement prediction system 162 may compare the amount of the applied force with a force threshold established for interactions involving the anatomical structure and the medical device when the device is in a particular orientation relative to the anatomical structure. In an illustrative example, the measurement prediction system 162 may determine the measurement based at least on the applied force and at least on whether the medical device is oriented in a first orientation (e.g., an operational orientation) or a second orientation (e.g., a non-operational orientation). In the first orientation, the medical device may be configured to perform a first function involving the application of force (e.g., clamping or grasping the anatomical structure). In the second orientation, the medical device may be configured to perform one or more second functions (e.g., pushing the anatomical structure to the side), which involve applying different forces. In an example, the measurement prediction system 162 may determine the amount of the applied force and compare the applied force with a force threshold established for a given interaction.

[0109] In some implementations, the measurement prediction system 162 determines the measurement based at least on a comparison of the amount of applied force with a force threshold established for interactions involving anatomical structures and manipulations performed using a medical device. In an example, the measurement prediction system 162 may compare the amount of applied force with a force threshold established for interactions involving anatomical structures and manipulations performed using a medical device. In an illustrative example, the measurement prediction system 162 may determine the measurement based at least on a first manipulation involving the application of force (e.g., clamping or grasping an anatomical structure) during a first manipulation.

[0110] The metric prediction system 162 can determine the metric based at least on a comparison of the amount of force applied with a force threshold established for the amount of time a patient needs to recover relative to an anatomical structure. For example, when a medical device applies a force to an anatomical structure during a medical procedure, that force can be compared to the amount of time corresponding to different stages of the patient's recovery. In one illustrative example, when a force is applied to an anatomical structure and that force meets a first threshold associated with the patient's recovery on a given day (e.g., a force applied during an outpatient medical procedure), the metric prediction system 162 can determine the amount of time (e.g., minutes, hours) required for the patient to recover before leaving the medical facility where surgery was performed, resuming light activity, resuming normal activity, etc. In another illustrative example, when a force is applied to an anatomical structure and that force meets a second threshold associated with the patient's recovery over multiple days (e.g., a force applied during a medical procedure requiring postoperative monitoring), the metric prediction system 162 can determine the amount of time (e.g., minutes, hours, days) required for the patient to recover before leaving the medical facility, resuming light activity, resuming normal activity, etc.

[0111] The measurement prediction system 162 can determine an indication of a measurement. For example, the measurement prediction system 162 can determine an indication of a measurement that directs the execution of a medical procedure, wherein the indication represents the amount of force applied to an anatomical structure. In an example, the indication may represent the amount of force applied to the anatomical structure at a specific point in time (e.g., an instantaneous force). In some examples, the indication may represent the amount of force applied to the anatomical structure over a period of time (e.g., a cumulative force).

[0112] The measurement prediction system 162 can determine an indication of the measurement, wherein the indication represents the three-dimensional orientation of the medical device in contact with the anatomical structure. In an example, the indication may represent the orientation of the medical device in contact with the anatomical structure at a certain point in time. In some embodiments, the orientation of the medical device in contact with the anatomical structure may be represented by two-dimensional or three-dimensional indicators (e.g., lines, arrows, and / or the like).

[0113] The measurement prediction system 162 can provide measurement indications via a robotic system. For example, the measurement prediction system 162 can provide measurement indications at least in part based on data generated by the measurement prediction system 162 and associated with a user interface. In such examples, the measurement prediction system 162 can generate the user interface at least in part based on the measurement indications. In examples, the measurement prediction system 162 can further generate data associated with a user interface configured to cause a display device (e.g., the display device of the user input system 168) to provide output representing the measurement indications. In some embodiments, the user interface may also include images representing medical devices and anatomical structures within the field of view of the imaging device of the sensing system 164.

[0114] In some implementations, the measurement prediction system 162 may provide an indication of a measurement based at least in part on data generated by the measurement prediction system 162 that is associated with tactile feedback. For example, the measurement prediction system 162 may provide an indication of a measurement based at least in part on data generated by the measurement prediction system 162 that is associated with tactile feedback corresponding to a measurement instructing the execution of a medical procedure. In one illustrative example, as the amount of force applied to an anatomical structure increases, the measurement prediction system 162 may compare the amount of force with a force threshold established for the type of anatomical structure to determine a measurement instructing the execution of a medical procedure. In this illustrative example, the measurement prediction system 162 may determine the measurement as the difference between the applied force and the force threshold decreases, and generate data associated with tactile feedback based at least on the change in the difference between the applied force and the force threshold. The data associated with the tactile feedback may enable the surgeon to engage controls (e.g., with...) during the medical procedure. Figure 5 The user control system 510 (controls that are the same as or similar to those controls) vibrates at different intensities. This allows the surgeon to receive feedback indicating the measurements described herein.

[0115] In some implementations, the measurement prediction system 162 generates the user interface based at least in part on data streams and measurement indications associated with a medical procedure. For example, the measurement prediction system 162 may generate the user interface based at least in part on images representing medical instruments or anatomical structures within the field of view of an imaging device. In one example, the measurement prediction system 162 may generate the user interface based at least in part on images representing medical instruments or anatomical structures within the field of view of an imaging device, wherein the imaging device is included in a sensing system 164. In this example, images may be captured at one or more time points (sometimes referred to as time points), corresponding to one or more time points at which the measurement prediction system 162 determines the measurement. In some implementations, the measurement prediction system 162 generates the user interface based at least in part on images and measurement indications. For example, the measurement prediction system 162 may generate the user interface such that the measurement indications are included in the user interface.

[0116] In some implementations, the indication includes a color-coded or binary indicator. For example, the indicator may be associated with an area of ​​the user interface colored in a color that represents the amount of force applied to the anatomical structure during interaction. In one example, when the amount of force meets a force limit (e.g., within a force limit), the indicator may be associated with one or more areas of the user interface colored in different colors. In another example, when the amount of force applied to the anatomical structure does not meet a force limit, the indicator may be associated with one or more areas of the user interface colored in different colors. In some implementations, when the amount of force applied to the anatomical structure approaches a force limit, the indicator may be associated with an area of ​​the user interface colored in another color (e.g., a third color).

[0117] In some implementations, the measurement indication includes a numerical representation of the amount of force applied during the interaction between the medical device and the anatomical structure. For example, when a surgeon brings the medical device into contact with the anatomical structure, or when the medical device moves while in contact with the anatomical structure, the indication may include the amount of force involved in contacting the anatomical structure.

[0118] In some embodiments, the measurement indication includes a numerical representation of the proportion or speed of movement of one or more medical devices. For example, when a surgeon moves a medical device toward an anatomical structure, or when the medical device moves while in contact with an anatomical structure, the indication may include the speed of at least a portion of the medical device (e.g., in units such as cm / s, mm / s, etc.). In some embodiments, the measurement prediction system 162 determines the speed based at least on the relative motion of at least a portion of the medical device relative to the anatomical structure or the patient.

[0119] In some implementations, the measurement prediction system 162 provides one or more images associated with a data stream corresponding to an anatomical structure, to be displayed via a user interface. For example, the measurement prediction system 162 may generate data associated with the user interface, configured to cause a display device to provide an output representing the user interface, wherein the user interface at least partially represents one or more images associated with the data stream corresponding to an anatomical structure. As the medical procedure continues and the medical device interacts with the anatomical structure, the measurement prediction system 162 may update the user interface at least in part based on updates to the measurements described herein. For example, the measurement prediction system 162 may update the user interface by updating one or more pixels of one or more images at least in part based on measurements during the execution of the procedure.

[0120] In some implementations, the metric prediction system 162 determines at least one region associated with at least one overlay. For example, the metric prediction system 162 may determine at least one region associated with at least one overlay, wherein the at least one region corresponds to at least a portion of an anatomical structure represented by one or more images. In this example, the overlay may be configured to cause the user interface to update the representation of at least one region when displayed on a display. In some implementations, the metric prediction system 162 then updates one or more pixels associated with the user interface (e.g., one or more pixels of an image associated with the user interface) at least in part based on the at least one overlay. In the example, the metric prediction system 162 may construct the overlay at least in part based on at least one region and the metrics described herein. In one illustrative example, the metric prediction system 162 then updates one or more pixels associated with the user interface by coloring one or more pixels with one or more shadows or one or more colors, at least based on the amount of force involved in the interaction between the medical device and the anatomical structure. In another illustrative example, the metric prediction system 162 then updates multiple pixels associated with the user interface by coloring multiple pixels according to a segmentation mask (as described above) corresponding to the anatomical structure involved in the interaction and one or more metrics corresponding to the interaction as described herein. In one illustrative example, when approaching force limits during a surgical procedure and the probability of tissue damage to one or more anatomical structures increases, the metric prediction system 162 can update a segmentation mask (e.g., by updating the color of part or all of the segmentation mask) to indicate an increased likelihood of tissue damage if further interaction occurs between the medical device and the anatomical structure. Thus, the metric prediction system 162 can generate a user interface, for example, one that color-codes a particular anatomical structure or the like, at least based on the amount of force involved in the interaction between the medical device and the anatomical structure. In some embodiments, the overlay may be associated with one or more colors or shades that represent the metrics described herein, as they correspond to the anatomical structures involved in the medical procedure.

[0121] In some implementations, the metric prediction system 162 constructs a heatmap. For example, the metric prediction system 162 may construct a heatmap that includes a visual representation of metrics overlaid on the anatomical structures involved in the interaction. In an example, the metric prediction system 162 may construct the heatmap based at least in part on at least one region and the metrics described herein. In some implementations, the heatmap may include one or more regions of shaded shadows or colors. The heatmap may include one or more gradient regions of shadows or colors. In an illustrative example, the heatmap may be a gradient of color (e.g., red) corresponding to a metric representing an instantaneous or cumulative force applied to one or more parts of an anatomical structure during a medical procedure. In some implementations, the metric prediction system 162 then updates one or more pixels (e.g., an image associated with video data 166) based at least in part on at least one overlay or heatmap. The metric prediction system 162 may continuously update the user interface to indicate to the surgeon the amount of force applied to a specific region of the anatomical structure during a surgical procedure, thereby improving the surgeon's awareness of the context associated with a given anatomical structure. In the illustrative example, where the metric to be performed represents cumulative force (e.g., force applied to a given anatomical structure over a period of time), the metric prediction system 162 may construct a heatmap based at least in part on the cumulative force of at least one region and associated with (e.g., corresponding to) one or more parts of the anatomical structure.

[0122] In some implementations, the metric prediction system 162 generates a visual representation of anatomical structures involved in a medical procedure, based at least on the shape and orientation of the anatomical structure relative to the patient's body. For example, the metric prediction system 162 may generate the visual representation of the anatomical structure involved in the medical procedure based at least on a data stream generated during the medical procedure. In this example, the metric prediction system 162 may update the visual representation of the anatomical structure based at least on the metrics described herein. For example, the metric prediction system 162 may update the visual representation of the anatomical structure by updating one or more intensity or color values ​​corresponding to pixels representing the anatomical structure, at least based on the metrics.

[0123] In some implementations, the measurement prediction system 162 determines at least one first region where the amount of applied force satisfies a first threshold; and the measurement prediction system 162 determines at least one second region where the amount of applied force satisfies a second threshold. For example, in the case where multiple anatomical structures are located within the field of view of the imaging device of the sensing system 164, at least one first region may correspond to a first anatomical structure, and at least one second region may correspond to a second anatomical structure. The measurement prediction system 162 can then update one or more pixels of the image generated by the imaging device based on at least one of the at least one first region and at least one of the at least one second region. In an illustrative example, a pixel associated with a first region can be updated by coloring the pixel using a first color or a first shading; and a pixel associated with a second region can be updated by coloring the pixel using a second color or a second shading based at least on a measurement associated with the anatomical structure.

[0124] In some implementations, the measurement prediction system 162 may index and store data associated with the interaction between the medical procedure and the robotic system (e.g., the medical device of the robotic system) and the patient's anatomy. For example, the measurement prediction system 162 may index and store data associated with the amount of instantaneous or cumulative force applied to the anatomical structures involved in the medical procedure. The measurement prediction system 162 may then determine the expected amount of time for patient recovery based at least on the instantaneous or cumulative force applied to the anatomical structures. The measurement prediction system 162 may then determine the expected amount of time for patient recovery based at least on one or more aspects of the interaction between the medical device and the anatomical structures during the medical procedure.

[0125] In some implementations, the measurement prediction system 162 generates reports. For example, the measurement prediction system 162 may generate reports that include measurements determined during a medical procedure. In one example, the measurement prediction system 162 may generate reports that include measurements determined for multiple medical procedures. In another example, the measurement prediction system 162 may generate reports that include measurements determined for multiple medical procedures involving a particular patient. In yet another example, the measurement prediction system 162 may generate reports that include measurements determined for multiple medical procedures involving a particular surgeon. In this example, the measurement prediction system 162 may provide reports in real time (e.g., via a user interface described herein) or upon request after the performance of a surgical procedure.

[0126] In some implementations, the measurement prediction system 162 generates a report indicating the correlation between one or more force features or one or more stacked force features, and the probability of an outcome associated with a type of medical procedure. For example, the measurement prediction system 162 may receive feedback corresponding to one or more medical procedures, indicating whether the procedure was successful or unsuccessful (e.g., whether tissue was properly treated or improperly treated, whether the patient recovered as expected or not, etc.). The measurement prediction system 162 can then cross-correlate the feedback with one or more force features or stacked force features involved in the medical procedure. In some implementations, the measurement prediction system 162 may determine the probability of future outcomes (e.g., whether tissue was properly treated or improperly treated, whether the patient will recover as expected or not, etc.) based at least on the correlation between the feedback and one or more force features or stacked force features involved in similar medical procedures.

[0127] In some implementations, the measurement prediction system 162 may receive data associated with patient or surgeon feedback. For example, the patient or surgeon may provide feedback indicating how long their recovery process took, whether the patient experienced discomfort, the degree of discomfort, the presence of observable tissue damage, the amount of time required for damaged tissue to heal, or similar information. The measurement prediction system 162 can then cross-correlate the patient feedback with a measurement, which is determined by combining medical procedures (e.g., the amount of force applied to anatomical structures) with the interaction between the robotic system and the patient's anatomy, and update one or more force limits for the patient or other patients described herein.

[0128] Robot system data 170 includes or indicates robot system events corresponding to the state or activity of attributes or aspects of the robot system. Robot system data 170 may be generated by the robot system during its normal operation (e.g., in the form of a robot system log). Robot system data is determined at least based on user input received by the robot system or sensor data from sensors on the robot system, via user input system 168. The robot system may include one or more sensors (e.g., cameras, infrared sensors, ultrasonic sensors, etc.), actuators, interfaces, and consoles that can output information for detecting such system events.

[0129] Figure 2 This is a flowchart illustrating an example method 200 for determining the amount of force applied by an instrument during remote operation, according to some embodiments. Method 200 can be... Figure 1A , Figure 1B , Figure 3 , Figure 5 and Figure 6The one or more systems, devices, or components described herein perform, including, for example Figure 1B The measurement prediction system 162.

[0130] In operation 210, a data stream of a medical procedure performed by a robotic medical system is received. For example, a measurement prediction system (e.g., measurement prediction system 162) may receive the data stream of a medical procedure performed by the robotic medical system.

[0131] In operation 220, the type of anatomical structure to which the medical procedure is performed is identified. For example, a metric prediction system (e.g., metric prediction system 162) can use a data stream and one or more models trained with machine learning to identify the type of anatomical structure to which the medical procedure is performed.

[0132] In operation 230, the amount of force applied to the anatomical structure is determined. For example, a measurement prediction system (e.g., measurement prediction system 162) can determine the amount of force applied to the anatomical structure.

[0133] In operation 240, a metric indicating the execution of a medical procedure is determined. For example, a metric prediction system (e.g., metric prediction system 162) can determine the metric indicating the execution of a medical procedure.

[0134] In operation 250, an indication of a metric is provided. For example, a metric prediction system (such as metric prediction system 162) can provide a metric that indicates the execution of a medical procedure.

[0135] Figure 3 This is an image of an example user interface 300 according to some embodiments. As shown, the user interface 400 displays four anatomical structures 302, 304, 306, 308, and other anatomical structures. The four anatomical structures 302, 304, 306, and 308 are all shown with overlays associated with different colors (e.g., a first color, a second color, a third color, and a fourth color, respectively). The user interface 300 also includes labels 310 corresponding to points in time during the execution of a medical procedure.

[0136] Figure 4 This is a diagram illustrating exemplary force limits according to some implementation methods. As shown, the interaction between the medical device and the anatomical structure can be associated with an interaction type, labeled along the X-axis as "anatomy," "drive needle," "manipulation" (e.g., movement), "retraction," and "knot suture." Each interaction type can also be associated with one or more sub-limits, which correspond to specific aspects of each interaction type. As shown, the force limit can be between 0 Newtons and 14 Newtons.

[0137] Figure 5This is a schematic diagram of a medical environment according to some embodiments. Medical environment 500 may refer to or include a surgical environment or surgical system. Medical environment 500 may include a robotic medical system 524, a user control system 510, and an auxiliary system 515 that are communicatively connected to each other. Visualization tool 520 may be connected to auxiliary system 515, which in turn may be connected to robotic medical system 524. Therefore, when visualization tool 520 is connected to auxiliary system 515 and auxiliary system is connected to robotic medical system 524, visualization tool can be considered connected to robotic medical system. In some embodiments, visualization tool 520 may be directly connected to robotic medical system 524. Measurement prediction system 162 may be connected to user control system 510, which in turn may be connected to robotic medical system 524. Measurement prediction system 162 may be directly connected to robotic medical system 524. Therefore, when measurement prediction system 162 is connected to user control system 510 and measurement prediction system 160 is connected to robotic medical system 524, visualization tool can be considered connected to robotic medical system.

[0138] Medical environment 500 can be used to perform computer-aided medical procedures on patient 525. In some embodiments, the surgical team may include surgeon 530A and other medical personnel 530B-530D, such as medical assistants, nurses, and anesthesiologists, as well as other suitable team members who may assist in surgical procedures or medical sessions. A medical session may include surgical procedures performed on patient 525, as well as any preoperative (e.g., which may include setting up medical environment 500, including preparation for procedures for patient 525) and postoperative (e.g., which may include patient cleanup or post-treatment care), or other processes during the medical session. Although described in the context of surgical procedures, medical environment 500 may be implemented in non-surgical procedures or other types of medical procedures or diagnoses that may benefit from the accuracy and convenience of a surgical system.

[0139] The robotic medical system 524 may include multiple manipulator arms 535A-535D, and multiple medical devices (e.g., the devices described herein) may be coupled to or mounted to or supported by the manipulator arms. In some embodiments, the multiple manipulator arms 535A-535D may include one or more links. Each medical device may be any suitable surgical instrument (e.g., a tool with tissue interaction capabilities), imaging device (e.g., an endoscope, ultrasound tool, etc.), sensing instrument (e.g., a force-sensitive surgical instrument), diagnostic instrument, or other suitable instrument that can be used to perform computer-assisted surgical procedures on a patient 525 (e.g., by being at least partially inserted into and manipulated to perform computer-assisted surgical procedures on the patient). Although the robotic medical system 524 is shown as including four manipulator arms (e.g., manipulator arms 535A-535D), in other embodiments, the robotic medical system may include more or fewer than four manipulator arms. Furthermore, not all manipulator arms may have a medical device mounted thereon for all the time of the medical session. In addition, in some embodiments, a medical device mounted on a manipulator arm may be suitably replaced by another medical device.

[0140] One or more of the manipulator arms 535A-535D, or medical devices attached to the manipulator arms, may include one or more displacement transducers, orientation sensors, position sensors, or other types of sensors and devices to measure parameters or generate kinematic information. One or more components of the medical environment 500 may be configured to use the measured parameters or kinematic information to track (e.g., determine the posture of the medical device) or control the medical device and anything connected to the medical device or manipulator arms 535A-535D.

[0141] Surgeon 530A may use user control system 510 to control (e.g., move) one or more manipulator arms 535A-535D or medical instruments attached to the manipulator arms. To facilitate control of the manipulator arms 535A-535D and tracking of the progress of a medical session, user control system 510 may include a display that can provide surgeon 530A with images (e.g., high-definition 3D images) of the surgical site associated with patient 525, captured by a medical instrument mounted on one of the manipulator arms 535A-535D. User control system 510 may include a stereoscopic viewer with two or more displays in which surgeon 530A can view stereoscopic images of the surgical site associated with patient 525 and generated by a stereoscopic imaging system. In some embodiments, user control system 510 may also receive images from assistive system 515 and visualization tool 520.

[0142] Surgeon 530A can use the images displayed on user control system 510 to perform one or more procedures using one or more medical devices attached to manipulator arms 535A-535D. To facilitate control of manipulator arms 535A-535D or the medical devices mounted thereon, user control system 510 may include a set of controls. These controls can be manipulated by surgeon 530A to control the movement of manipulator arms 535A-535D or the medical devices mounted thereon. The controls can be configured to detect various hand, wrist, and finger movements of surgeon 530A to allow the surgeon to visually perform procedures on patient 525 using one or more medical devices mounted on manipulator arms 535A-535D.

[0143] The auxiliary system 515 may include one or more computer systems configured to perform processing operations within the medical environment 500 (e.g., with...). Figure 6 The computing device 600 is the same as or similar to the computing device 524. For example, one or more computer systems can control or coordinate operations performed by various other components of the medical environment 500 (e.g., robotic medical system 524, user control system 510). The computer system included in the user control system 510 can transmit instructions to the robotic medical system 524 via one or more computing devices of the auxiliary system 515. The auxiliary system 515 can receive and process image data representing images captured by one or more imaging devices (e.g., medical instruments) attached to the robotic medical system 524, as well as other data stream sources received from visualization tools. For example, one or more image capture devices may be located within the medical environment 500. These image capture devices can capture images from various viewpoints within the medical environment 500. These images (e.g., video streams) can be transmitted to the visualization tool 520, which can then transmit these images as a single combined data stream to the auxiliary system 515. The auxiliary system 515 can then transmit a single video stream (including any data streams received from the robotic medical system 524 or its medical instruments) to be displayed on the monitor of the user control system 510.

[0144] In some implementations, the assistive system 515 may be configured to present visual content (e.g., a single combined data stream) to other team members (e.g., medical personnel 530B-530D) who do not have access to the user control system 510. Therefore, the assistive system 515 may include a display 640 configured to display one or more user interfaces, such as images of surgical sites, information associated with the patient 525 or surgical procedures, or any other visual content (e.g., a single combined data stream). In some implementations, the display 540 may be a touchscreen display or include other features that allow medical personnel 530B-530D to interact with the assistive system 515.

[0145] The robotic medical system 524, user control system 510, and auxiliary system 515 can be communicatively connected to each other in any suitable manner. For example, in some embodiments, the robotic medical system 524, user control system 510, and auxiliary system 515 can be communicatively connected via a control line 545, which can represent any wired or wireless communication link that can serve a particular implementation. Therefore, each of the robotic medical system 524, user control system 510, and auxiliary system 515 can include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0146] It should be understood that the medical environment 500 may include other or additional components or elements that are necessary or considered desirable for a medical session using a surgical system.

[0147] Figure 6 This is a block diagram depicting the architecture of a computing device 600, which includes elements that can be used to implement the systems and methods described and illustrated herein, including... Figure 1A-Figure 1B , Figure 3 or Figure 5 The various aspects of the system described in the text and Figure 2 The methods described herein. For example, the measurement prediction system 162, sensing system 164, user input system 168, and devices described with respect to medical environment 500 may include one or more components or functions of computing device 600. Computing device 600 may be any computing device as used herein and may include or be used to implement a data processing system or components thereof. Computing device 600 includes at least one bus 605 or other communication components or interfaces for communicating information between various elements of a computer system. The computer system also includes at least one processor 610 or processing circuitry coupled to bus 605 for processing information. Computing device 600 also includes at least one main memory 615, such as random access memory (RAM) or other dynamic storage device, coupled to bus 605 for storing information and instructions to be executed by processor 610. Main memory 615 may be used to store information during the execution of instructions by processor 610. Computing device 600 may also include at least one read-only memory (ROM) 620 or other static storage device coupled to bus 605 for storing static information and instructions for processor 610. Storage devices 625, such as solid-state devices, disks, or optical discs, can be connected to bus 605 to persistently store information and instructions.

[0148] Computing device 600 can be connected to display 630, such as a liquid crystal display or an active matrix display, via bus 605 for displaying information. Input device 635, such as a keyboard or voice interface, can be connected to bus 605 for transmitting information and commands to processor 610. Input device 635 may include a touchscreen display (e.g., display 630). Input device 635 may include sensors for detecting gestures. Input device 635 may also include cursor controls, such as a mouse, trackball, or cursor arrow keys, for transmitting directional information and command selection to processor 610 and for controlling cursor movement on display 630.

[0149] The processes, systems, and methods described herein can be implemented by computing device 600 in response to processor 610 executing an arrangement of instructions contained in main memory 615. Such instructions may be read into main memory 615 from another computer-readable medium, such as storage device 625. Execution of the arrangement of instructions contained in main memory 615 causes computing device 600 to perform the illustrative processes described herein. One or more processors in a multiprocessor arrangement may also be employed to execute the instructions contained in main memory 615. Hardwired circuitry may be used in place of or in combination with software instructions, as well as the systems and methods described herein. The systems and methods described herein are not limited to any particular combination of hardware circuitry and software.

[0150] Processor 610 can execute one or more instructions associated with system 100. Processor 610 may include an electronic processor, integrated circuit, etc., including one or more of digital logic, analog logic, digital sensors, analog sensors, communication buses, volatile memory, non-volatile memory, and the like. Processor 610 may include, but is not limited to, at least one microcontroller unit (MCU), microprocessor unit (MPU), central processing unit (CPU), graphics processing unit (GPU), physical processing unit (PPU), embedded controller (EC), or the like. Processor 610 may include or be associated with main memory 615, which is operable to store or store components for operating system 100 and operating components operatively coupled to processor 610, one or more non-transient computer-readable instructions. For example, one or more instructions may include at least one of firmware, software, hardware, operating system, or embedded operating system. Processor 610 or system 100 typically includes at least one communication bus controller to enable communication between the system processor and other components of system 100.

[0151] Main memory 615 may include one or more hardware storage devices for storing binary data, digital data, or the like. Main memory 615 may include one or more electrical components, electronic components, programmable electronic components, reprogrammable electronic components, integrated circuits, semiconductor devices, flip-flops, arithmetic units, or the like. Main memory 615 may include at least one of non-volatile memory devices, solid-state memory devices, flash memory devices, NAND memory devices, volatile memory devices, etc. Main memory 615 may include one or more addressable memory regions disposed on one or more physical memory arrays.

[0152] Despite Figure 6 An example computing system is described herein, but the subject matter including the operations described herein can be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in a combination of one or more of them.

[0153] The topics described herein sometimes illustrate different components contained within or connected to different other components. It should be understood that the architectures described in this way are illustrative, and in fact, many other architectures can be implemented to achieve the same functionality. Conceptually, any arrangement of components that achieve the same functionality is effectively “associated” to achieve the desired function. Therefore, any two components combined in this document to achieve a particular function can be considered “associated” with each other to achieve the desired function, regardless of the architecture or intermediate components. Similarly, any two such associated components can also be considered “operably connected” or “operably linked” with each other to achieve the desired function, and any two components that can be so associated can also be considered “operably linkable” with each other to achieve the desired function. Specific examples of being operablely linkable include, but are not limited to, physically matchable or physically interactive components, or wirelessly interactive or wirelessly interactive components, or logically interactive or logically interactive components.

[0154] Regarding the use of plural or singular terms in this document, those skilled in the art can translate them from plural to singular or vice versa as needed by the context or application. For clarity, various singular / plural arrangements may be explicitly described herein.

[0155] Those skilled in the art will understand that, in general, the terminology used herein, and especially the terminology used in the appended claims (e.g., the body of the appended claims), is typically “open” terminology (e.g., the term “comprising” should be interpreted as “including but not limited to”, the term “having” should be interpreted as “at least having”, the term “including” should be interpreted as “including but not limited to”, etc.).

[0156] Although the accompanying drawings and descriptions may show a specific order of method steps, the order of these steps may differ from the order depicted and described unless otherwise stated above. Furthermore, unless otherwise specified above, two or more steps may be performed simultaneously or partially simultaneously. For example, such variations may depend on the chosen software and hardware system and the designer's choices. All such variations are within the scope of this disclosure. Similarly, the software implementation of the described methods can be implemented using standard programming techniques based on rule-based logic and other logic to perform various connection steps, processing steps, comparison steps, and decision steps.

[0157] Those skilled in the art will further understand that if the intent of an introduced claim statement is specific, then such intent will be explicitly stated in the claim; if no such statement is present, then such intent does not exist. For example, to aid understanding, the appended claims may include the introductory phrases “at least one” and “one or more” to introduce the claim statement. However, the use of these phrases should not be construed as implying that introducing a claim statement with the indefinite article “a” or “an” limits any particular claim containing such an introduced claim statement to an invention containing only one such statement, even if the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an” (e.g., “a” or “an” should generally be interpreted as “at least one” or “one or more”); the same applies to the use of definite articles used to introduce claim statements. Furthermore, even when a specific number of introduced claim statements is explicitly cited, those skilled in the art will recognize that such a statement should generally be interpreted as meaning at least the number cited (e.g., the simple statement “two statements” without other modifiers generally means at least two statements, or two or more statements).

[0158] Furthermore, when using conventions such as "at least one of A, B, and C," such expressions are generally interpreted in the sense that a person skilled in the art would understand the convention to be (e.g., "a system having at least one of A, B, and C" includes, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, or A, B, and C together, etc.). When using conventions such as "at least one of A, B, or C," such expressions are generally interpreted in the sense that a person skilled in the art would understand the convention to be (e.g., "a system having at least one of A, B, or C" includes, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, or A, B, and C together, etc.). A person skilled in the art will further understand that, whether in the specification, claims, or drawings, almost any extractive word or phrase presenting two or more alternative terms should be understood to include the possibility of including one term, any term, or both terms. For example, the phrase "A or B" will be understood to include the possibility of including "A" or "B" or "A and B."

[0159] In addition, unless otherwise stated, the use of words such as “approximate,” “about,” “around,” and “substantially” means plus or minus 10%.

[0160] For purposes of illustration and description, the above description of illustrative embodiments has been given. It is not intended to be exhaustive or to limit the precise forms disclosed, and modifications and variations can be made based on the foregoing teachings, or may be derived from practice of the disclosed embodiments. The scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A system comprising: One or more processors, coupled to memory, are used for: Receives data streams from medical procedures performed by a robotic medical system; The medical procedure uses the data stream and one or more models trained with machine learning to identify the type of anatomical structure, and is performed on the anatomical structure. The data stream is used to determine the amount of force applied to the anatomical structure by one or more surgical instruments; A metric instructing the execution of the medical procedure is determined, at least based on a comparison of the amount of force applied to the anatomical structure with a force threshold established for the type of the anatomical structure. and Instructions are provided for the metrics used to control the execution of the medical procedure.

2. The system of claim 1, wherein the one or more processors are configured to: Receive the data stream of the medical procedure, the data stream including data associated with force vectors, the force vectors representing the direction and magnitude of force interactions between parts of instruments and anatomical structures involved in the medical procedure, the force vectors being captured at multiple time points; and The amount of force applied to the anatomical structure is determined at least based on the force vector.

3. The system of claim 1, wherein the one or more processors are configured to: Receive the data stream of the medical procedure, the data stream including data associated with a gripping force vector, the gripping force vector representing the direction and magnitude of force interaction between a portion of the instrument and the anatomical structures gripped by the portion of the instrument during the medical procedure, the gripping force vector being captured at multiple time points; and The amount of force applied by the instrument to the corresponding part of the anatomical structure is determined at least based on the gripping force vector.

4. The system of claim 1, wherein the one or more processors are configured to: Determine the amount of force applied to the anatomical structure, the amount of force representing the interaction between the instruments involved in the medical procedure and the anatomical structure at a certain point in time.

5. The system of claim 1, wherein the one or more processors are configured to: Determine the amount of cumulative force applied to the anatomical structure over a period of time, the amount of cumulative force being determined at least based on the amount of force applied to the anatomical structure by the instruments involved in the medical procedure at one or more time points within the period of time; and The amount of force applied to the anatomical structure is determined at least based on the amount of the cumulative force applied to the anatomical structure.

6. The system of claim 1, wherein the one or more processors are configured to: The amount of the applied force is determined based at least on the contact between at least one instrument and the anatomical structure; and Determine the three-dimensional orientation associated with the contact between the at least one instrument and the anatomical structure.

7. The system of claim 1, wherein the one or more processors are configured to: The amount of force applied to the anatomical structure is determined at least based on one or more sensor signals generated by one or more torque sensors. The one or more torque sensors are configured to measure torque at one or more joints of one or more arms of a surgical robot, the one or more arms being supported by instruments that apply the amount of force to the anatomical structure.

8. The system of claim 7, wherein the one or more processors are configured to: The amount of force applied to the anatomical structure is updated at least based on the type of instrument associated with the instrument to which the amount of force is being applied; and The metric that indicates the execution of the medical procedure is determined, at least based on an update of the amount of force applied to the anatomical structure.

9. The system of claim 1, wherein the one or more processors are configured to: The metric is determined at least based on a comparison of the amount of force with a force threshold established for the type of anatomical structure, and one or more of the following: The types of devices involved in the medical procedure. The orientation of the instruments involved in the medical procedure. The manipulation associated with the timing of the amount of force applied to the anatomical structure, or The amount of time it takes for the patient to recover, in relation to the amount of force applied to the anatomical structure.

10. The system of claim 1, wherein the one or more processors are configured to: The user interface is generated based at least on the data flow of the medical procedure and the metrics instructing the execution of the medical procedure; and The display device displays the user interface, which occurs during or after a point in time when the medical procedure is performed by the robotic medical system.

11. The system of claim 10, wherein the one or more processors are configured to: Generate a visual representation of the anatomical structures involved in the medical procedure; and The visual representation of the anatomical structure is updated at least based on the metric instructing the execution of the medical procedure; and The display device displays the user interface, which includes at least a portion of the visual representation of the anatomical structure.

12. The system of claim 11, wherein the one or more processors are configured to: The visual representation of the anatomical structure is updated at least based on a heatmap associated with the anatomical structure, the heatmap including a visual representation of the measure indicating the execution of the medical procedure superimposed on the anatomical structure.

13. The system of claim 11, wherein the metric indicating execution represents a cumulative force, the system comprising the one or more processors for: The visual representation of the anatomical structure is updated at least based on a heatmap associated with the anatomical structure, the heatmap including a visual representation of the measure indicating the execution of the medical procedure superimposed on the anatomical structure over a period of time.

14. The system of claim 10, wherein the one or more processors are configured to: Determine the indication of the metric that directs the execution of the medical procedure; and The user interface is updated based on at least one or more indicators to include the one or more indicators at a location on the user interface.

15. The system of claim 14, wherein the one or more processors are configured to: The indication for determining the metric used to control the execution of the medical procedure, the indication representing the amount of force applied to the anatomical structure at the point in time when the force is applied to the anatomical structure.

16. The system of claim 14, wherein the one or more processors are configured to: The indication for determining the metric used to control the execution of the medical procedure, the indication representing the amount of force applied to the anatomical structure during the time period in which force is applied to the anatomical structure.

17. The system of claim 14, wherein the one or more processors are configured to: The indication is defined as the metric used to control the execution of the medical procedure, the indication representing the three-dimensional orientation of the instrument contacting the anatomical structure.

18. The system according to any one of claims 14 to 17, wherein the one or more processors are further configured to: The indication of the metric used to control the execution of the medical procedure is determined at least based on the metric that indicates the execution of the medical procedure.

19. A method comprising: One or more processors receive data streams from medical procedures executed by the robotic medical system; The medical procedure is performed on the anatomical structures by the one or more processors using the data stream and one or more models trained with machine learning to identify the types of anatomical structures. The amount of force applied to the anatomical structure is determined by the one or more processors using the data stream; A metric instructing the execution of the medical procedure is determined by the one or more processors and based at least on a comparison of the amount of force applied to the anatomical structure with a force threshold established for the type of the anatomical structure. and The one or more processors provide indications of the metrics used to control the execution of the medical procedure.

20. The method of claim 19, wherein the data stream includes data associated with one or more of the hyperspectral images or Raman hyperspectral images.

21. The method of claim 19, further comprising: The data stream of the medical procedure is received by the one or more processors, the data stream including data associated with force vectors, the force vectors representing the direction and magnitude of force interactions between instruments and anatomical structures involved in the medical procedure, the force vectors being captured at multiple time points; and The amount of force applied to the anatomical structure is determined by the one or more processors, at least based on the force vector.

22. The method of claim 19, further comprising: The amount of force applied to the anatomical structure is determined by the one or more processors, the amount of force representing the interaction between the instruments involved in the medical procedure and the anatomical structure at a certain point in time.

23. The method of claim 19, further comprising: The data associated with tactile feedback is generated by the one or more processors based at least on metrics instructing the execution of the medical procedure; and The one or more processors provide data associated with the haptic feedback so that a device associated with the user control system outputs the haptic feedback.

24. A non-transitory computer-readable medium storing processor-executable instructions, which, when executed by one or more processors, cause the one or more processors to: Receives data streams from medical procedures performed by a robotic medical system; The medical procedure uses the data stream and one or more models trained with machine learning to identify the type of anatomical structure, and is performed on the anatomical structure. The amount of force applied to the anatomical structure is determined using the data stream; A metric instructing the execution of the medical procedure is determined, at least based on a comparison of the amount of force applied to the anatomical structure with a force threshold established for the type of the anatomical structure. and Provide instructions for controlling the execution of the medical procedure using the metrics.

Images