Automatic generation of instrument life scores using multi-modal data streams

Abstract

A computer system may obtain medical procedure data representative of use of a first removable instrument. The medical procedure data may comprise a plurality of data streams, where a first data stream comprises data relating to use of the first removable instrument and a second data stream comprises image data. The computer system may input the plurality of data streams into a respective projection layer of a machine learning model. The projection layer for the first data stream includes an embedding layer and a transformer. The machine learning model is configured to output an instrument life metric and is trained using historical procedure data. The instrument life metric is indicative of a current instrument life score. The computing system may receive an instrument life metric as an output of the machine learning model and use the output instrument life metric to perform various intra-procedure or post-procedure operations.

Description

AUTOMATIC GENERATION OF INSTRUMENT EIFE SCORES USING MULTI- MODAE DATA STREAMSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of the filing date of provisional U.S. Patent Application No. 63 / 665,890 entitled “AUTOMATIC GENERATION OF INSTRUMENT LIFE SCORES USING MULTI-MODAL DATA STREAMS,” filed on June 28, 2024. The entire contents of the provisional application are hereby expressly incorporated herein by reference.FIELD

[0002] The present disclosure relates generally to computer-assisted systems and more particularly to training and utilizing artificial intelligence to automatically generate instrument life scores from multi-modal data streams connected with the procedure.BACKGROUND

[0003] Computer-assisted systems, including robotically assisted systems or robotic systems, may include one or more manipulators that can be operated with the assistance of an electronic controller (e.g., computer or control system) to move and control functions of one or more instruments coupled to the manipulators. A manipulator generally includes mechanical links connected by joints. An instrument is removably (or permanently) coupled to one of the links, typically a distal link of the plural links. In some embodiments, manipulator systems are used in conjunction with one or more auxiliary devices (e.g., a surgical bed, an insufflator, etc.).

[0004] Over the course of multiple uses, the removable variants of the instruments degrade until they are no longer suitable for use in procedures performed using the computer assisted systems. Specifically, the imposition of forces, application of energy, etc., on or from the removable instruments during procedures performed with the computer assisted systems uses up portions of finite lifespans for the removable instruments.

[0005] The current remaining life for removable instruments is tracked and monitored to prevent expired removable instruments (e.g., instruments with no remaining life) from being used in procedures and / or situations where removable instruments would expire during the procedure. However, previous methods for determining instrument life may not besufficiently accurate and tend to over- or underestimate the remaining life such as by under decrementing or over decrementing a current life counter. These inaccuracies can lead to wasting of an actually non-expired instrument, use of expired instruments, and various hassles and frustrations with having to abort or delay procedures in order to switch instruments.

[0006] Accordingly, there is a need for systems and methods that accurately determine the remaining life of removable instruments using reliable multi-modal data streams from a procedure that utilizes the removable instrument. In particular, there is a need for systems that can determine remaining life of removable instruments both post-procedurally and intra- procedurally.SUMMARY

[0007] In some aspects, the techniques described herein relate to a computer system including: one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the computer system to: obtain medical procedure data representative of use of a first removable instrument with one or more manipulators of a computer-assisted system during a medical procedure, wherein: the medical procedure data includes a plurality of data streams from one or more data sources, a first data stream of the plurality of data streams includes data relating to operation of the first removable instrument by the computer-assisted system during the medical procedure, and a second data stream of the plurality of data includes image data; input each of the plurality of data streams into a respective projection layer of a first machine learning model, wherein: the respective projection layer for the first data stream includes an embedding layer and a transformer, the first machine learning model is configured to output an instrument life metric based on outputs from the projection layers for the plurality of data streams, wherein the instrument life metric is indicative of a current instrument life score,; receive an instrument life metric for the first removable instrument as an output of the first machine learning model; and associate the current instrument life score with the first removable instrument based on the output instrument life metric.

[0008] In some aspects, the techniques described herein relate to a computer-implemented method including: obtaining medical procedure data representative of use of a first removable instrument with one or more manipulators of a computer-assisted system during a medical procedure, wherein: the medical procedure data includes a plurality of data streams from one or more data sources, a first data stream of the plurality of data streams includes data relatingto operation of the first removable instrument by the computer-assisted system during the medical procedure, and a second data stream of the plurality of data includes image data; inputting each of the plurality of data streams into a respective projection layer of a first machine learning model, wherein: the respective projection layer for the first data stream includes an embedding layer and a transformer, the first machine learning model is configured to output an instrument life metric based on outputs from the projection layers for the plurality of data streams, wherein the instrument life metric is indicative of a current instrument life score; receiving an instrument life metric for the first removable instrument as an output of the first machine learning model; and associating the current instrument life score with the first removable instrument based on the output instrument life metric.

[0009] In some aspects, the techniques described herein relate to a computer system including: one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the computer system to: detect an indication of a planned medical procedure that utilizes removable instruments of a particular type; obtain a set of current instrument life scores respectively associated with a set of removable instruments of the particular type utilized in the planned medical procedure, wherein: the set of current instrument life scores include at least one current instrument life score that was based on an output from a first machine learning model when input with a plurality of data streams of medical procedure data; the medical procedure data is representative of use of an associated removable instrument with one or more manipulators of a computer-assisted system during a past medical procedure; select a particular removable instrument from the set of removable instruments for use in the planned medical procedure based on the obtained set of current instrument life scores; and display a notification indicating that the particular removable instrument is to be used for the planned medical procedure.

[0010] In some aspects, the techniques described herein relate to a computer-implemented method including: detecting an indication of a planned medical procedure that utilizes removable instruments of a particular type; obtaining a set of current instrument life scores respectively associated with a set of removable instruments of the particular type utilized in the planned medical procedure, wherein: the set of current instrument life scores include at least one current instrument life score that was based on an output from a first machine learning model when input with a plurality of data streams of medical procedure data; the medical procedure data is representative of use of an associated removable instrument withone or more manipulators of a computer-assisted system during a past medical procedure; selecting a particular removable instrument from the set of removable instruments for use in the planned medical procedure based on the obtained set of current instrument life scores; and displaying a notification indicating that the particular removable instrument is to be used for the planned medical procedure.

[0011] In some aspects, the techniques described herein relate to a computer-assisted system including: one or more manipulators; one or more removable instruments coupled to the one or more manipulators; and a control system operably coupled to the one or more manipulators, wherein the control system is configured to: obtain a plurality of data streams from one or more data sources, wherein: a first data stream of the plurality of data streams includes data relating to operation of the of the one or more removable instruments by the computer-assisted system during the medical procedure, and a second data stream of the plurality of data includes image data; input each of the plurality of data streams into a respective projection layer of a first machine learning model, wherein: the respective projection layer for the first data stream includes an embedding layer and a transformer, the first machine learning model is configured to output an instrument life metric based on outputs from the projection layers for the plurality of data streams, wherein the instrument life metric is indicative of a current instrument life score; receive an instrument life metric for the one or more removable instruments as an output of the first machine learning model; and display an alert on a display of the control system based on the current instrument life score indicated by the output instrument life metric.

[0012] In some aspects, the techniques described herein relate to a computer-implemented method including: obtaining a plurality of data streams from one or more data sources, wherein: a first data stream of the plurality of data streams includes data relating to operation of one or more removable instruments by the computer-assisted system during the medical procedure, and a second data stream of the plurality of data includes image data; inputting each of the plurality of data streams into a respective projection layer of a first machine learning model, wherein: the respective projection layer for the first data stream includes an embedding layer and a transformer, the first machine learning model is configured to output an instrument life metric based on outputs from the projection layers for the plurality of data streams, wherein the instrument life metric is indicative of a current instrument life score; receiving an instrument life metric for the one or more removable instruments as an output ofthe first machine learning model; and displaying an alert on a display of a control system based on the current instrument life score indicated by the output instrument life metric.

[0013] In some aspects, the techniques described herein relate to a computer system including: one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the computer system to: obtain a current instrument life score for a particular removable instrument selected for use in a planned medical procedure, wherein: the current instrument life score is based on an output from a first machine learning model when input with a plurality of data streams of medical procedure data; and the medical procedure data is representative of use of the particular removable instrument with one or more manipulators of a computer-assisted system during a past medical procedure evaluate the suitability of the particular removable instrument for use in the planned medical procedure based upon the obtained current instrument life score; and display a notification indicating that the particular removable instrument is suitable or not suitable to be used for the planned medical procedure.

[0014] In some aspects, the techniques described herein relate to a computer-implemented method including: obtaining a current instrument life score for a particular removable instrument selected for use in a planned medical procedure, wherein: the current instrument life score is based on an output from a first machine learning model when input with a plurality of data streams of medical procedure data; and the medical procedure data is representative of use of the particular removable instrument with one or more manipulators of a computer-assisted system during a past medical procedure evaluating the suitability of the particular removable instrument for use in the planned medical procedure based upon the obtained current instrument life score; and displaying a notification indicating that the particular removable instrument is suitable or not suitable to be used for the planned medical procedure.

[0015] In some aspects, the techniques described herein relate to a computer-readable media storing instructions that, when executed by a control system of a computer-assisted system, causes the computer-assisted system to perform any of the methods described herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIGs. 1A and IB are diagrams for a computer-assisted system in accordance with one or more embodiments.

[0017] FIG. 1C is a schematic side view of an example instrument in accordance with one or more embodiments.

[0018] FIG. 2A is a block diagram of an artificial intelligence model in accordance with one or more embodiments.

[0019] FIG. 2B is a block diagram of a system of artificial intelligence models in accordance with one or more embodiments.

[0020] FIG. 2C is a block diagram of a system of artificial intelligence models in accordance with one or more embodiments.

[0021] FIG. 3 is a block diagram of an artificial intelligence model in accordance with one or more embodiments.

[0022] FIG. 4 is a flow diagram of a method in accordance with one or more embodiments.

[0023] FIG. 5 is a flow diagram of a method in accordance with one or more embodiments.

[0024] FIG. 6 is a flow diagram of a method in accordance with one or more embodiments.

[0025] FIG. 7 is a flow diagram of a method in accordance with one or more embodiments.

[0026] Examples of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, wherein showings therein are for purposes of illustrating examples of the present disclosure and not for purposes of limiting the same.DETAILED DESCRIPTION

[0027] In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.

[0028] Further, the terminology in this description is not intended to limit the invention. For example, spatially relative terms-such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like-may be used to describe the relation of one element orfeature to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass different positions (z.e., locations) and orientations (z.e., rotational placements) of the elements or their operation in addition to the position and orientation shown in the figures. For example, if the content of one of the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features. A device may be otherwise oriented and the spatially relative descriptors used herein interpreted accordingly. Likewise, descriptions of movement along and around various axes include various special element positions and orientations. In addition, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context indicates otherwise. Additionally, the terms “comprises”, “comprising”, “includes”, and the like specify the presence of stated features, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

[0029] Elements described in detail with reference to one embodiment, implementation, system, or module may, whenever practical, be included in other embodiments, implementations, systems, or modules in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or implementation non-functional, or unless two or more of the elements provide conflicting functions.

[0030] In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0031] This disclosure describes various devices, elements, and portions of computer- assisted systems and elements in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an element or a portion of an element (e.g., three degrees of translational freedom in a three-dimensional space, such as along Cartesian x-, y-, and z-coordinates). As used herein, the term “orientation” refers to therotational placement of an element or a portion of an element (e.g., three degrees of rotational freedom in three-dimensional space, such as about roll, pitch, and yaw axes, represented in angle-axis, rotation matrix, quaternion representation, and / or the like). As used herein, and for a device with a kinematic series, such as with a repositionable structure with a plurality of links coupled by one or more joints, the term “proximal” refers to a direction toward a base of the kinematic series, and “distal” refers to a direction away from the base along the kinematic series.

[0032] As used herein, the term “pose” refers to the multi-degree of freedom (DOF) spatial position and orientation of a coordinate system of interest attached to a rigid body. In general, a pose includes a pose variable for each of the DOFs in the pose. For example, a full 6-DOF pose for a rigid body in three-dimensional space would include 6 pose variables corresponding to the 3 positional DOFs (e.g., x, y, and z) and the 3 orientational DOFs (e.g., roll, pitch, and yaw). A 3-DOF position only pose would include only pose variables for the 3 positional DOFs. Similarly, a 3-DOF orientation only pose would include only pose variables for the 3 rotational DOFs. Further, a velocity of the pose captures the change in pose over time (e.g., a first derivative of the pose). For a full 6-DOF pose of a rigid body in three- dimensional space, the velocity would include 3 translational velocities and 3 rotational velocities. Poses with other numbers of DOFs would have a corresponding number of velocities translational and / or rotational velocities.

[0033] This disclosure occasionally refers to the disclosed techniques being applied to “patients” undergoing a “medical procedure.” It should be appreciated that these references are not intended to limit the application of the disclosed techniques to applied medicine contexts. For example, the described techniques can be applied to facilitate physician training, equipment testing and / or calibration, and / or other contexts. Accordingly, any reference to the term “patient” is done for ease of explanation and also envisions the application of the described techniques to a generic “subject.”

[0034] The word “task” is used herein to refer to a discrete portion of a procedure that may be autonomously, semi-autonomously, or manually implemented in furtherance of the procedure. For example, a task may be to move an endoscope to a particular position, to advance an instrument to a particular depth, to replace an instrument coupled to a manipulator, and so on. In some embodiments, a task is associated with component tasks to accomplish an overall goal. For example, a task to analyze a worksite may include component tasks related to moving an endoscope to view the worksite, advancing aninstrument to predetermined depth, and enabling a functionality supported by the instrument. These component tasks may also be referred to as “subtasks.”

[0035] As used herein, the term “instrument life score” is an identifier that indicates the remaining life span of a removable instrument. This identifier can include a percentage value, a numerical score such as between 0-100, and other similar indicators known in the art. As used herein, the term “instrument life metric” is a value indicative of an instrument life score. For example, the instrument life metric may include a value useable to calculate or otherwise determine an instrument life score as described herein.

[0036] Aspects of this disclosure are described in reference to computer-assisted systems, which can include devices that are teleoperated, externally manipulated, autonomous, semiautonomous, and / or the like. Further, aspects of this disclosure are described in terms of an implementation using a teleoperated surgical system, such as the da Vinci® Surgical System commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that inventive aspects disclosed herein may be embodied and implemented in various ways, including teleoperated and non-teleoperated, and medical and non-medical embodiments and implementations. Implementations on da Vinci® Surgical Systems are merely exemplary and are not to be considered as limiting the scope of the inventive aspects disclosed herein. For example, techniques described with reference to surgical instruments and surgical methods may be used in other contexts. Thus, the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperated systems. As further examples, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, and / or the like. Additional example applications include use for procedures on tissue removed from human or animal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.

[0037] FIG. 1A illustrates an embodiment of a computer-assisted system. The system can be used, for example, in surgical, diagnostic, therapeutic, biopsy, or non-medical procedures, and is generally indicated by the reference numeral 100. As shown in FIG. 1A, the computer-assisted system 100 can include one or more manipulator assemblies 102 for operating one or more medical instrument systems 104 in performing various procedures on a patient P positioned on a table T in a medical environment 101. For example, the manipulator assembly 102 can drive catheter or end effector motion, can apply treatment to target tissue, and / or can manipulate control members. The manipulator assembly 102 can be teleoperated, non-teleoperated, or a hybrid teleoperated and non-teleoperated assembly with select degrees of freedom of motion that can be motorized and / or teleoperated and select degrees of freedom of motion that can be non-motorized and / or non-teleoperated. An operator input system 106, which can be inside or outside of the medical environment 101, generally includes one or more input devices 107 (see FIG. IB) for controlling manipulator assembly 102. Manipulator assembly 102 supports medical instrument system 104 and can optionally include a plurality of actuators or motors that drive inputs on medical instrument system 104 in response to commands from a control system 112. The actuators can optionally include drive systems that when coupled to medical instrument system 104 can advance medical instrument system 104 into a naturally or surgically created anatomic orifice. Other drive systems can move the distal end of medical instrument in multiple degrees of freedom, which can include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and in three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). The manipulator assembly 102 can support various other systems for irrigation, treatment, or other purposes. Such systems can include fluid systems (including, for example, reservoirs, heating / cooling elements, pumps, and valves), generators, lasers, interrogators, and ablation components.

[0038] Computer-assisted system 100 also includes a display system 110 for displaying an image or representation of the surgical site and medical instrument system 104 generated by an imaging system 109 which can include an imaging system, such as an endoscopic imaging system. The outputs of the imaging system 109 can comprise a portion of multimodal data 202 (FIG. 2A) as described in more detail below. Display system 110 and operator input system 106 can be oriented so an operator O can control medical instrument system 104 and operator input system 106 with the perception of telepresence. A graphical user interface can be displayable on the display system 110 and / or a display system of an independent planning workstation.

[0039] In some examples, the endoscopic imaging system components of the imaging system 109 can be integrally or removably coupled to medical instrument system 104.However, in some examples, a separate imaging device, such as an endoscope, attached to a separate manipulator assembly can be used with medical instrument system 104 to image the surgical site. The endoscopic imaging system 109 can be implemented as hardware, firmware, software, or a combination thereof which interact with or are otherwise executed by one or more computer processors, which can include a processor system 114 of the control system 112.

[0040] Computer-assisted system 100 can also include a sensor system 108. The sensor system 108 can include a position / location sensor system (e.g., an actuator encoder or an electromagnetic (EM) sensor system) and / or a shape sensor system (e.g., an optical fiber shape sensor) for determining the position, orientation, speed, velocity, pose, and / or shape of the medical instrument system 104. The sensor system 108 can also include temperature, pressure, force, or contact sensors or the like. The outputs of the sensor system 108 can comprise a portion of the multimodal data 202 (FIG. 2A) as described in more detail below.

[0041] Computer-assisted system 100 can also include the control system 112. Control system 112 includes memory 116 and the processor system 114 for effecting control between medical instrument system 104, operator input system 106, sensor system 108, and display system 110. Control system 112 also includes programmed instructions (e.g., a non-transitory machine-readable or computer-readable mediums storing the instructions) to implement a procedure using the manipulator assembly 102 including for navigation, steering, imaging, engagement feature deployment or retraction, applying treatment to target tissue (e.g., via the application of energy), or the like.

[0042] Control system 112 can optionally further include a virtual visualization system to provide navigation assistance to operator O when controlling medical instrument system 104 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system can be based upon reference to an acquired pre-operative or intra-operative dataset of anatomic passageways. The virtual visualization system processes images of the surgical site imaged using imaging technology such as computerized tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or the like. The control system 112 can use a pre-operative image to locate the target tissue (using vision imaging techniques and / or by receiving user input) and create a pre-operative plan, including an optimal first location for performing treatment. The pre-operative plan can include, for example, a planned size to expand an expandable device, a treatment duration, a treatment temperature, and / or multiple deployment locations.

[0043] FIG. IB is another example diagram of a computer-assisted system 100, according to various embodiments. In the example of FIG. IB, the operator input system 106 comprises a workstation that includes the one or more input devices 107. The one or more input devices 107 may comprise one or more leader input devices that are designed to be contacted and manipulated by the operator O. For example, the one or more input devices 107 may be configured for use by the hands, the head, or some other body part(s) of the operator O. The one or more input devices 107, in the example of FIG. IB, are supported by the operator input system 106 and can be mechanically grounded. In some embodiments, an ergonomic support 118 (e.g., forearm rest) can be provided on which the operator O can rest his or her forearms. In some examples, the operator O can perform tasks at a worksite within a workspace near the manipulator assembly 102 during a procedure, by commanding the manipulator assembly 102 using the one or more input devices 107. In a medical example, the worksite may be a surgical worksite within the medical environment 101 associated with the patient P.

[0044] In the example of FIG. IB, the display system 110 may be configured to display images for viewing by the operator O and be configured moved in various degrees of freedom (DOFs) to accommodate the viewing position of the operator O and / or to provide control functions. In embodiments where the display system 110 provides control functions, the one or more input devices 107 may include the display system 110. The display system 110 may display images that depict a worksite at which the operator O is performing various tasks by manipulating the one or more input devices 107 and / or the display system 110. In some examples, images displayed by display system 110 may be received by the operator input system 106 from one or more imaging devices arranged at a worksite (e.g., images from the imaging system 109). In other examples, the images displayed by display system 110 may be generated by the display system 110 (or by a different connected device or system), such as for virtual representations of tools, the worksite, or for user interface components. In some embodiments, the display system 110 may display one or more tasks for the operator O to perform with respect to any component of the computer-assisted system 100.

[0045] As illustrated, the computer-assisted system 100 also includes the manipulator assembly 102 acting as a follower device that can be commanded by the operator input system 106. In a medical example, the manipulator assembly 102 can be located near anoperating table (e.g., the table T of FIG. 1A a bed, or other support) on which the patient P can be positioned. In some medical examples, the manipulator assembly 102 is provided on the operating table, e.g., on or in a patient, simulated patient, or model, training dummy, etc. (not shown). As illustrated, the manipulator assembly 102 may include a plurality of repositionable structures 120 (sometimes referred to as “manipulator arms” in robotic embodiments). In some embodiments, the repositionable structures 120 may include a plurality of links that are rigid members and joints that can be individually actuated as part of a kinematic series. Additionally, each of the repositionable structures 120 is configured to couple to a specific one of removable instruments 122 (such as an instrument of the medical instrument system 104 of FIG. 1A). While FIG. IB illustrates the manipulator assembly 102 having four repositionable structures 120a- 120d, in other embodiments, the manipulator assembly 102 may include one, two, three, four, five, six, or additional or fewer repositionable structures 120a- 120d.

[0046] The removable instruments 122 of the medical instrument system 104 can include, for example, a working portion 126 and one or more structures for supporting and / or driving the working portion 126. Example working portions 126 include end effectors that physically contact or manipulate material, energy application elements that apply electrical, RF, ultrasonic, or other types of energy, sensors that detect characteristics of the workspace environment (such as temperature sensors, imaging devices, etc.), and the like.

[0047] In various embodiments, examples of removable instruments 122 include, without limitation, a sealing instrument, a cutting instrument, a sealing-and-cutting instrument, an energy instrument for applying energy, a gripping instrument (e.g., clamps, jaws), a stapler, an imaging instrument such as one using optical, RF, or ultrasonic imaging modalities, a sensing instrument, an irrigation instrument, a suction instrument, and / or the like. In addition, the removable instruments 122 may include a transmission mechanism 128 that can be coupled to a drive assembly 130 of the respective repositionable structure 120a- 120d. The drive assembly 130 may include a drive and / or other mechanisms controllable from the operator input system 106 that transmit forces to the transmission mechanism 128 to articular or otherwise actuate the removable instruments 122.

[0048] As illustrated, each of the removable instruments 122 may be mounted to a portion of a respective repositionable structure 120a- 120d. In FIG. IB, this is shown with the drive assembly 130 physically coupled to the transmission mechanism 128. The distal portion of each repositionable structure 120a- 120d further includes a cannula mount 124 to which acannula (not shown) is mounted. When a cannula is mounted to the cannula mount 124, a shaft of that one of the removable instruments 122 passes through the cannula and into a workspace.

[0049] In various embodiments, one or more of the working portions 126 of the removable instruments 122 may include an imaging device for capturing images as part of the imaging system 109. The imaging devices may include any sensing technology capable of acquiring an image. Example imaging instruments include an optical endoscope, a hyperspectral camera, an ultrasonic sensor, etc. Imaging instruments may comprise monoscopic imagers, stereoscopic imagers, and / or the like. Imaging devices based on radiofrequency domains may capture images in any frequency spectrum, including visible light, infrared light, ultraviolet light, and / or the like. The imaging devices may include an illumination source to light the region being imaged. In embodiments where the working portions 126 of one or more of the removable instruments 122 include an imaging device, this instrument of the removable instruments 122 may be configured to capture images of a portion of the workspace for display via the display system 110.

[0050] In some embodiments, the repositionable structures 120a- 120d and / or removable instruments 122 can be controlled to move the working portions 126 in response to manipulation of the one or more input devices 107 by the operator O. Accordingly, the repositionable structures 120a- 120d and / or removable instruments 122 may be said to “follow” the one or more input devices 107 through teleoperation. This enables the operator O to perform tasks at the worksite using the repositionable structures 120a- 120d and / or removable instruments 122. For a surgical example, the operator O can direct the repositionable structures 120a- 120d of the manipulator assembly 102 to move the working portions 126 as part of a surgical procedure performed at an internal surgical site that is entered via one or more minimally invasive apertures or natural orifices. It should be appreciated that, in some embodiments, the manipulator assembly 102 may include nonteleoperated components that the operator O or other medical professional must manually manipulate to a desired pose.

[0051] In some embodiments, the repositionable structure 120a of the computer-assisted system 100 may be configured to support a working portion 126a that includes an imaging device (also referred to herein as an “imaging device 126a”). For convenience, an instrument of the removable instruments 122 / that includes an imaging device is also referred to as an “imaging instrument” herein. The control system 112 may be configured to command therepositionable structure 120a and / or the imaging instrument of the removable instruments 122 comprising the imaging device 126a to automatically position and / or orient (“pose”) the field of view (FOV) of the imaging device 126a to provide images of the workspace and / or other removable instruments 122.

[0052] In the illustrated embodiment, the control system 112 is communicatively coupled to the operator input system 106. In other embodiments, the control system 112 may be provided as a component of the operator input system 106 and / or the manipulator assembly 102. During teleoperation, as the operator O moves the one or more input devices 107, one or more sensors configured to detect the one or more input devices 107 generate spatial and / or orientation movement data that is provided to control system 112. The control system 112 may interpret the spatial and / or orientation information to determine and / or provide control signals to the manipulator assembly 102 to control the movement of repositionable structures 120a- 120d, removable instruments 122, and / or working portions 126. In addition to the components of the manipulator assembly 102, in some embodiments, the control system 112 is configured to interpret inputs received from the operator input system 106 to control operation of one or more auxiliary devices (not depicted) utilized in a procedure. For example, the operator input system 106 may be used to control a pose of a surgical bed or operation of an insufflator.

[0053] In one embodiment, the control system 112 supports one or more wired communication protocols, (e.g., Ethernet, USB, and / or the like) and / or one or more wireless communication protocols (e.g., Bluetooth, IrDA, HomeRF, IEEE 2102.11, DECT, Wireless Telemetry, and / or the like) for communications between the control system 112 and the operator input system 106 and / or the manipulator assembly 102.

[0054] In some embodiments, the control system 112 may be implemented at one or more computing systems. For example, one or more computing systems may be used to control the manipulator assembly 102. As another example, one or more computing systems may be used to control components of the operator input system 106, such as movement of display system 110.

[0055] As illustrated, the control system 112 includes the processor system 114, the memory 116, a control module 132, an artificial intelligent (Al) assist module 134. The memory 116 may store the control module 132 and the Al assist module 134, and a database 136. The processor system 114 may include one or more processors having different processing architectures for processing instructions. For example, the one or more processorsmay be one or more cores or micro-cores of a multi-core processor, a central processing unit (CPU), a microprocessor, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), a tensor processing unit (TPU), and / or the like.

[0056] In some embodiments, the processor system 114 includes circuity to support one or more communication interfaces (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.). Additionally, a communication interface of control system 112 may include an integrated circuit for connecting the control system 112 to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) and / or to another device, such as the operator input system 106 and / or the manipulator assembly 102.

[0057] Additionally, the memory 116 may include non-persistent storage (e.g., volatile memory, such as random access memory (RAM), cache memory), persistent storage (e.g., a hard disk, an optical drive such as a compact disk (CD) drive or digital versatile disk (DVD) drive, a flash memory, a floppy disk, a flexible disk, a magnetic tape, any other magnetic medium, any other optical medium, programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a FLASH-EPROM, and / or any other memory chip or cartridge. The non-persistent storage and persistent storage are examples of non- transitory, tangible machine-readable media that can store executable code that, when run by one or more processors (e.g., processor system 114), can cause the one or more processors to perform one or more of the techniques and / or methods disclosed herein.

[0058] The Al assist module 134 may implement one or more machine learning models and / or training protocols therefor. For example, the Al assist module 134 may implement one or more neural networks, deep learning models, decision trees, support vector machines, linear regression, generative Al models, reinforced learning models, random forests, Naive Bayes models, large language models (LLMs), generative adversarial networks, foundation models, image recognition models, linear discriminant analysis models, creative applications, autoregressive models, supervised or unsupervised learning models, multimodal models, vision language models (VLMs), vision foundation models (VFMs), large multi-modal models (LMMs), Transformer models (including Robotic Transformer models), or another machine learning or Al model for performing the methods described herein. The structure of the one or more machine learning models is described in more detail below. The Al assist module 134 may include dedicated processors and memory for storing and performing Alprocesses, or the Al assist module 134 may utilize resources of the processor system 114 and the memory 116 to store and / or perform any processing or tasks required to perform the methods described herein.

[0059] The database 136 may be a part of or separate from the memory 116. When separate, the database 136 may be electrically coupled to the processor system 114 via wire or wireless methods known in the art, including through various wide area or local networking protocols. The database 136 is configured to store data for processing by the control system 112 and outputs generated by the control system 112. It should be appreciated that in embodiments where the database 136 is external to the control system 112, one or more additional control systems associated with additional computer-assisted systems may also be communicative coupled with the database 136. As a result, each control system associated with a facility may have access to the instrument life scores for any instrument utilized at the facility.

[0060] Additionally, the control system 112 may also include one or more input devices (such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device) and / or output devices (such as a display device, a speaker, external storage, a printer, or any other output device). In some embodiments, the control system 112 may be implemented on a particular node of a distributed computing system (e.g., a cloud computing system). As another example, different functionalities associated with the control system 112 may be implemented on different nodes of the distributed computing system. Further, one or more elements of the aforementioned control system 112 may be located at a remote location and connected to the other elements over a network.

[0061] In an endoscopic surgery example, the imaging instrument comprising the imaging device 126a may be inserted into the patient prior to the other removable instruments 122, including a second instrument 122b comprising a second working portion 126b. The second instrument 122b can include any appropriate working portion 126b, and can even include a second imaging device. Accordingly, the imaging device 126a may be maneuvered to positioned to identify a target to which other instruments may interact with as part of another task. The control system 112 may, for example, automatically command the corresponding repositionable structures 120a and 120b to position respective instruments 122a and 122b to perform one or more tasks in tandem, or sequentially based on the specific task, instruments, and positions of the repositionable structures 120a and 120b.

[0062] In examples described further herein, the control system 112 may perform Al processes and algorithms via the Al assist module 134 to derive instrument life scores for removable instruments used on the manipulator assembly 102 (e.g., removable instruments 122). In particular, the instrument life scores may be derived based on instrument life metrics that are generated from different data stream modalities either in real-time during a procedure or on complete data sets after a procedure is completed. In the real-time examples, the generated instrument life metrics may be used to perform various intra-procedure operations such as displaying an alert regarding the current instrument life scores indicated by the instrument life metrics and / or controlling aspects of the computer-assisted system 100 based on the current instrument life scores indicated by the instrument life metrics. In the post procedure examples, the generated instrument life metrics may be used to associate the current instrument life scores indicated by the instrument life metrics with the removable instruments 122. The associated current instrument life scores may then be recalled at later times to assist in selecting one or more of the removable instruments 122 for use in a new procedure. Utilizing the Al assist module 134 to derive the instrument life scores based on the multi-modal procedure data may result in the control system 112 avoiding the problems of under- and overestimating instrument life scores associated with currently known methods.

[0063] The Al assist module 134 may be coupled to the manipulator assembly 102 or be a stand alone device located in the medical environment 101 or other location that employs the computer-assisted system 100. In either case, the Al assist module 134 may act as an independent node of the computer-assisted system 100 to derive the life scores associated with the removable instruments 122 either during or after a procedure.

[0064] Referring now to FIG. 1C, a schematic, side view of an example instrument 140 of the removable instruments 122 is depicted according to some embodiments. The instrument 140 can be, or include, an instrument used to perform medical (e.g., surgical, diagnostic, and / or therapeutic) or non-medical procedures (e.g., industrial inspection applications). The instrument 140 includes a shaft 142 elongated along a longitudinal axis AL, between proximal end portion 144 and distal end portion 146. In implementations in which the instrument is a medical instrument, such as for use in minimally-invasive medical procedures, the shaft 142 is on the order of a few millimeters in diameter, for example from five to eight millimeters in diameter. However, it should be appreciated that the instrument 140 shown in FIG. 1C is merely one possible example instrument type used in the computer-assistedsystem 100, and that other types of instruments including flexible catheter instruments for endoluminal applications are contemplated.

[0065] The instrument 140 further includes an end effector 148 coupled to the distal end portion 146 and a force transmission system 150 (only the exterior housing portion of which is depicted) coupled to the proximal end portion 144. The end effector 148 is configured to carry out a medical or non-medical (such as industrial) procedure. For example, the end effector 148 can include one or more tools such as gripping tools, staplers, shears, ligation clip appliers, electrosurgical tools, ultrasonic tools, suturing tools, translating sleds, translating cutting tools, or other types of tools. While the illustration of FIG. 1C depicts an end effector 148 having jaw members 152 configured to move toward and away from each other (either by one or both jaw members pivoting about a pivot axis), such a configuration is exemplary and non-limiting and those of ordinary skill in the art would appreciate the instrument 140 can have any of a variety of end effectors without departing from the scope of the present disclosure.

[0066] In FIG. 1C, the instrument 140 further optionally includes an articulable structure 154 coupling the end effector 148 to the shaft 142. As shown in FIG. 1C, the articulable structure 154 can be positioned along the distal end portion 146 of the shaft 142. But the disclosure is not so limited and the articulable structure 154 can be positioned at any location along the shaft 142 without limitation. In addition, the instrument 140 can include more than one articulable structure 154, such as two, three, or more articulable structures located in series or at multiple spaced apart locations along the length of the shaft 142. The articulable structure 154 can be controlled and actuated via actuation members (not illustrated in FIG. 1C), such as pull-pull type or push-pull type actuation members, operably coupled to one or more drive components of the force transmission system 150, and thus able to be actuated via a manipulator through the force transmission system. In various embodiments, as those having ordinary skill in the art would be familiar with, an articulable structure can serve as a wrist mechanism supporting and coupling the end effector 148 to the shaft 142 so as to allow orientation of the end effector 148 relative to the shaft in pitch and / or yaw.

[0067] In the embodiment of FIG. 1C, the force transmission system 150 is coupled to the proximal end portion 144 of the shaft 142. In other embodiments, the force transmission system 150 may be coupled at various locations along the shaft 142, and in some cases moveable along the shaft, but generally in a position such that it remains external to a remote site (such as a patient’s body) at which the end effector 148 and a distal end portion 146 ofthe shaft 142 are inserted to perform a procedure, thereby permitting access to manipulate inputs on the force transmission system 150.

[0068] Force transmission system 150 includes a housing 158 supporting an input drive portion 160. Input drive portion 160 includes a drive interface 162. Drive interface 162 provides mechanical connections to the other control features of force transmission system 150, such as various output drives configured to be operated to transmit force to control the moveable components and operations at the distal end portion 146 of the instrument 140. In the implementation of FIG. 1C, drive interface 162 is configured to couple to a manipulator system.

[0069] Furthermore, as shown in FIG. 1C, the instrument 140 may include a force sensor 164 that is integrated therewith or otherwise coupled thereto. For example, the force sensor 164 may be a proximal force sensor located with the housing 158. In other embodiments, the force sensor 164 may be a distal force sensor at the distal end portion 146. In yet additional embodiments, the force sensor 164 may comprise one or more distal sensors and one or more proximal sensors. For example, the one or more distal sensors may be configured to detect, sense, or measure force(s) exerted or experienced by the distal end portion 146 in one or more first directions (e.g., in the plane perpendicular to the axis AL) and the one or more proximal sensors may be configured to detect, sense or measure force(s) exerted or experienced by the distal end portion 146 in one or more second directions (e.g., along the axis AL). Because the forces exerted or experienced by an instrument can impact the lifetime of the instrument, data (e.g., time-series data) generated by the force sensor 164 may be used to generate an instrument life metric and / or instrument life score for the instrument.

[0070] Furthermore, in some embodiments, the instrument 140 can include a life indicator 166 configured to present a user-perceptible indication of a current instrument life score for the instrument 140. For example, user-perceptible indication may be presented via a display screen, one or more light-emitting diodes (LEDs), an electronic ink (e-ink) display, an audio output, and / or other means known in the art. It should be appreciated that the life indicator 166 may be operable when the instrument 140 is not coupled to the control system 112. As a result, when the instruments 140 are in storage, facility personnel can view the indication of the current instrument life score to select an appropriate instrument to use in a procedure. As illustrated, the life indicator 166 may be embedded or otherwise coupled to the instrument 140 such as within the housing 158. Further, the life indicator 166 may include or be coupled to a memory 168 for storing the current instrument life score therein.

[0071] It should be appreciated by those having ordinary skill in the art that other ones of the removable instruments 122 of types different from the instrument 140 depicted in FIG. 1C may also include force sensors and life indicators similar to force sensor 164 and life indicator 166.

[0072] FIG. 2A is a diagram of a system 200 for generating an instrument life metric that is indicative of a current instrument life score for the removable instruments 122 of FIG. IB. As shown in FIG. 2A, the system 200 includes medical procedure data comprising a plurality of data streams 202 from one or more data sources. The medical procedure data comprising the plurality of data streams 202 may be either post-procedure data from completed procedures or real-time data gathered from a procedure that is currently in process. In general, the medical procedure data is representative of use of the removable instruments 122. However, the plurality of data streams 202 may include data from the computer-assisted system 100 gathered during the procedure that, while not directly related to the use of the removable instruments 122, could still be used to infer the amount of life consumed during a procedure. For example, the additional data can include data indicate of the type of procedure being performed, the particular doctor or operator performing the procedure, etc.

[0073] A first data stream 202A of the plurality of data streams 202 comprises data relating to operation of the removable instruments 122 and / or the manipulator assembly 102 by the computer-assisted system 100 of FIG. 1A or IB during the medical procedure. In some embodiments, the first data stream 202A may comprise time-series or sequential data elements that include both a temporal indicator and a magnitude value. In particular, the first data stream 202A may comprise data relating to operation of the removable instruments by the computer-assisted system during a medical procedure. The first data stream 202A may include one or more of: driving power data, system event data generated by the control system 112 (e.g., data indicating events such as instrument installs, insertions, removals, etc. associated with the removable instruments 122), and / or kinematic data associated with the removable instruments 122 and / or the repositionable structures 120 (e.g., data indicating movements of the instruments 122 and / or the repositionable structures 120 during the medical procedure), removable instruments 122, and / or other auxiliary devices associated therewith. In some embodiments, the first data stream 202A can include data on installation and / or removal information for the removable instruments 122 with respect to the manipulator assembly 102 or other portions of the computer-assisted system 100. In someembodiments the first data stream 202A can include energy application information for the removable instruments 122.

[0074] A second data stream 202B of the plurality of data streams 202 may comprises image data (e.g., image data from the imaging system 109). In particular, the image data can include medical environment (e.g., operating room) image data such as an external video data stream and endoscope image data such as a procedure video data stream. The image data can also include post procedure image data showing how the removable instruments 122 are sanitized, transported, stored, etc. Use of video data (e.g., a linked sequence of statice image frames) can enable analysis of both spatial and temporal components by the system 200 or other systems described herein.

[0075] A third data stream 202C of the plurality of data streams 202 may comprise data relating to forces exerted upon a distal end portions 146 of the removable instruments 122 during the medical procedure (e.g., data from the force sensor 164). Like the first data stream 202A, the third data stream 202C may comprise time- series data. The third data stream 202C may be generated by a distal force sensor disposed in a distal portion of an instrument 122 or by a proximal force sensor in a proximal portion of the instrument 122. It should be appreciated that the data stream 202C may be processed in a different manner depending upon the arrangement of the force sensor(s).

[0076] The plurality of data streams 202 may include a plurality of additional data streams 202N that may have data modalities the same as or different from the first data stream 202A, the second data stream 202B, and the third data stream 202C. For example, in some embodiments, the plurality of additional data streams 202N may include text data, audio data, 2D data from sensors, time-of-flight data, etc.

[0077] It should also be appreciated that in some embodiments, other combinations of the plurality of data streams 202 are possible. For example, in some embodiments, the second data stream 202B and the additional data streams 202N may be omitted. In these embodiments, the system 200 may operate using only the first data stream 202A and the third data stream 202C as input data streams to generate the instrument life metric as described herein. In particular, the system 200 may generate the instrument life metric based solely on kinematic data associated with the removable instruments 122 and / or the repositionable structures 120 (e.g., first data stream 202A) and the force data for the removable instruments 122 (e.g., third data stream 202C).

[0078] The system 200 also includes a first machine learning model 203. The first machine learning model 203 comprises a set of interconnected nodes, layers, trained parameter values (e.g., multiplicative weights, additive bias, etc.), etc. The trained parameters are set via backpropagation techniques in a training process that uses historical data inputs. The training process may be divided into multiple steps that include a self-supervised pretraining process and a fine-tunning supervised process. For the self-supervised process for the first machine learning model 203, the historical inputs may include historical procedure data (e.g., historical versions of the plurality of data streams 202). However, for the fine-tuning supervised process, the historical data inputs can include historical procedure data that is further labeled with known metadata. The known metadata can include with known instrument life metrics resulting from the historical procedure data and / or other metadata values useful in setting the parameters of the first machine learning model 203. In some embodiments, the known instrument life metrics are obtained by mechanically assessing test removable instruments after use during historical medical and / or testing procedures from which the historical procedure data was generated. In some embodiments, the testing procedures may include a-typical operations (e.g., operations not generally performed during a procedure) that better demonstrate the effect of various operation modes of the computer- assisted system 100 on the remaining life of an instrument as compared with typical operations generally performed during a procedure.

[0079] In some embodiments, the first machine learning model 203 is configured using a multi-modal transformer type architecture that can receive inputs of different modalities (e.g., the plurality of data streams 202) in order to generate the instrument life metric as described herein. However, it should be appreciated that this architecture may be substituted or supplemented with other Al architectures including, but not limited to, convolutional neural network (CNN) architectures, recurrent / recursive neural network (RNN) architectures, sorting / clustering architectures, etc.

[0080] The first machine learning model 203 may be executable by the control system 112, such as by the Al assist module 134, and / or another similar computing system connected to the control system 112. The first machine learning model 203, when executed, is configured to output one or more instrument life metrics (e.g., one instrument life metric for each of the removable instruments 122 used or in use during a procedure). The instrument life metrics may be based on input of the plurality of data streams 202. As described herein, execution of the first machine learning model 203 can include transforming the input plurality of datastreams 202 into embedded tokens, data values, etc. to which various modification functions and the trained parameter values are applied to generate the output instrument life metrics. In some embodiments, the output instrument life metric includes a set of component metrics associated with component parts of the removable instruments 122. Accordingly, the techniques disclosed herein with respect to updating an instrument life score for the removable instruments 122 envisions a similar application to updating an instrument life score for one or more components of the removable instruments 122.

[0081] As shown in FIG. 2A, the first machine learning model 203 includes a plurality of projection layers 204. The plurality of projection layers 204 are configured to receive the plurality of data streams 202 and convert the underlying data into machine readable formats suitable for processing by the first machine learning model 203. As described in more detailed herein, the machine-readable formats can include token elements representative of the data in the plurality of data streams 202. In some embodiments, the token elements are natural language descriptions of the corresponding data stream 202. The plurality of projection layers 204 may include a distinct layer for each of the plurality of data streams 202 or, alternatively, some of the plurality of projection layers 204 can be configured to receive multiple data streams of the plurality of data streams 202 such as ones having the same modality. For example, a first projection layer 204A can be configured to receive the first data stream 202A a second projection layer 204B can be configured to receive the second data stream 202B, a third projection layer 204C can be configured to receive the third data stream 202C, and corresponding projection layers 204N can be configured to receive the additional data streams 202N. However, in cases where some of the plurality of projection layers 204 are configured to receive multiple data streams, the first projection layer 204 A may be configured to receive multiple time series type data streams and the second projection layer 204B may be configured to receive multiple image type data streams (e.g., both the external video data stream and the procedure video data stream described above). Similarly, in some embodiments, the third projection layer 204C can be omitted and the first projection layer 204A can receive the third data stream 202C along with the first data stream 202A.

[0082] In either case, each of the plurality of data streams 202 may be input into one of the plurality of projection layers 204 and passed onto additional / hidden layers 206 before being received by an output layer 208. The output layer 208 aggregates outputs from each of the plurality of projection layers 204 and the additional / hidden layers 206 to generate the instrument life metrics. As such, the instrument life metrics output by the first machinelearning model 203 are based on outputs from the projection layers 204 for the plurality of data streams 202.

[0083] As described herein, the instrument life metrics output from the first machine learning model 203 are indicative of the instrument life scores for the associated ones of the removable instruments 122. In some embodiments, the instrument life metrics output from the first machine learning model 203 are instrument life decrement values. In these embodiments, the current instrument life scores for the removable instruments 122 are determined by decrementing previous instrument life scores for the removable instruments 122 by the output instrument life decrement values. However, it should be appreciated that in other embodiments, the first machine learning model 203 may be configured to directly output the current instrument life scores without any further calculation or processing needed. In these embodiments, the previous instrument life scores for the removable instruments 122 may be included as one of the plurality of data streams 202.

[0084] The first projection layer 204A includes an embedding layer 210 and a transformer 212 and the third projection layer 204C includes an embedding layer 214 and a transformer 216. The embedding layers 210, 214 are configured to project the data into machine readable tokens with embedded positional values. In some embodiments, the projection layers 204A, 204C or the embedding layers 210, 214 in particular are trained using a plurality of historical time series data streams having associated time stamp information such that the embedded positional values are directly derived from the time stamp information encoded with the data itself. Furthermore, the transformers 212, 216 of the projection layers 204A, 204C, respectively, include at least one attention layer. These attention layers are trained to modify received inputs into refined embeddings that represent contextual relationships between the data points in the input data stream. The attention layer may be trained using ProbSparse Attention techniques or similar so that important data points are identified while avoiding quadratic complexity issues associated with longer-duration time series data. Additionally, AutoFormer, Spacetimeformer, Informer, Pyraf ormer, FEDformer, etc. type architectures can be employed as the classifier that accepts the data as an input. In some embodiments, the inputs into the attention layers comprise outputs of the embedding layers 210, 214. However, the projection layers 204A, 204C may employ multiple sequential attention layers interposed between other neural network layers such as feed forward layers or the like.

[0085] In general, the second projection layer 204B for the image data stream 202B is configured to transform the image data into embedded data useable by the remaining portionsof the first machine learning model 203. For example, the second projection layer 204B can include a convolutional neural network (CNN) or a large multimodal model (LMM) trained to process and classify image data.

[0086] The additional / hidden layers 206 may comprise various Al layers known in the art. Such layers include additional multiheaded self-attention or cross-attention layers, feed forward layers, etc. In some embodiments, the additional / hidden layers 206 may be omitted and the output layer 208 may be directly connected to the plurality of projection layers 204. However, in other embodiments, the additional / hidden layers 206 and the output layer 208 may comprise layers of a pretrained Al model such as a large language model (LLM) or similar. This pretrained Al model can include third party provided models that are either finetuned for use with the computer-assisted system 100 or that are used as is without any additional tunning or training. When the pretrained Al model utilized is an LLM, the plurality of projection layers 204 are configured to transform the input plurality of data streams 202 into machine readable tokens representing textual descriptions of the underlying data elements that can be fed into the general word embedding layers of the pretrained LLM. As described in more detail below, the plurality of projection layers 204 may be configured to generate these textual descriptions via dedicated training processes.

[0087] Once the instrument life metrics are generated and output from the output layer 208 of the first machine learning model 203, they can be received by the computing system that is executing the first machine learning model 203. Once received, the computing system can leverage the instrument life metrics and / or the instrument life scores indicated thereby to perform various intra-procedure or post-procedure operations. The computing system can include the control system 112, a similar computing environment that implements the system 200, and / or combinations thereof. For example, in some embodiments related to postprocedure operations or pre-procedure instrument selection, the first machine learning model 203 may be executed by a server or other similar processing environment sperate from the control system 112 to generate the instrument life metrics and derive the associated instrument life scores as described herein. Then, the derived instrument life scores may be sent to the control system 112 to perform the intra-procedure operations as described herein.

[0088] The post-procedure operations may include associating the current instrument life scores with the corresponding ones of removable instruments 122 based on the output instrument life metrics. In some embodiments, associating the current instrument life scores with the removable instruments 122 includes storing the current instrument life scores inrecords of database 136 that are associated with removable instruments 122. These records can then be accessed by the computing system at a later time to retrieve the current life scores for the removable instruments 122. These retrieved life scores may then be presented on a display (e.g., display system 110) and / or via the life indicators 166 as user-perceptible indications. In some embodiments, the display presents the user-perceptible indication of the current instrument life scores by displaying a portion of a web portal interface for the database 136. In some embodiments, associating the current instrument life scores with the removable instruments 122 includes storing the current instrument life scores in respective memories coupled to the removable instruments 122 (e.g., the memory 168 of the life indicator 166).

[0089] Additionally or alternatively, associating the current instrument life scores with the removable instruments 122 includes storing the current instrument life scores in a memory coupled to the removable instruments 122 themselves (e.g., memory 168). In these embodiments, the current instrument life score may be viewed by facility personnel by interfacing directly with the removable instruments 122, even when the removable instruments 122 are not coupled to a manipulator system. In these embodiments, the removable instruments 122 may include the life indicator 166, which may include a display to indicate a life indicator value, a life gauge that includes one or more LED outputs (e.g., a particular color light is used to indicate a range of instrument life score values), and / or other life indication means known in the art.

[0090] The intra-procedure operation may include displaying an alert regarding the current instrument life scores indicated by the instrument life metrics on a display (e.g., the display system 110) and / or controlling aspects of the computer-assisted system 100 based on the current instrument life scores. The alert may include a user-perceptible indication of the current instrument life scores. In some embodiments, the alert may include a notification containing an indication of whether a projected failure time for one or more of the removable instruments 122 will occur during a remaining time for completing the ongoing medical procedure. In these embodiments, the control system 112, via the control module 132 and / or the Al assist module 134 is further configured to compare the output instrument life metrics or related instrument life scores to previous instrument life metrics or instruments life scores for the removable instruments 122 to determine a degradation rate of the removable instruments 122. The degradation rate may then be used to project the failure time for the removable instruments 122. 1

[0091] Controlling aspects of the computer-assisted system 100 may include controlling the manipulator assembly 102 and the repositionable structures 120, in particular, based on the current instrument life scores output by the first machine learning model 203 in real-time during the procedure. This real-time control may include the control system 112, via the control module 132 or Al assist module 134, generating a set of tasks to be performed using the repositionable structures 120 or other portions of the computer-assisted system 100 and selecting a particular manipulator / instrument of the manipulator assembly 102 (e.g., one of the repositionable structures 120 and one of the removable instruments 122) to perform one or more of the set of tasks based on the current instrument life scores.

[0092] As shown in FIG. 2B, in some embodiments, the instrument life metrics output from the first machine learning model 203 or the life scores derived therefrom may be input into additional machine learning models 218 and 220 to facilitate the real-time control operations of the computer-assisted system 100. Such a system may also utilize some or all of the plurality of data streams 202 as inputs. For example, the additional machine learning model 218 may be configured to generate the set of tasks using the current instrument life scores or the instrument life metrics for the removable instruments 122 as inputs. The additional machine learning model 220 may be configured to select the particular manipulator or instrument to perform one or more of the set of tasks by using the set of tasks and the current instrument life scores as input. The additional machine learning model 220 may also be configured to assign the tasks to the selected instruments / manipulator. The machine learning model 218 and 220 may include task generation, task selection, and / or instrument / arm selection modules of a system for determining task specific data streams for controlling a manipulator assembly or similar repositionable structure (such as the manipulator assembly 102) as shown and described in U.S. Provisional application 63 / 663,539 titled MULTI-MODAL AND TASK- ADAPTIVE SYSTEM ARCHITECTURE FOR AN INTELLIGENT SURGICAL ROBOT and filed on June 24, 2024, which is incorporated by reference herein in its entirety. In some embodiments, the set of tasks include a set of future tasks needed to complete the procedure. In these embodiments, the machine learning models 218 and / or 220 are configured to project an amount of instrument life that will be consumed by the set of future tasks and associate the one or more removable instruments with one or more of the set of future tasks based on (1) the projected amount of instrument life consumed and (2) the current instrument life score. In some scenarios, the models 218 and / or 220 may select instruments such that future low life states for theremovable instruments 122 are avoided. Additionally, the models 218 and / or 220 may select instruments to maximize instrument usage prior to the instrument entering the low life state. This may increase procedure efficiency by optimizing the number of tasks that can be completed across the set of instruments used in the procedure. Regardless, the control system 112 may control the operation of the computer-assisted system 100 based on the current instrument life score by commanding the particular manipulator or instrument to perform the set of tasks.

[0093] Furthermore, in embodiments where the output instrument life metrics are used to control the computer-assisted system 100 in real-time during a procedure, the first machine learning model 203 and / or another component of the system 200 may be configured to output a confidence score documenting the reliability of the output instrument life metrics. This confidence score may be used to dictate when the output instrument life metrics are useable for automatic performance of the control operations. For example, the confidence score may indicate when a sufficient threshold of reliability is reached, which limits situations where incorrect instrument life metrics (e.g., those generated from too little data) would be used. For example, during the initial phases of the procedure, the outputs of the first machine learning model 203 may not be usable for real-time control application because the low amount of data generated at that point in the procedure correlates with a low confidence score. However, as the first machine learning model 203 receives more data the confidence score may increase until the confidence score passes over a predefined reliability threshold. The predefined reliability threshold may include a value at which outputs from the first machine learning model 203 are deemed to be sufficiently reliable for use in real-time tasks. In some embodiments, the predefined reliability threshold may be identified from testing the first machine learning model 203 on historical data sets (e.g., post-procedure medical procedure data as described herein).

[0094] In some embodiments related to procedure planning, a computing system may obtain the current instrument life scores based upon outputs from the first machine learning model 203 to select one or more of available removable instruments 122 at a facility to be used in a new planned procedure. The computing system may include the control system 112, a similar type of system, a server, etc. In these embodiments, the computing system obtains a set of the current instrument life scores for the removable instruments 122 (e.g., recalls those scores from the database 136). In some embodiments, the set of the current life scores includes values for only those of the removable instruments 122 of particular types to be usedin the new planned procedure. The computing system then selects a particular one or multiple ones of the removable instrument 122 for use in the planned procedure based on the obtained set of current instrument life scores. The computing system may also display a notification indicating the particular removable instrument(s) is to be used for the planned procedure. The selection of the particular removable instrument(s) may be done by determining which of the obtained set of current instrument life scores are sufficient to be used for the planned medical procedure and selecting the particular removable instrument(s) as one(s) of the set of removable instruments determined to be sufficient.

[0095] As shown in FIG. 2C, in some embodiments, the instrument life metrics output from the first machine learning model 203 or the life scores indicated thereby may be input into a further machine learning model 222 along with new procedure information 224 associated with the planned procedure to determine which of the obtained set of current instrument life scores are sufficient. The further machine learning model 222 is configured to estimate a respective amount of life consumed by each one of the set of removable instruments when used to perform the planned procedure. The particular removable instrument(s) may be one(s) of the set of removable instruments determined to be sufficient and for which the respective amount of life consumed is closest to the current instrument life score for the particular removable instrument(s). However, in other embodiments, the particular removable instrument(s) are the one(s) of the set of removable instruments determined to be sufficient and for which the respective amount of life consumed is the least such that the maximum amount of overall life associated with the selected instrument is utilized. By ensuring that the maximum sufficient amount of life remaining is utilized, the further machine learning model 222 may reduce the likelihood of instruments having remaining life, but not of a sufficient amount to be utilized in a procedure. As a result, the further machine learning model 222 is able to reduce waste and improve the instrument life usage efficiency across an inventory of instruments at a facility.

[0096] In some similar procedure planning embodiments, a computing system may obtain a current instrument life score based upon outputs from the first machine learning model 203 to evaluate a particular one of the removable instruments 122 selected for use in a new planned procedure. The computing system may include the control system 112, a similar type of system, a server, etc. In these embodiments, the computing system obtains the current instrument life scores for the particular one of the removable instruments 122 (e.g., recalls the life scores from the database 136 and / or instrument memories 168). The computing systemthen evaluates the suitability of the particular removable instrument for use in the planned medical procedure based upon the obtained current instrument life score and displays a notification indicating that the particular removable instrument is suitable or not suitable to be used for the planned medical procedure.

[0097] In some embodiments, the computing system may evaluate the suitability of the selected particular one of the removable instruments 122 using the further machine learning model 222 and the new procedure information 224 as shown in FIG. 2C. For example, the computing system may evaluate the suitability of the particular removable instrument for use in the planned medical by inputting new medical procedure information 224 and the obtained current instrument life score into the further machine learning model 222, which is configured to estimate a respective amount of life consumed by the particular one of the removable instruments 122 when used to perform the planned medical procedure.

[0098] An example process for training the first machine learning model 203 will be described with reference now to FIG. 3. As described above, the training may include an initial self- supervised process followed by a supervised fine-tuning process. In either case, historical time series data 302, historical video data 304, and historical time series force data 306 (e.g., historical data from the force sensor 164) are compiled and associated with ground truth data 308. For the self- supervised portion of the training process, the ground truth data 308 can include a portion of the historical time series data 302 such that the first machine learning model 203 is trained to predict next token values in the historical time series data 302, the historical video data 304, and the historical time series force data 306. For the supervised fine-tuning portion of the training process, the ground truth data 308 may include known instrument life metrics resulting from the historical time series data 302, the historical video data 304, and the historical time series force data 306 such that the first machine learning model 203 is further trained to output instrument life metrics. The historical time series data 302 may include saved time series data streams from past procedures that includes force data, event data, and / or kinematic data as described herein. It should be appreciated that the event data may include a state of any robotic component associated with the computer- assisted system 100 at a particular time. The historical video data 304 may include saved external video data streams (e.g., medical environment video) and / or procedure video data streams (e.g., image data generated by an endoscope) form the same procedures that produced the historical time series data 302. In some embodiments, the ground truth data mayalso include data such as a procedure date and time, procedure name, hospital name, surgeon name, etc.

[0099] The historical time series data 302, the historical video data 304, and the historical time series force data 306 may be input into an initialized version of the first machine learning model 203 (e.g., a version where the parameter values are randomized) and processed through the plurality of projection layers 204 and the additional / hidden layers 206 until the output layer 208 generates training output values. These training output values may then be compared to the ground truth data 308 to identify an error 310 associated with the training output values. The error 310 may then be used to update each parameter value in the first machine learning model 203 using backpropagation techniques as known in the art. This process of generating training output values from the historical time series data 302, the historical video data 304, the historical time series force data 306 and backpropagating the resulting error 310 to update the parameter values of the first machine learning model 203 may be repeated until a threshold condition indicating reliability of the first machine learning model 203 is achieved. This threshold can include a predetermined or minimum number of iterations through the process and / or value indicating negligible improvement to the error 310 value from further updates to the parameter values of the first machine learning model 203.

[0100] It should be appreciated that similar training techniques may be employed to train the additional machine learning models 218, 220, and 22 as shown in FIGs. 2B and 2C.

[0101] It should also be appreciated that in some embodiments, the systems and methods described herein may include a computer system (e.g., the control system 112 or similar computing system) that is configured to obtain medical procedure data representative of use of a first removable instrument (e.g., one of the removable instruments 122) with one or more manipulators (e.g., manipulator assembly 102 and repositionable structures 120) of a computer-assisted system (e.g., computer-assisted system 100) during a medical procedure, the medical procedure data may comprises one or more data streams from one or more data sources, and a first data stream (e.g., first data stream 202A) of the one or more data streams may comprises data relating to operation of the first removable instrument by the computer- assisted system during the medical procedure. In some embodiments, a second data stream (e.g., third data stream 202C) of the one or more data streams may comprises force data for the one or more manipulators operated by the computer-assisted system.

[0102] The computer system may also be configured to input each of the one or more data streams into a respective projection layer of a first machine learning model (e.g., plurality ofprojection layers 204 of the first machine learning model 203). The respective projection layer for the first data stream includes an embedding layer and a transformer, and the first machine learning model is configured to output an instrument life metric based on outputs from the projection layers for the one ore more data streams. The instrument life metric is indicative of a current instrument life score. The computer system may also be configured to receive an instrument life metric for the first removable instrument as an output of the first machine learning model and associate the current instrument life score with the first removable instrument based on the output instrument life metric. It should be appreciated that this computer system embodiment may be modified in a similar manner described with respect to any of the methods 400, 500, 600, or 700 described below.

[0103] FIG. 4 shows a method 400 for generating instrument life metrics using an artificial intelligence model (such as first machine learning model 203). The method 400 may be executed by the control system 112, or a similar computing system having a processing unit and a memory, such as a remote server communicatively coupled to the control system 112.

[0104] At block 410, the method 400 includes the control system 112 or similar computing system obtaining medical procedure data (e.g., plurality of data streams 202) representative of use of a first removable instrument (e.g., one of the removable instruments 122) with one or more manipulators (e.g., manipulator assembly 102 and repositionable structures 120) of a computer-assisted system (e.g., computer-assisted system 100) during a medical procedure. The medical procedure data comprises a plurality of data streams (e.g., plurality of data streams 202) from one or more data sources, a first data stream (e.g., first data stream 202A) of the plurality of data streams comprises data relating to operation of the first removable instrument by the computer-assisted system during the medical procedure, and a second data stream (e.g., second data stream 202B) of the plurality of data comprises image data. The image data may be endoscope image data, operating room image data, or image data of the first removable instrument being sanitized. The data of the first data stream may include one or more of event data, kinematics data, or force data for the one or more manipulators operated by the computer-assisted system. The data of the first data stream may also include installation and / or removal information for the first removable instrument with respect to the one or more manipulators. The data of the first data stream may also include energy application information.

[0105] At block 420, the method 400 includes the control system 112 or similar computing system inputting each of the plurality of data streams into a respective projection layer (e.g.,plurality of projection layers 204) of a first machine learning model (e.g., first machine learning model 203). The respective projection layer for the first data stream includes an embedding layer and a transformer (e.g., embedding layers 210 and transformer 212). The first machine learning model is configured to output an instrument life metric based on outputs from the projection layers for the plurality of data streams. The instrument life metric is indicative of a current instrument life score. In some embodiments, the first machine learning model is trained using historical procedure data labeled with known instrument life metrics resulting from the historical procedure data. The known instrument life metrics are obtained by mechanically assessing removable instruments after use during historical medical procedures from which the historical procedure data was generated. The respective projection layer for the second data stream includes a convolutional neural network (CNN) or a large multimodal model (LMM). A third data stream (e.g., third data stream 202C) of the plurality of data streams comprises data relating to forces exerted upon a distal end of the first removable instrument during the medical procedure. The respective projection layer for the third data stream may include an embedding layer and a transformer. The respective projection layer for the first data stream may be configured to convert the data into machine readable tokens with embedded positional values. The respective projection layer may be trained using a plurality of historical time series data streams having associated time stamp information. The respective projection layer for the first data stream may include at least one attention layer trained to modify inputs into the attention layer into refined embeddings that embody contextual relationships between data points in the first data stream. The input into the attention layer may comprise outputs of the embedding layer for the first data stream.

[0106] At block 430, the method 400 includes the control system 112 or similar computing system receiving an instrument life metric for the first removable instrument as an output of the first machine learning model.

[0107] At block 440, the method 400 includes the control system 112 or similar computing system associating the current instrument life score with the first removable instrument based on the output instrument life metric. The current instrument life score may include a percentage value. The output instrument life metric may comprise an instrument life decrement value, and associating the current instrument life score with the first removable instrument may include decrementing a previous instrument life score for the first removable instrument by the instrument life decrement value to generate the current instrument life score. Associating the current instrument life score with the first removable instrument maycomprise storing the current instrument life score in a memory coupled to the first removable instrument. Associating the current instrument life score with the first removable instrument may comprise storing the current instrument life score in a record of a database associated with the first removable instrument.

[0108] The method 400 may also include the control system 112 or similar computing device presenting a user-perceptible indication of the current instrument life score on a life indicator of the first removable instrument. The method 400 may also include the control system 112 or similar computing device retrieving the current instrument life score from the record of the database; and presenting a user-perceptible indication of the retrieved current instrument life score on a display of the computer-assisted system. Presenting the user- perceptible indication of the current instrument life score on the display may include displaying a portion of a web portal interface for the database.

[0109] Fig. 5 shows a method 500 for generating instrument life metrics using an artificial intelligence model (such as first machine learning model 203). The method 500 may be executed by the control system 112, or a similar computing system having a processing unit and a memory, such as a remote server communicatively coupled to the control system 112.

[0110] At block 510, the method 500 includes the control system 112 or similar computing system obtaining a plurality of data streams (e.g., plurality of data streams 202) from one or more data sources. A first data stream (e.g., first data stream 202A) of the plurality of data streams comprises data relating to operation of one or more removable instruments (e.g., removable instruments 122) by the computer-assisted system during the medical procedure, and a second data stream (e.g., second data stream 202B) of the plurality of data comprises image data. The image data may be endoscope image data, operating room image data, or is image data of the one or more removable instruments being sanitized. The data of the first data stream may include one or more of event data, kinematics data, or force data for the one or more manipulators. The data of the first data stream may also include installation and / or removal information for the one or more removable instruments with respect to the one or more manipulators. The data of the first data stream may also include energy application information.

[0111] At block 520, the method 500 includes the control system 112 or similar computing system inputting each of the plurality of data streams into a respective projection layer (e.g., plurality of projection layers 204) of a first machine learning model (e.g., first machine learning model 203) The respective projection layer for the first data stream includes anembedding layer and a transformer (e.g., embedding layers 210 and transformer 212). The first machine learning model is configured to output an instrument life metric based on outputs from the projection layers for the plurality of data streams. The instrument life metric is indicative of a current instrument life score. In some embodiments, the first machine learning model is trained using historical procedure data labeled with known instrument life metrics resulting from the historical procedure data. The known instrument life metrics may be obtained by mechanically assessing removable instruments after use during historical medical procedures from which the historical procedure data was generated. The respective projection layer for the second data stream may include a convolutional neural network (CNN) or a large multimodal model (LMM). A third data stream (e.g., third data stream 202C) of the plurality of data streams comprises data relating to forces exerted upon a distal end of the one or more removable instruments during the medical procedure. The respective projection layer for the third data stream may include an embedding layer and a transformer. The respective projection layer for the first data stream may be configured to convert the data into machine readable tokens with embedded positional values, and, the respective projection layer may be trained using a plurality of historical time series data streams having associated time stamp information. The respective projection layer for the first data stream may include at least one attention layer trained to modify inputs into the attention layer into refined embeddings that embody contextual relationships between data points in the first data stream. The input into the attention layer may comprise outputs of the embedding layer for the first data stream.

[0112] At block 530, the method 500 includes the control system 112 or similar computing system receiving an instrument life metric for the one or more removable instruments as an output of the first machine learning model. The output instrument life metric may include a set of component metrics associated with component parts of the one or more removable instruments.

[0113] At block 540, the method 500 includes the control system 112 or similar computing system displaying an alert on a display of a control system based on the current instrument life score indicated by the output instrument life metric. The current instrument life score may include a percentage value. The output instrument life metric may comprise an instrument life decrement value, and, the method 600 includes decrementing a previous instrument life score for the one or more removable instruments by the instrument lifedecrement value to generate the current instrument life score. The alert may include a user- perceptible indication of the current instrument life score.

[0114] In some embodiments, the method 500 includes the control system 112 or similar computing system controlling operation of the one or more manipulators based on the current instrument life score. In some embodiments, the method 500 includes the control system 112 or similar computing system generating a set of tasks to be performed using the one or more manipulators, and selecting a particular manipulator / instrument to perform one or more of the set of tasks based on the current instrument life score. The method 500 may also include generating the set of tasks using a second machine learning model; inputting the current instrument life scores for the one or more removable instruments into the second machine learning model. The method 500 may include selecting the particular manipulator / instrument to perform the one or more of the set of tasks by inputting the set of tasks and the current instrument life scores into a second machine learning model that is configured to assign tasks to instruments / manipulator and inputting the current life scores for the one or more removable instruments into the second machine learning model. The set of tasks may include a set of future tasks needed to complete the medical procedure and the second machine learning model is configured to project an amount of instrument life consumed by the set of future tasks; and associate the one or more removable instruments with one or more of the set of future tasks based on the projected amount of instrument life consumed and the current instrument life score such that future low life scenarios for the one or more removable instruments are avoided. Controlling operation of the one or more manipulators based on the current instrument life score may include commanding the particular manipulator / instrument to perform the set of tasks.

[0115] In some embodiments, the method 500 includes the control system 112 or similar computing system comparing the output instrument life metric to a previous instrument life metric for the one or more removable instruments to determine a degradation rate of the one or more removable instruments, projecting a failure time for the one or more removable instruments using the degradation rate, and displaying a notification containing an indication of whether the projected failure time will occur during a remaining time for completing the medical procedure as the alert.

[0116] Fig. 6 shows a method 600 for selecting particular removable instrument to use for a procedure. The method 600 may be executed by the control system 112, or a similarcomputing system having a processing unit and a memory, such as a remote server communicatively coupled to the control system 112.

[0117] At block 610, the method 600 includes the control system 112 or similar computing system detecting an indication of a planned medical procedure that utilizes removable instruments of a particular type.

[0118] At block 620, the method 600 includes the control system 112 or similar computing system obtaining a set of current instrument life scores respectively associated with a set of removable instruments (e.g., removable instruments 122) of the particular type utilized in the planned medical procedure. The set of current instrument life scores include at least one current instrument life score that was based on an output from a first machine learning model (e.g., first machine learning model 203) when input with a plurality of data streams of medical procedure data (e.g., plurality of data streams 202), the medical procedure data is representative of use of an associated removable instrument with one or more manipulators of a computer-assisted system during a past medical procedure. In some embodiments, the first machine learning model is trained using historical procedure data labeled with known instrument life metrics resulting from the historical procedure data. The known instrument life metrics may be obtained by mechanically assessing removable instruments after use during historical medical procedures from which the historical procedure data was generated. The set of current instrument life scores may include percentage values. The method 600 may include obtaining the set of current instrument life scores from respective memories coupled to the set of removable instruments or obtaining the set of current instrument life scores from respective records in a database associated with the set of removable instruments.

[0119] At block 630, the method 600 includes the control system 112 or similar computing system selecting a particular removable instrument from the set of removable instruments for use in the planned medical procedure based on the obtained set of current instrument life scores. Selecting the particular removable instrument may comprises determining which of the obtained set of current instrument life scores are sufficient to be used for the planned medical procedure and selecting the particular removable instrument as one of the set of removable instruments determined to be sufficient. Determining which of the obtained set of current instrument life scores are sufficient may comprise inputting new medical procedure information associated with the planned medical procedure and the set of obtained current instrument life scores into a second machine learning model (e.g., further machine learning model 222) configured to estimate a respective amount of life consumed by each one of theset of removable instruments when used to perform the planned medical procedure. The particular removable instrument may be one of the set of removable instruments determined to be sufficient and for which the respective amount of life consumed is closest to the current instrument life score for the particular removable instrument. The particular removable instrument may be the one of the set of removable instruments determined to be sufficient and for which the respective amount of life consumed is the least.

[0120] At block 640, the method 600 includes the control system 112 or similar computing system displaying a notification indicating that the particular removable instrument is to be used for the planned medical procedure.

[0121] The method 600 includes the control system 112 or similar computing system presenting user-perceptible indications of the obtained set of current instrument life scores on a display of the computer-assisted system. Presenting the user-perceptible indications of the current instrument life score includes displaying a portion of a web portal interface for the database on the display.

[0122] FIG. 7 shows a method 700 for evaluating a particular removable instrument for use in a procedure. The method 700 may be executed by the control system 112, or a similar computing system having a processing unit and a memory, such as a remote server communicatively coupled to the control system 112.

[0123] At block 710, the method 700 includes obtaining a current instrument life score for a particular removable instrument selected for use in a planned medical procedure (e.g., from the database 136 and / or the instrument memory 168). The current instrument life score is based on an output from a first machine learning model when input with a plurality of data streams of medical procedure data; and the medical procedure data is representative of use of the particular removable instrument with one or more manipulators of a computer-assisted system during a past medical procedure.

[0124] At block 720, the method 700 includes evaluating the suitability of the particular removable instrument for use in the planned medical procedure based upon the obtained current instrument life score. In some embodiments, evaluating the suitability of the particular removable instrument for use in the planned medical may include inputting new medical procedure information associated with the planned medical procedure and the obtained current instrument life score into a second machine learning model configured to estimate a respective amount of life to be consumed by the particular removable instrument when used to perform the planned medical procedure.

[0125] At block 730, the method 700 includes displaying a notification indicating that the particular removable instrument is suitable or not suitable to be used for the planned medical procedure.

[0126] It is understood that the blocks of the methods 400, 500, 600, and 700 need not occur strictly in the order shown.

[0127] Although the systems, methods, devices, and components thereof, have been described in terms of exemplary embodiments, they are not limited thereto. The detailed description is to be construed as exemplary only and does not describe every possible embodiment of the invention because describing every possible embodiment would be impractical, if not impossible. Numerous alternative embodiments could be implemented, using either current technology or technology developed after the filing date of this patent that would still fall within the scope of the claims defining the invention.

[0128] Those skilled in the art will recognize that a wide variety of modifications, alterations, and combinations can be made with respect to the above-described embodiments without departing from the scope of the invention, and that such modifications, alterations, and combinations are to be viewed as being within the ambit of the inventive concept.

Claims

What is claimed is:

1. A computer system comprising: one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the computer system to: obtain medical procedure data representative of use of a first removable instrument with one or more manipulators of a computer-assisted system during a medical procedure, wherein: the medical procedure data comprises a plurality of data streams from one or more data sources, a first data stream of the plurality of data streams comprises data relating to operation of the first removable instrument by the computer- assisted system during the medical procedure, and a second data stream of the plurality of data comprises image data; input each of the plurality of data streams into a respective projection layer of a first machine learning model, wherein: the respective projection layer for the first data stream includes an embedding layer and a transformer, and the first machine learning model is configured to output an instrument life metric based on outputs from the projection layers for the plurality of data streams, wherein the instrument life metric is indicative of a current instrument life score; receive an instrument life metric for the first removable instrument as an output of the first machine learning model; and associate the current instrument life score with the first removable instrument based on the output instrument life metric.

2. The computer system of claim 1, wherein the first machine learning model is trained using historical procedure data labeled with known instrument life metrics resulting from the historical procedure data.

3. The computer system of claim 2, wherein the known instrument life metrics are obtained by mechanically assessing removable instruments after use during historical medical procedures from which the historical procedure data was generated.

4. The computer system of claim 1, wherein the image data is endoscope image data or operating room image data.

5. The computer system of claim 1, wherein the image data is image data of the first removable instrument being sanitized.

6. The computer system of claim 1, wherein the respective projection layer for the second data stream includes a convolutional neural network (CNN) or a large multimodal model (LMM).

7. The computer system of claim 1, wherein a third data stream of the plurality of data streams comprises data relating to forces exerted upon a distal end of the first removable instrument during the medical procedure.

8. The computer system of claim 7, wherein the respective projection layer for the third data stream includes an embedding layer and a transformer.

9. The computer system of claim 1, wherein the respective projection layer for the first data stream is configured to convert the data into machine readable tokens with embedded positional values, and, wherein the respective projection layer is trained using a plurality of historical time series data streams having associated time stamp information.

10. The computer system of claim 1, wherein the respective projection layer for the first data stream includes at least one attention layer trained to modify inputs into the attention layer into refined embeddings that embody contextual relationships between data points in the first data stream.

11. The computer system of claim 10, wherein the input into the attention layer comprises outputs of the embedding layer for the first data stream.

12. The computer system of claim 1, wherein the data of the first data stream includes one or more of event data, kinematics data, or force data for the one or more manipulators operated by the computer-assisted system.

13. The computer system of claim 1, wherein the data of the first data stream includes installation and / or removal information for the first removable instrument with respect to the one or more manipulators.

14. The computer system of claim 1, wherein the data of the first data stream includes energy application information.

15. The computer system of claim 1, wherein the current instrument life score includes a percentage value.

16. The computer system of claim 1, wherein the output instrument life metric comprises an instrument life decrement value, and, wherein execution of the instructions associates the current instrument life score with the first removable instrument by decrementing a previous instrument life score for the first removable instrument by the instrument life decrement value to generate the current instrument life score.

17. The computer system of claim 1, wherein associating the current instrument life score with the first removable instrument comprises storing the current instrument life score in a memory coupled to the first removable instrument.

18. The computer system of claim 17, wherein the first removable instrument includes a life indicator configured to present a user-perceptible indication of the current instrument life score.

19. The computer system of claim 1, wherein associating the current instrument life score with the first removable instrument comprises storing the current instrument life score in a record of a database associated with the first removable instrument.

20. The computer system of claim 19, wherein the instructions, when executed by the one or more processors, cause the computer system to: retrieve the current instrument life score from the record of the database; and present a user-perceptible indication of the retrieved current instrument life score on a display of the computer-assisted system.

21. The computer system of claim 20, wherein the display presents the user-perceptible indication of the current instrument life score by displaying a portion of a web portal interface for the database.

22. A computer-implemented method comprising: obtaining medical procedure data representative of use of a first removable instrument with one or more manipulators of a computer-assisted system during a medical procedure, wherein: the medical procedure data comprises a plurality of data streams from one or more data sources, a first data stream of the plurality of data streams comprises data relating to operation of the first removable instrument by the computer- assisted system during the medical procedure, and a second data stream of the plurality of data comprises image data; inputting each of the plurality of data streams into a respective projection layer of a first machine learning model, wherein: the respective projection layer for the first data stream includes an embedding layer and a transformer, and the first machine learning model is configured to output an instrument life metric based on outputs from the projection layers for the plurality of data streams, wherein the instrument life metric is indicative of a current instrument life score; receiving an instrument life metric for the first removable instrument as an output of the first machine learning model; and associating the current instrument life score with the first removable instrument based on the output instrument life metric.

23. The computer-implemented method of claim 22, wherein the first machine learning model is trained using historical procedure data labeled with known instrument life metrics resulting from the historical procedure data24. The computer-implemented method of claim 23, wherein the known instrument life metrics are obtained by mechanically assessing removable instruments after use during historical medical procedures from which the historical procedure data was generated.

25. The computer- implemented method of claim 22, wherein the image data is endoscope image data or operating room image data.

26. The computer- implemented method of claim 22, wherein the image data is image data of the first removable instrument being sanitized.

27. The computer- implemented method of claim 22, wherein the respective projection layer for the second data stream includes a convolutional neural network (CNN) or a large multimodal model (LMM).

28. The computer- implemented method of claim 22, wherein a third data stream of the plurality of data streams comprises data relating to forces exerted upon a distal end of the first removable instrument during the medical procedure.

29. The computer-implemented method of claim 28, wherein the respective projection layer for the third data stream includes an embedding layer and a transformer.

30. The computer-implemented method of claim 22, wherein the respective projection layer for the first data stream is configured to convert the data into machine readable tokens with embedded positional values, and, wherein the respective projection layer is trained using a plurality of historical time series data streams having associated time stamp information.

31. The computer- implemented method of claim 22, wherein the respective projection layer for the first data stream includes at least one attention layer trained to modify inputs intothe attention layer into refined embeddings that embody contextual relationships between data points in the first data stream.

32. The computer-implemented method of claim 31, wherein the input into the attention layer comprises outputs of the embedding layer for the first data stream.

33. The computer-implemented method of claim 22, wherein the data of the first data stream includes one or more of event data, kinematics data, or force data for the one or more manipulators operated by the computer-assisted system.

34. The computer-implemented method of claim 22, wherein the data of the first data stream includes installation and / or removal information for the first removable instrument with respect to the one or more manipulators.

35. The computer- implemented method of claim 22, wherein the data of the first data stream includes energy application information.

36. The computer-implemented method of claim 22, wherein the current instrument life score includes a percentage value.

37. The computer- implemented method of claim 22, wherein the output instrument life metric comprises an instrument life decrement value, and, wherein associating the current instrument life score with the first removable instrument includes decrementing a previous instrument life score for the first removable instrument by the instrument life decrement value to generate the current instrument life score.

38. The computer- implemented method of claim 22, wherein associating the current instrument life score with the first removable instrument comprises storing the current instrument life score in a memory coupled to the first removable instrument.

39. The computer-implemented method of claim 38, further comprising presenting a user- perceptible indication of the current instrument life score on a life indicator of the first removable instrument.

40. The computer-implemented method of claim 22, wherein associating the current instrument life score with the first removable instrument comprises storing the current instrument life score in a record of a database associated with the first removable instrument.

41. The computer-implemented method of claim 40, further comprising retrieving the current instrument life score from the record of the database; and presenting a user-perceptible indication of the retrieved current instrument life score on a display of the computer-assisted system.

42. The computer-implemented method of claim 41, wherein presenting the user- perceptible indication of the current instrument life score on the display include displaying a portion of a web portal interface for the database.

43. A non-transitory machine-readable medium comprising a plurality of machine- readable instructions that when executed by one or more processors are adapted to cause the one or more processors to perform the method of any one of claims 22-42.

44. A computer system comprising: one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the computer system to: detect an indication of a planned medical procedure that utilizes removable instruments of a particular type; obtain a set of current instrument life scores respectively associated with a set of removable instruments of the particular type utilized in the planned medical procedure, wherein: the set of current instrument life scores include at least one current instrument life score that was based on an output from a first machine learning model when input with a plurality of data streams of medical procedure data; andthe medical procedure data is representative of use of an associated removable instrument with one or more manipulators of a computer-assisted system during a past medical procedure; select a particular removable instrument from the set of removable instruments for use in the planned medical procedure based on the obtained set of current instrument life scores; and display a notification indicating that the particular removable instrument is to be used for the planned medical procedure.

45. The computer system of claim 44, wherein the first machine learning model is trained using historical procedure data labeled with known instrument life metrics resulting from the historical procedure data46. The computer system of claim 45, wherein the known instrument life metrics are obtained by mechanically assessing removable instruments after use during historical medical procedures from which the historical procedure data was generated.

47. The computer system of claim 44, wherein to select the particular removable instrument, the instructions cause the one or more processors to: determine which of the obtained set of current instrument life scores are sufficient to be used for the planned medical procedure; and select the particular removable instrument as one of the set of removable instruments determined to be sufficient.

48. The computer system of claim 47, wherein the instructions cause the one or more processors to determine which of the obtained set of current instrument life scores are sufficient by: inputting new medical procedure information associated with the planned medical procedure and the set of obtained current instrument life scores into a second machine learning model configured to estimate a respective amount of life consumed by each one of the set of removable instruments when used to perform the planned medical procedure.

49. The computer system of claim 48, wherein the particular removable instrument is one of the set of removable instruments determined to be sufficient and for which the respective amount of life consumed is closest to the current instrument life score for the particular removable instrument.

50. The computer system of claim 48, wherein the particular removable instrument is the one of the set of removable instruments determined to be sufficient and for which the respective amount of life consumed is the least.

51. The computer system of claim 44, wherein the set of current instrument life scores include percentage values.

52. The computer system of claim 44, wherein the set of current instrument life scores are obtained from respective memories coupled to the set of removable instruments.

53. The computer system of claim 44, wherein the set of current instrument life scores are obtained from respective records in a database associated with the set of removable instruments.

54. The computer system of claim 53, wherein the instructions, when executed by the one or more processors, cause the computer system to: present user-perceptible indications of the obtained set of current instrument life scores on a display of the computer-assisted system.

55. The computer system of claim 54, wherein the display presents the user-perceptible indications of the current instrument life score by displaying a portion of a web portal interface for the database.

56. A computer-implemented method comprising: detecting an indication of a planned medical procedure that utilizes removable instruments of a particular type;obtaining a set of current instrument life scores respectively associated with a set of removable instruments of the particular type utilized in the planned medical procedure, wherein: the set of current instrument life scores include at least one current instrument life score that was based on an output from a first machine learning model when input with a plurality of data streams of medical procedure data; the medical procedure data is representative of use of an associated removable instrument with one or more manipulators of a computer-assisted system during a past medical procedure; and the first machine learning model is trained using historical procedure data labeled with known instrument life metrics resulting from the historical procedure data; selecting a particular removable instrument from the set of removable instruments for use in the planned medical procedure based on the obtained set of current instrument life scores; and displaying a notification indicating that the particular removable instrument is to be used for the planned medical procedure.

57. The computer-implemented method of claim 56, wherein to selecting the particular removable instrument comprises: determining which of the obtained set of current instrument life scores are sufficient to be used for the planned medical procedure; and selecting the particular removable instrument as one of the set of removable instruments determined to be sufficient.

58. The computer-implemented method of claim 57, wherein determining which of the obtained set of current instrument life scores are sufficient comprises: inputting new medical procedure information associated with the planned medical procedure and the set of obtained current instrument life scores into a second machine learning model configured to estimate a respective amount of life consumed by each one of the set of removable instruments when used to perform the planned medical procedure.

59. The computer-implemented method of claim 58, wherein the particular removable instrument is one of the set of removable instruments determined to be sufficient and for which the respective amount of life consumed is closest to the current instrument life score for the particular removable instrument.

60. The computer-implemented method of claim 58, wherein the particular removable instrument is the one of the set of removable instruments determined to be sufficient and for which the respective amount of life consumed is the least.

61. The computer-implemented method of claim 56, wherein the set of current instrument life scores include percentage values.

62. The computer-implemented method of claim 56, further comprising obtaining the set of current instrument life scores from respective memories coupled to the set of removable instruments.

63. The computer-implemented method of claim 56, further comprising obtaining the set of current instrument life scores from respective records in a database associated with the set of removable instruments.

64. The computer-implemented method of claim 63, further comprising: presenting user-perceptible indications of the obtained set of current instrument life scores on a display of the computer-assisted system.

65. The computer- implemented method of claim 64, wherein presenting the user- perceptible indications of the current instrument life score includes displaying a portion of a web portal interface for the database on the display.

66. The computer-implemented method of claim 56, wherein the known instrument life metrics are obtained by mechanically assessing removable instruments after use during historical medical procedures from which the historical procedure data was generated.

67. A non-transitory machine-readable medium comprising a plurality of machine- readable instructions that when executed by one or more processors are adapted to cause the one or more processors to perform the method of any one of claims 56-66.

68. A computer-assisted system comprising: one or more manipulators; one or more removable instruments coupled to the one or more manipulators; and a control system operably coupled to the one or more manipulators, wherein the control system is configured to: obtain a plurality of data streams from one or more data sources, wherein: a first data stream of the plurality of data streams comprises data relating to operation of the one or more removable instruments by the computer-assisted system during a medical procedure, and a second data stream of the plurality of data comprises image data; input each of the plurality of data streams into a respective projection layer of a first machine learning model, wherein: the respective projection layer for the first data stream includes an embedding layer and a transformer, and the first machine learning model is configured to output an instrument life metric based on outputs from the projection layers for the plurality of data streams, wherein the instrument life metric is indicative of a current instrument life score; receive an instrument life metric for the one or more removable instruments as an output of the first machine learning model; and display an alert on a display of the control system based on the current instrument life score indicated by the output instrument life metric.

69. The computer-assisted system of claim 68, wherein the first machine learning model is trained using historical procedure data labeled with known instrument life metrics resulting from the historical procedure data.

70. The computer-assisted system of claim 69, wherein the known instrument life metrics are obtained by mechanically assessing removable instruments after use during historical medical procedures from which the historical procedure data was generated.

71. The computer-assisted system of claim 68, wherein the control system is configured to control operation of the one or more manipulators based on the current instrument life score.

72. The computer-assisted system of claim 71, wherein the control system is further configured to: generate a set of tasks to be performed using the one or more manipulators; and select a particular manipulator / instrument to perform one or more of the set of tasks based on the current instrument life score.

73. The computer-assisted system of claim 72, wherein the control system generates the set of tasks using a second machine learning model.

74. The computer-assisted system of claim 73, wherein the current instrument life scores for the one or more removable instruments are inputs into the second machine learning model.

75. The computer-assisted system of claim 72, wherein the control system is configured to select the particular manipulator / instrument to perform the one or more of the set of tasks by inputting the set of tasks and the current instrument life scores into a second machine learning model that is configured to assign tasks to instruments / manipulator.

76. The computer-assisted system of claim 75, wherein the current life scores for the one or more removable instruments are inputs into the second machine learning model.

77. The computer-assisted system of claim 75, wherein the set of tasks includes a set of future tasks needed to complete the medical procedure and the second machine learning model is configured to: project an amount of instrument life consumed by the set of future tasks; andassociate the one or more removable instruments with one or more of the set of future tasks based on the projected amount of instrument life consumed and the current instrument life score such that future low life scenarios for the one or more removable instruments are avoided.

78. The computer-assisted system of claim 72, wherein the control system is configured to control operation of the one or more manipulators based on the current instrument life score by commanding the particular manipulator / instrument to perform the set of tasks.

79. The computer-assisted system of claim 68, wherein the control system is further configured to: compare the output instrument life metric to a previous instrument life metric for the one or more removable instruments to determine a degradation rate of the one or more removable instruments; project a failure time for the one or more removable instruments using the degradation rate; and display a notification containing an indication of whether the projected failure time will occur during a remaining time for completing the medical procedure as the alert.

80. The computer-assisted system of claim 68, wherein the output instrument life metric includes a set of component metrics associated with component parts of the one or more removable instruments.

81. The computer-assisted system of claim 68, wherein the image data is endoscope image data or operating room image data.

82. The computer-assisted system of claim 68, wherein the image data is image data of the one or more removable instruments being sanitized.

83. The computer-assisted system of claim 68, wherein the respective projection layer for the second data stream includes a convolutional neural network (CNN) or a large multimodal model (LMM).

84. The computer-assisted system of claim 68, wherein a third data stream of the plurality of data streams comprises data relating to forces exerted upon a distal end of the one or more removable instruments during the medical procedure.

85. The computer-assisted system of claim 84, wherein the respective projection layer for the third data stream includes an embedding layer and a transformer.

86. The computer-assisted system of claim 68, wherein the respective projection layer for the first data stream is configured to convert the data into machine readable tokens with embedded positional values, and, wherein the respective projection layer is trained using a plurality of historical time series data streams having associated time stamp information.

87. The computer-assisted system of claim 68, wherein the respective projection layer for the first data stream includes at least one attention layer trained to modify inputs into the attention layer into refined embeddings that embody contextual relationships between data points in the first data stream.

88. The computer-assisted system of claim 87, wherein the input into the attention layer comprises outputs of the embedding layer for the first data stream.

89. The computer-assisted system of claim 68, wherein the data of the first data stream includes one or more of event data, kinematics data, or force data for the one or more manipulators operated by the computer-assisted system.

90. The computer-assisted system of claim 68, wherein the data of the first data stream includes installation and / or removal information for the one or more removable instruments with respect to the one or more manipulators.

91. The computer-assisted system of claim 68, wherein the data of the first data stream includes energy application information.

92. The computer-assisted system of claim 68, wherein the current instrument life score includes a percentage value.

93. The computer-assisted system of claim 68, wherein the output instrument life metric comprises an instrument life decrement value, and, wherein the control system is configured to decrement a previous instrument life score for the one or more removable instruments by the instrument life decrement value to generate the current instrument life score.

94. The computer-assisted system of claim 68, wherein the alert includes a user- perceptible indication of the current instrument life score.

95. A computer-implemented method comprising: obtaining a plurality of data streams from one or more data sources, wherein: a first data stream of the plurality of data streams comprises data relating to operation of one or more removable instruments by a computer- assisted system during a medical procedure, and a second data stream of the plurality of data comprises image data; inputting each of the plurality of data streams into a respective projection layer of a first machine learning model, wherein: the respective projection layer for the first data stream includes an embedding layer and a transformer, and the first machine learning model is configured to output an instrument life metric based on outputs from the projection layers for the plurality of data streams, wherein the instrument life metric is indicative of a current instrument life score; receiving an instrument life metric for the one or more removable instruments as an output of the first machine learning model; and displaying an alert on a display of a control system based on the current instrument life score indicated by the output instrument life metric.

96. The computer- implemented method of claim 95, wherein the first machine learning model is trained using historical procedure data labeled with known instrument life metrics resulting from the historical procedure data.

97. The computer-implemented method of claim 96, wherein the known instrument life metrics are obtained by mechanically assessing removable instruments after use during historical medical procedures from which the historical procedure data was generated.

98. The computer-implemented method of claim 95, further comprising controlling operation of one or more manipulators of the computer-assisted system based on the current instrument life score.

99. The computer-implemented method of claim 98, further comprising: generating a set of tasks to be performed using the one or more manipulators; and selecting a particular manipulator / instrument to perform one or more of the set of tasks based on the current instrument life score.

100. The computer-implemented method of claim 99, further comprising generating the set of tasks using a second machine learning model.

101. The computer- implemented method of claim 100, further comprising inputting the current instrument life scores for the one or more removable instruments into the second machine learning model.

102. The computer-implemented method of claim 99, further comprising selecting the particular manipulator / instrument to perform the one or more of the set of tasks by inputting the set of tasks and the current instrument life scores into a second machine learning model that is configured to assign tasks to instruments / manipulator.

103. The computer-implemented method of claim 102, further comprising inputting the current life scores for the one or more removable instruments into the second machine learning model.

104. The computer-implemented method of claim 102, wherein the set of tasks includes a set of future tasks needed to complete the medical procedure and the second machine learning model is configured to: project an amount of instrument life consumed by the set of future tasks; andassociate the one or more removable instruments with one or more of the set of future tasks based on the projected amount of instrument life consumed and the current instrument life score such that future low life scenarios for the one or more removable instruments are avoided.

105. The computer-implemented method of claim 99, wherein controlling operation of the one or more manipulators based on the current instrument life score includes commanding the particular manipulator / instrument to perform the set of tasks.

106. The computer-implemented method of claim 95, further comprising: comparing the output instrument life metric to a previous instrument life metric for the one or more removable instruments to determine a degradation rate of the one or more removable instruments; projecting a failure time for the one or more removable instruments using the degradation rate; and displaying a notification containing an indication of whether the projected failure time will occur during a remaining time for completing the medical procedure as the alert.

107. The computer-implemented method of claim 95, wherein the output instrument life metric includes a set of component metrics associated with component parts of the one or more removable instruments.

108. The computer-implemented method of claim 95, wherein the image data is endoscope image data or operating room image data.

109. The computer- implemented method of claim 95, wherein the image data is image data of the one or more removable instruments being sanitized.

110. The computer-implemented method of claim 95, wherein the respective projection layer for the second data stream includes a convolutional neural network (CNN) or a large multimodal model (LMM).

111. The computer-implemented method of claim 95, wherein a third data stream of the plurality of data streams comprises data relating to forces exerted upon a distal end of the one or more removable instruments during the medical procedure.

112. The computer-implemented method of claim 111, wherein the respective projection layer for the third data stream includes an embedding layer and a transformer.

113. The computer-implemented method of claim 95, wherein the respective projection layer for the first data stream is configured to convert the data into machine readable tokens with embedded positional values, and, wherein the respective projection layer is trained using a plurality of historical time series data streams having associated time stamp information.

114. The computer-implemented method of claim 95, wherein the respective projection layer for the first data stream includes at least one attention layer trained to modify inputs into the attention layer into refined embeddings that embody contextual relationships between data points in the first data stream.

115. The computer- implemented method of claim 114, wherein the input into the attention layer comprises outputs of the embedding layer for the first data stream.

116. The computer- implemented method of claim 95, wherein the data of the first data stream includes one or more of event data, kinematics data, or force data for one or more manipulators of the computer-assisted system.

117. The computer- implemented method of claim 95, wherein the data of the first data stream includes installation and / or removal information for the one or more removable instruments with respect to one or more manipulators of the computer-assisted system.

118. The computer- implemented method of claim 95, wherein the data of the first data stream includes energy application information.

119. The computer- implemented method of claim 95, wherein the current instrument life score includes a percentage value.

120. The computer-implemented method of claim 95, wherein the output instrument life metric comprises an instrument life decrement value, and, wherein the method further comprises decrementing a previous instrument life score for the one or more removable instruments by the instrument life decrement value to generate the current instrument life score.

121. The computer-implemented method of claim 95, wherein the alert includes a user- perceptible indication of the current instrument life score.

122. A non-transitory machine-readable medium comprising a plurality of machine- readable instructions that when executed by one or more processors are adapted to cause the one or more processors to perform the method of any one of claims 95-121.

123. A computer system comprising: one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the computer system to: obtain a current instrument life score for a particular removable instrument selected for use in a planned medical procedure, wherein: the current instrument life score is based on an output from a first machine learning model when input with a plurality of data streams of medical procedure data; and the medical procedure data is representative of use of the particular removable instrument with one or more manipulators of a computer-assisted system during a past medical procedure; evaluate suitability of the particular removable instrument for use in the planned medical procedure based upon the obtained current instrument life score; and display a notification indicating that the particular removable instrument is suitable or not suitable to be used for the planned medical procedure.

124. The computer system of claim 123, wherein the instructions cause the one or more processors evaluate the suitability of the particular removable instrument for use in the planned medical procedure by: inputting new medical procedure information associated with the planned medical procedure and the obtained current instrument life score into a second machine learning model configured to estimate a respective amount of life to be consumed by the particular removable instrument when used to perform the planned medical procedure.

125. A computer-implemented method comprising: obtaining a current instrument life score for a particular removable instrument selected for use in a planned medical procedure, wherein: the current instrument life score is based on an output from a first machine learning model when input with a plurality of data streams of medical procedure data; and the medical procedure data is representative of use of the particular removable instrument with one or more manipulators of a computer-assisted system during a past medical procedure; evaluating suitability of the particular removable instrument for use in the planned medical procedure based upon the obtained current instrument life score; and displaying a notification indicating that the particular removable instrument is suitable or not suitable to be used for the planned medical procedure.

126. The computer-implemented method of claim 125, wherein evaluating the suitability of the particular removable instrument for use in the planned medical comprises: inputting new medical procedure information associated with the planned medical procedure and the obtained current instrument life score into a second machine learning model configured to estimate a respective amount of life to be consumed by the particular removable instrument when used to perform the planned medical procedure.

127. A non-transitory machine-readable medium comprising a plurality of machine- readable instructions that when executed by one or more processors are adapted to cause the one or more processors to perform the method of any one of claims 125 and 126.

Images