Systems and methods for generating pre-operative and intra operative guidance using adaptive artificial intelligence

Abstract

A system may obtain pre-operative multi-modal data relating to a planned medical procedure from one or more data sources and analyze, via an instrument guidance machine learning model, the pre-operative multi-modal data to generate an instrument guidance. An instrument guidance constitution is input into the instrument guidance machine learning model. The instrument guidance constitution includes rules that control how the instrument guidance machine learning model generates the instrument guidance. The system may receive the instrument guidance as an output of the instrument guidance machine learning model. The instrument guidance may include an indication of a set of instruments to use when performing the planned medical procedure. The system may provide the instrument guidance to a display unit to direct pre-operative preparation for performing the planned medical procedure.

Description

Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519SYSTEMS AND METHODS FOR GENERATING PRE-OPERATIVE AND INTRA OPERATIVE GUIDANCE USING ADAPTIVE ARTIFICIAL INTELLIGENCE CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of the filing date of provisional U.S. Patent Application No. 63 / 708,145 entitled “SYSTEMS AND METHODS FOR GENERATING PRE-OPERATIVE AND INTRA OPERATIVE GUIDANCE USING ADAPTIVE ARTIFICIAL INTELLIGENCE,” filed on October 16, 2024. The entire contents of the provisional application are hereby expressly incorporated herein by reference.TECHNICAL FIELD

[0002] The present disclosure relates generally to providing medical procedure guidance and more particularly to generating pre-operative and intra operative guidance for a medical procedure using adaptive artificial intelligence.BACKGROUND

[0003] Medical procedures performed by individuals alone or in conjunction with computer-assisted systems may utilize multiple different possible combinations of instruments. Currently, the setup for these procedures involves extensive planning by surgeons or other personnel. For example, the surgeons or other personnel may review patient medical records and other data to develop a specialized plan for the procedure that includes particular instruments to be used during the procedure. Additionally, the plan may include details on the medical and technical support teams used to set up the operating room and directions for how to arrange system components and prepare different surgical tools.

[0004] However, this setup process can be costly and time-consuming because different surgeons may prefer different instruments for the same procedure and / or change their preferences between procedures. Consequently, care teams might open non-preferred, disposable instruments unnecessarily, leading to waste and increased costs.

[0005] To manage instrument selection, care teams have traditionally maintained “pick lists” that list instruments based on a particular procedure being performed. However, these pick lists are static and do not account for the different preferences across personnel. Additionally, these static pick lists fail to account for the specific patient conditions, and may result in an instrument needed to respond to an adverse event not being prepared for usage.

[0006] Therefore, there is a need for a system offering better guidance in automatically selecting instrument sets in advance of planned surgical procedures and for providing intraoperative adjustments to the instrument selections to account for unplanned conditions thatIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 may arise during the procedure. Such techniques may allow improved surgical outcomes, improved task workflows, reduced setup time for a procedure, and reduce instrument waste.SUMMARY

[0007] In some aspects, the techniques described herein relate to a computer system including: one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the computer system to: obtain pre-operative multi-modal data relating to a planned medical procedure from one or more data sources; analyze, via an instrument guidance machine learning model, the pre-operative multi-modal data to generate an instrument guidance, wherein: an instrument guidance constitution is input into the instrument guidance machine learning model, and the instrument guidance constitution includes rules that control how the instrument guidance machine learning model generates the instrument guidance, receive the instrument guidance as an output of the instrument guidance machine learning model, wherein the instrument guidance includes an indication of a set of instruments to use when performing the planned medical procedure; and provide the instrument guidance to a display unit to direct pre-operative preparation for performing the planned medical procedure.

[0008] In some aspects, the techniques described herein relate to a computer-implemented method including: obtaining pre-operative multi-modal data relating to a planned medical procedure from one or more data sources; analyzing, via an instrument guidance machine learning model, the pre-operative multi-modal data to generate an instrument guidance, wherein: an instrument guidance constitution is input into the instrument guidance machine learning model, and the instrument guidance constitution includes rules that control how the instrument guidance machine learning model generates the instrument guidance, receiving the instrument guidance as an output of the instrument guidance machine learning model, wherein the instrument guidance includes an indication of a set of instruments to use when performing the planned medical procedure; and providing the instrument guidance to a display unit to direct pre-operative preparation for performing the planned medical procedure.

[0009] In some further aspects, a non-transitory machine -readable medium comprising a plurality of machine-readable instructions that when executed by one or more processors are adapted to cause the one or more processors to perform any of the methods described herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG. 1 is a diagram of a computer-assisted system in accordance with one or more embodiments.Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519

[0011] FIG. 2A is a schematic diagram of a system for generating pre-operative instrument guidance.

[0012] FIG. 2B is a schematic diagram of a system for generating task segmentations for the pre-operative instrument guidance generated by the system of FIG. 2A.

[0013] FIG. 2C is a schematic diagram of a system for generating video snippets portions of the pre-operative instrument guidance of FIG. 2A.

[0014] FIGs. 3A-3E are example pre-operative instrument guidance generated by the systems of FIGs. 2A, 2B, and / or 2C.

[0015] FIG. 4A is a schematic diagram of a system for generating intra-operative instrument guidance.

[0016] FIG. 4B is a schematic diagram of a system for generating updated instrument guidance.

[0017] FIG. 5 is a flow diagram of a method for generating instrument guidance utilizing adaptive artificial intelligence.

[0018] Examples of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, wherein showings therein are for purposes of illustrating examples of the present disclosure and not for purposes of limiting the same.DETAILED DESCRIPTION

[0019] In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.

[0020] Further, the terminology in this description is not intended to limit the invention. For example, spatially relative terms-such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like-may be used to describe the relation of one element orIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 feature to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass different positions (i.e., locations) and orientations (i.e., rotational placements) of the elements or their operation in addition to the position and orientation shown in the figures. For example, if the content of one of the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features. A device may be otherwise oriented and the spatially relative descriptors used herein interpreted accordingly. Likewise, descriptions of movement along and around various axes include various special element positions and orientations. In addition, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context indicates otherwise. Additionally, the terms “comprises”, “comprising”, “includes”, and the like specify the presence of stated features, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

[0021] Elements described in detail with reference to one embodiment, implementation, system, or module may, whenever practical, be included in other embodiments, implementations, systems, or modules in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or implementation non-functional, or unless two or more of the elements provide conflicting functions.

[0022] In some instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0023] This disclosure describes various devices, elements, and portions of computer- assisted systems and elements in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an element or a portion of an element (e.g., three degrees of translational freedom in a three-dimensional space, such as along Cartesian x-, y-, and z-coordinates). As used herein, the term “orientation” refers to theIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 rotational placement of an element or a portion of an element (e.g., three degrees of rotational freedom in three-dimensional space, such as about roll, pitch, and yaw axes, represented in angle-axis, rotation matrix, quaternion representation, and / or the like). As used herein, and for a device with a kinematic series, such as with a repositionable structure with a plurality of links coupled by one or more joints, the term “proximal” refers to a direction toward a base of the kinematic series, and “distal” refers to a direction away from the base along the kinematic series.

[0024] As used herein, the term “pose” refers to the multi-degree of freedom (DOF) spatial position and orientation of a coordinate system of interest attached to a rigid body. In general, a pose includes a pose variable for each of the DOFs in the pose. For example, a full 6-DOF pose for a rigid body in three-dimensional space would include 6 pose variables corresponding to the 3 positional DOFs (e.g., x, y, and z) and the 3 orientational DOFs (e.g., roll, pitch, and yaw). A 3-DOF position only pose would include only pose variables for the 3 positional DOFs. Similarly, a 3-DOF orientation only pose would include only pose variables for the 3 rotational DOFs. Further, a velocity of the pose captures the change in pose over time (e.g., a first derivative of the pose). For a full 6-DOF pose of a rigid body in three- dimensional space, the velocity would include 3 translational velocities and 3 rotational velocities. Poses with other numbers of DOFs would have a corresponding number of velocities translational and / or rotational velocities.

[0025] This disclosure occasionally refers to the disclosed techniques being applied to “patients” undergoing a “medical procedure” or “operation.” It should be appreciated that these references are not intended to limit the application of the disclosed techniques to applied medicine contexts. For example, the described techniques can be applied to facilitate physician training, equipment testing and / or calibration, and / or other contexts. Accordingly, any reference to the term “patient” is done for ease of explanation and also envisions the application of the described techniques to a generic “subject.”

[0026] The word “task” is used herein to refer to a discrete portion of procedure that may be autonomously, semi-autonomously, or manually implemented in furtherance of a procedure. For example, a task may be to move an endoscope to a particular portion, to advance an instrument to a particular depth, to replace an instrument coupled to a manipulator, and so on. In some embodiments, a task is associated with component tasks to accomplish an overall goal. For example, a task to analyze a worksite may includeIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 component tasks related to moving an endoscope to view the worksite, advancing an instrument to predetermined depth, and enabling a functionality supported by the instrument.

[0027] Aspects of this disclosure are described in reference to computer-assisted systems, which can include devices that are teleoperated, externally manipulated, autonomous, semiautonomous, and / or the like. Further, aspects of this disclosure are described in terms of an implementation using a teleoperated surgical system, such as the da Vinci® Surgical System commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that inventive aspects disclosed herein may be embodied and implemented in various ways, including teleoperated and non-teleoperated, and medical and non-medical embodiments and implementations. Implementations on da Vinci® Surgical Systems are merely exemplary and are not to be considered as limiting the scope of the inventive aspects disclosed herein. For example, techniques described with reference to surgical instruments and surgical methods may be used in other contexts. Thus, the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperated systems. As further examples, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, and / or the like. Additional example applications include use for procedures on tissue removed from human or animal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.

[0028] The disclosure generally relates to systems and methods for training and intelligently utilizing adaptive artificial intelligence (Al) or machine learning (ML) techniques and models to generate pre-operative guidance for a planned medical procedure and intra-operative guidance during execution of the planned medical procedure. In general, the pre-operative or intra-operative guidance may provide setup, planning, and operational assist tools for the planned medical procedure that document, for example, instruments selected for use in the procedure, tasks to be accomplished during the procedure, particular roles assigned to the documented instruments or task, explanations for the selectedIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 instruments, tasks, roles, etc. and / or instructive examples (e.g., text, audio, video, etc.) for how to use the selected instruments to carry out the underlying task of the planned procedure.

[0029] In some embodiments, the pre-operative or intra-operative guidance may at least partially relate to aspects of a computer-assisted system such as a computer-assisted medical system to be used during the medical procedure. In these embodiments, the pre-operative guidance and intra-operative guidance can include material specific to the computer-assisted system such as instruments to be used with the computer-assisted system, tasks to be performed by the computer-assisted system, instructive examples for how to use the computer-assisted system, etc. Furthermore, the pre-operative or intra-operative guidance may also, in whole or in part, include material that does not directly involve the computer- assisted system such as standard operating room instruments, tasks to be performed using the standard instruments, and / or instructive examples for how to use the standard instruments.

[0030] The adaptive Al and ML techniques and models described herein relate to dynamically adapting the inputs to machine learning models as function of multi-modal data to be processed by the model and / or user inputs. In particular, the adapted inputs may include adaptive constitutions with modifiable instruction sets that direct how the machine learning model is to process the multi-modal data. As a result, the performance of the machine learning models can be adapted without the cost and time of tuning the underlying models.

[0031] To adapt a constitution, in some embodiments, the systems described herein may construct the instruction sets from different sets of predefined rules that are stored in a data store. For example, the sets of predefined rules may include rule sets relating to different: (1) instruments for use in various different procedures; (2) roles (e.g., surgeon, assistant, scrub tech, nurse etc.) for people carrying out the medical procedure; (3) procedure types, (4) levels of experience (expert, novice, intermediate, etc.), (5) patient conditions, and / or (6) tasks divisions for various different medical procedures. The system may also include in the constitution a fixed set of rules that control how the target machine learning model should generally operate to produce the expected output (e.g., the pre-operative guidance, intraoperative guidance, etc.). The fixed set of rules may also include instructions that control how the target machine learning model should consider the other predefined rules adaptively inserted into the constitution and define other overarching concerns to be considered by the model when generating the relevant outputs (e.g., optimization of patient outcomes or reduction of waste). Accordingly, the systems described herein may be configured to process multi-model input data and / or user inputs to identify an appropriate set of rules and update aIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 constitution in accordance therewith. In some embodiments, if the systems do not detect an appropriate set of rules, the systems may utilize a generative Al model to generate one or more rules intended to implement the desired function of the target ML model.

[0032] The user input may be provided in natural language (e.g., via a chatbot- type interface), as audio data, as non-linguistic user interactions with a GUI (e.g., tapping or gesturing on a touch- sensitive display screen), or in any other suitable form for assessment by the systems described herein. By modifying the constitutions based on input data and user inputs, the systems described herein are able to dynamically optimize the outputs of a given model for a specific task at hand, environment, and / or conditions without having to fine-tune the models and systems.

[0033] FIG. 1 is a simplified diagram of an example computer-assisted system 100, according to various embodiments. The computer-assisted system 100 may be a computer- assisted medical system for assisting with performing tasks for medical procedures. Further, the computer-assisted system 100 may utilize adaptable Al models for providing preoperative and / or intra-operative guidance in relation to the computer-assisted system 100 as described herein. In some examples, the computer-assisted system 100 is a teleoperated system. In medical examples, the computer-assisted system 100 can be a teleoperated medical system such as a surgical system. As shown, the computer-assisted system 100 includes a follower device 104 that can be teleoperated by being controlled by one or more leader devices (also called “leader input devices” when designed to accept external input), described in greater detail below. Systems that include a leader device and a follower device are referred to as leader-follower systems, and also sometimes referred to as master-slave systems. Also shown in FIG. 1 is an input system that includes a workstation 102 (e.g., a console), and in various embodiments the input system can be in any appropriate form and may or may not include the workstation 102.

[0034] In the example of FIG. 1, the workstation 102 includes one or more leader input devices 106 that are designed to be contacted and manipulated by an operator 108. For example, the workstation 102 may comprise one or more leader input devices 106 for use by the hands, the head, or some other body part(s) of operator 108. The leader input devices 106 in this example are supported by the workstation 102 and can be mechanically grounded. In some embodiments, an ergonomic support 110 (e.g., forearm rest) can be provided on which the operator 108 can rest his or her forearms. In some examples, the operator 108 can perform tasks at a worksite within a workspace near the follower device 104 during a procedure, byIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 commanding the follower device 104 using the leader input devices 106. In a medical example, the worksite may be a surgical worksite associated with a patient.

[0035] A display device 112 is also included in the workstation 102. The display device 112 may be configured to display images for viewing by the operator 108. The display device 112 can be moved in various DOFs to accommodate the viewing position of the operator 108 and / or to provide control functions. In embodiments where the display device 112 provides control functions, the leader input devices 106 may include the display device 112. In the example of the computer-assisted system 100, displayed images may depict a worksite at which the operator 108 is performing various tasks by manipulating the leader input devices 106 and / or the display device 112. In some examples, images displayed by display device 112 may be received by the workstation 102 from one or more imaging devices arranged at a worksite. In other examples, the images displayed by the display device 112 may be generated by the display device 112 (or by a different connected device or system), such as for virtual representations of tools, the worksite, or for user interface components. As will be explained below, in some embodiments the display device 112 may display pre-operative and / or intra-operative guidance such as instrument guidance in relation to the computer- assisted system 100 or the medical procedure being performed thereby.

[0036] In examples, the display device 112 may be a touch-screen device and may receive user-input via the touch screen. The touch screen may provide a user interface that a user may select from various options to provide one or more user preferences for performing a procedure or medical task. Additionally, the display device 112 may display one or more images such as an intraoperative or pre-operative image of a patient, organ, surgical site etc., and the user may provide user input via the touch screen to indicate one or more regions or elements displayed in the intraoperative or pre-operative images. The display device 112 may provide one or more images or user interfaces and a user may interact with and provide user indications and input via another device such as a keyboard, mouse, audio device, etc. For example, a user may provide text input such as natural language text via a keyboard to provide a user input to the system. A user may use a mouse or another similar device to click on or indicate selection of an option or user feedback based on one or more images presented by the display device. The workstation 102 may further include a microphone that may capture and record audio of a user providing indications of user preferences and user inputs to the system. One or more processors, or controllers as discussed further herein, may then derive natural language text from the recorded audio data to derive a user input to the systemIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519100. In examples, the user input may further be determined from the audio data as a user responding to one or more prompts to confirm, reject, or otherwise indicate a user preference or input. Additionally, user input may be provided via one or more sources such as video, haptic input, data banks, instruments (e.g., by a user changing a setting on an instrument or auxiliary device), sensors, etc. It should be appreciated that while the foregoing describes obtaining the user input from the display device 112 of the workstation 102, in other embodiments, the user input may be obtained by other display devices associated with the operating room that are interacted with by personnel other than the operator 108.

[0037] As illustrated, the computer-assisted system 100 also includes a follower device 104 that can be commanded by the workstation 102. In a medical example, the follower device 104 can be located near an operating table (e.g., a table, bed, or other support) on which a patient can be positioned. In some medical examples, the workspace is provided on an operating table, e.g., on or in a patient, simulated patient, or model, training dummy, etc. (not shown). As illustrated, the follower device 104 may include a plurality of repositionable structures 120 (sometimes referred to as “manipulator arms” in robotic embodiments). In some embodiments, the repositionable structures 120 may include a plurality of links that are rigid members and joints that can be individually actuated as part of a kinematic series. Additionally, each of the repositionable structures 120 is configured to be coupled to an instrument 122. While FIG. 1 illustrates a follower device 104 that has four repositionable structures 120a-120d, in other embodiments, the follower device 104 may include one, two, three, four, five, six, or additional or fewer repositionable structures 120a-120d.

[0038] The instrument 122 can include, for example, a working portion 126 and one or more structures for supporting and / or driving the working portion 126. Example working portions 126 include end effectors that physically contact or manipulate material, energy application elements that apply electrical, RF, ultrasonic, or other types of energy, sensors that detect characteristics of the workspace environment (such as temperature sensors, imaging devices, etc.), and the like. In various embodiments, examples of instruments 122 include, without limitation, a sealing instrument, a cutting instrument, a sealing-and-cutting instrument, an energy instrument for applying energy, a gripping instrument (e.g., clamps, jaws), a stapler, an imaging instrument such as one using optical, RF, or ultrasonic imaging modalities, a sensing instrument, an irrigation instrument, a suction instrument, and / or the like. In addition, the instrument 122 may include a transmission mechanism 128 that can be coupled to a drive assembly 130 of the respective repositionable structure 120a- 120d. TheIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 drive assembly 130 may include a drive and / or other mechanisms controllable from workstation 102 that transmit forces to the transmission mechanism 128 to articulate or otherwise actuate the instrument 122.

[0039] As illustrated, each instrument 122 may be mounted to a portion of a respective repositionable structure 120a-120d. In FIG. 1, this is shown with the drive assembly 130 physically coupled to the transmission mechanism 128. The distal portion of each repositionable structure 120a-120d further includes a cannula mount 124 to which a cannula (not shown) is mounted. When a cannula is mounted to the cannula mount 124, a shaft of the instrument 122 passes through the cannula and into a workspace.

[0040] In various embodiments, one or more of the working portions 126 of the instruments 122 may include an imaging device for capturing images. The imaging device may include any sensing technology capable of acquiring an image. Example imaging instruments include an optical endoscope, a hyperspectral camera, an ultrasonic sensor, etc. Imaging instruments may comprise monoscopic imagers, stereoscopic imagers, and / or the like. Imaging devices based on radiofrequency domains may capture images in any frequency spectrum, including visible light, infrared light, ultraviolet light, and / or the like. The imaging device may include an illumination source to light the region being imaged. In embodiments where the working portions 126 of one or more of the instruments 122 include an imaging device, the instrument 122 may be configured to capture images of a portion of the workspace for display via the display device 112.

[0041] In some embodiments, the repositionable structures 120a-120d and / or instruments 122 can be controlled to move the working portion 126 in response to manipulation of the leader input devices 106 by the operator 108. Accordingly, the repositionable structures 120a- 120d and / or instruments 122 may be said to “follow” the leader input devices 106 through teleoperation. This enables the operator 108 to perform tasks at the worksite using the repositionable structures 120a- 120d and / or instruments 122. For a surgical example, the operator 108 can direct the repositionable structures 120a-120d of the follower device 104 to move the working portions 126 as part of a surgical procedure performed at an internal surgical site that is entered via one or more minimally invasive apertures or natural orifices. It should be appreciated that, in some embodiments, the follower device 104 may include nonteleoperated components that the operator 108 or other medical professional must manually manipulate to a desired pose.Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519

[0042] In some embodiments, a repositionable structure 120a of the computer-assisted system 100 may be configured to support a working portion 126a that includes an imaging device (also referred to herein as an “imaging device 126a”). For convenience, an instrument 122 that includes an imaging device is also referred to as an “imaging instrument” herein. The control system 140 may be configured to command the repositionable structure 120a and / or the imaging instrument 122 comprising the imaging device 126a to automatically position and / or orient (“pose”) the field of view (FOV) of the imaging device 126a to provide images of the workspace and / or other instruments 122.

[0043] In the illustrated embodiment, a control system 140 is communicatively coupled to the workstation 102. In other embodiments, the control system 140 may be provided as a component of the workstation 102 and / or the follower device 104. During teleoperation, as the operator 108 moves the leader input device(s) 106, one or more sensors configured to detect the leader input device(s) 106 generate spatial and / or orientation movement data that is provided to control system 140. The control system 140 may interpret the spatial and / or orientation information to determine and / or provide control signals to the follower device 104 to control the movement of repositionable structures 120a-120d, instruments 122, and / or working portions 126. In addition to the components of the follower device 104, in some embodiments, the control system 140 is configured to interpret inputs received from the workstation 102 to control operation of one or more auxiliary devices (not depicted) utilized in a procedure. For example, the workstation 102 may be used to control a pose of a surgical bed or operation of an insufflator.

[0044] In one embodiment, the control system 140 supports one or more wired communication protocols, (e.g., Ethernet, USB, and / or the like) and / or one or more wireless communication protocols (e.g., Bluetooth, IrDA, HomeRF, IEEE 1102.11, DECT, Wireless Telemetry, and / or the like) for communications between the control system 140 and the workstation 102 and / or the follower device 104.

[0045] In some embodiments, the control system 140 may be implemented at one or more computing systems. For example, one or more computing systems may be used to control the follower device 104. As another example, one or more computing systems may be used to control components of the workstation 102, such as movement of a display device 112.

[0046] As illustrated, the control system 140 includes a processor system 150, a memory 160, and an artificial intelligent (Al) assist module 180. The memory 160 may store a control module 170. The processor system 150 may include one or more processors having differentIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 processing architectures for processing instructions. For example, the one or more processors may be one or more cores or micro-cores of a multi-core processor, a central processing unit (CPU), a microprocessor, a field-programmable gate array (FPGA), an application- specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), a tensor processing unit (TPU), and / or the like.

[0047] In some embodiments, the processor system 150 includes circuity to support one or more communication interfaces (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.). Additionally, a communication interface of control system 140 may include an integrated circuit for connecting the control system 140 to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) and / or to another device, such as the workstation 102 and / or the follower device 104.

[0048] Additionally, the memory 160 may include non-persistent storage (e.g., volatile memory, such as random access memory (RAM), cache memory), persistent storage (e.g., a hard disk, an optical drive such as a compact disk (CD) drive or digital versatile disk (DVD) drive, a flash memory, a floppy disk, a flexible disk, a magnetic tape, any other magnetic medium, any other optical medium, programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a FLASH-EPROM, and / or any other memory chip or cartridge. The non-persistent storage and persistent storage are examples of non- transitory, tangible machine-readable media that can store executable code that, when run by one or more processors (e.g., processor system 150), can cause the one or more processors to perform one or more of the techniques and / or methods disclosed herein.

[0049] The Al assist module 180 may implement one or more machine learning models and / or training protocols therefor. For example, the Al assist module 180 may implement one or more neural networks, deep learning models, decision trees, support vector machines, linear regression, generative Al models, reinforced learning models, random forests, Naive Bayes models, large language models (LLMs), generative adversarial networks, foundation models, image recognition models, linear discriminant analysis models, creative applications, autoregressive models, supervised or unsupervised learning models, multimodal models, vision language models (VLMs), vision foundation models (VFMs), large multi-modal models (LMMs), Transformer models (including Robotic Transformer models), or another machine learning or Al model for performing the methods described herein. The structure of the one or more machine learning is described in more detail with respect to FIGs. 2A-2C,Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 705194A, and 4B. The Al assist module 180 may include dedicated processors and memory for storing and performing Al processes, or the Al assist module 180 may utilize resources of the processor system 150 and the memory 160 to store and / or perform any processing or tasks required to perform the methods described herein.

[0050] Additionally, the control system 140 may also include one or more input devices (such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device) and / or output devices (such as a display device, a speaker, external storage, a printer, or any other output device). In some embodiments, the control system 140 may be implemented on a particular node of a distributed computing system (e.g., a cloud computing system). As another example, different functionalities associated with the control system 140 may be implemented on different nodes of the distributed computing system. Further, one or more elements of the aforementioned control system 140 may be located at a remote location and connected to the other elements over a network.

[0051] In an endoscopic surgery example, the imaging instrument comprising the imaging device 126a may be inserted into the patient prior to the other instruments 122, including a second instrument 122b comprising a second working portion 126b. The second instrument 122b can include any appropriate working portion 126b, and can even include a second imaging device. Accordingly, the imaging device 126a may be maneuvered to be positioned to identify a target to which other instruments may interact with as part of another task. The control system 140 may, for example, automatically command the corresponding repositionable structures 120a and 120b to position respective instruments 122a and 122b to perform one or more tasks in tandem, or sequentially based on the specific task, instruments, and positions of the repositionable structures 120a and 120b. In examples, the control system 140 may perform Al processes and algorithms via the Al assist module 180 to provide the preoperative or intra-operative guidance as described herein. The control system 140, via the Al assist module 180, may also perform various task selection and identification processes such as those disclosed in U.S. Provisional application 63 / 667,234 titled “Multi-Task Al System For Dynamic and Intelligent Robotic Task Planning Based On User Input”, which is incorporated by reference herein in its entirety. Accordingly, the control system 140 is able to identify and control repositionable structures to perform tasks for a medical procedure in a variety of scenarios using adaptable Al.

[0052] It should be appreciated that while FIG. 1 depicts a system for performing a computer-assisted procedure, the disclosed techniques may be implemented for conventionalIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 procedures that do rely on computer-assisted systems. In these embodiments, the Al assist module 180 may be implemented at an alternate computing system associated with the operating room for the procedure.

[0053] FIG. 2A is a schematic diagram of a system 200 for using adaptive Al for generating pre-operative guidance in relation to a planned medical procedure. The system 200 includes a plurality of software or hardware modules that may be executed by a processing unit 201 to generate the pre-operative guidance. The processing unit 201 may include the control system 140 and the Al assist module 180 of FIG. 1 or other processing components known in the art. For example, the processing unit 201 may include one or more processors, each of which may be a programmable microprocessor or the like that executes software instructions stored in a memory unit 203 (e.g., memory 160 of FIG. 1 or another suitable memory unit not directly associated with the control system 140 of FIG. 1) to execute some or all of the functions of the system 200 as described herein. The processing unit 201 may include one or more graphics processing units (GPUs) and / or one or more central processing units (CPUs), for example. Alternatively, or in addition, one or more processors in the processing unit 201 may be other types of processors (e.g., applicationspecific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), language processing units (LPUs), etc.), and some of the functionality of the system 200 as described herein may instead be implemented in hardware.

[0054] As shown in FIG. 2A, the system 200 includes an instrument guidance machine learning model 202 which is executable by the processing unit 201. Generally, the instrument guidance machine learning model 202 is configured to generate an instrument guidance 208 based on pre-operative multi-modal data 204 input thereto as directed by an instrument guidance constitution 206.

[0055] The instrument guidance machine learning model 202 may comprise a set of interconnected nodes, layers, trained parameter values (e.g., multiplicative weights, additive bias, etc.), etc. The trained parameters may be set via backpropagation techniques in a training process that uses historical data inputs. In some embodiments, the instrument guidance machine learning model 202 is configured using a multi-modal transformer type architecture that can receive inputs of different modalities relating to the planned medical procedure. However, it should be appreciated that this architecture may be substituted or supplemented with other Al architectures including, but not limited to, convolutional neuralIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 network (CNN) architectures, recurrent / recursive neural network (RNN) architectures, sorting / clustering architectures, etc.

[0056] The pre-operative multi-modal data 204 input into the instrument guidance machine learning model 202 may generally include image or video data of the location associated with the planned medical procedure (see e.g., external video data 204D shown in FIG. 2B); patient information documenting details specific to the patient undergoing the planned medical procedure (e.g., details from patient medical records such as electronic medical records (EMRs) 204A shown in FIG. 2B); procedure data or information documenting information related to the planned medical procedure (e.g., a name of the procedure, facility details such an indication of whether the facility is a training or academic hospital / institution, a procedure configuration (e.g., a number of workstations 102, an instrument requirement, etc.) a degree of computer-assisted surgical involvement (e.g., fully autonomous, semi-autonomous, and / or details describing which autonomous features are enabled), a planned entry patient entry method, characteristics of the target anatomy, patient characteristics associated with known risks of particular adverse events occurring, etc. (see e.g., user input data 204F shown in FIG. 2B); personnel profiles or information documenting features of the personnel assigned to the planned medical procedure (e.g., roles of the personnel, experience levels, names of the personnel, etc.); preference information for the personnel assigned to the planned medical procedure (e.g., preferred methods for carrying out the planned medical procedure, preferred instruments to use for the planned medical procedure, etc.); equipment information indicating equipment assigned for use with the planned medical procedure (e.g., a make and model of the workstation 102 and follower device 104, imaging equipment, anesthesia equipment, etc.); and / or inventory data from an instrument inventory system (e.g., data store 212) indicating availability of different instruments.

[0057] The pre-operative multi-modal data 204 may also include historical procedure information on past medical procedures, such as instruments used in past procedures, tasks performed in past procedures, personnel that performed past procedures and their associated roles and experience levels, patient outcomes of past procedures, historical log data from the computer-assisted system 100 (see e.g., historical event data 204B and historical kinematic data 204C shown in FIG. 2B), historical video data from past procedures (see e.g., external video data 204D and historical procedure video data 204E shown in FIG. 2B), etc. The preoperative multi-modal data 204 may be obtained from one or more data sources, such as user input to the system 200 (e.g., audio input, text inputs, touch interaction with the user interfaceIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519216, etc.), the data store 212, imaging devices at the location associated with the planned medical procedure, and / or the computer-assisted system 100.

[0058] It should be appreciated that while the techniques described herein may process any number of the aforementioned types of pre-operative data, in some embodiments, the instrument guidance machine learning model 202 may additionally accept a care team provided instrument and / or procedure plan (e.g., an instrument “pick list”) as an input. In these embodiments, the care team may interact with a user interface 216 to input a current instrument and / or procedure plan and receive any recommended adjustments to the plan. In these embodiments, the system 200 may automatically supplement the care team-provided instrument and / or procedure plan with the pre-operative multi-modal 204 when generating the inputs to the instrument guidance machine learning model 202. In other embodiments where no instrument and / or procedure plan is provided by the care team, the instrument guidance machine learning model 202 may instead be configured to generate a new instrument and / or procedure plan based on the pre-operative multi-modal data 204.

[0059] The instrument guidance constitution 206 directs the instrument guidance machine learning model 202 on how to analyze pre-operative multi-modal data 204 to generate the instrument guidance 208. In particular, the instrument guidance constitution 206 includes a plurality of rules that control or otherwise instruct the instrument guidance machine learning model 202 on how to generate the instrument guidance 208. The plurality of rules may include different rules that relate to different portions of the pre-operative multi-modal data 204. For example, the plurality of rules may include instrument selection rules that direct how the instrument guidance machine learning model 202 selects a set of instruments to use when performing the planned medical procedure. The instrument selection rules may include definitions and instructions relating to different aspects of the pre-operative multi-modal data 204 such as rules based on personnel roles, personnel experience levels, the type of the planned medical procedure, patient conditions, personnel preferences, desired patient outcomes, etc.

[0060] The plurality of rules may also include a set of fixed rules. The fixed rules may be present in every variation of the instrument guidance constitution 206. For example, the fixed rules may include one or more rules associated with persistent analysis criteria the instrument guidance machine learning model 202 considers when generating any portion of the instrument guidance 208 (e.g., the selected instruments, task divisions for the planned medical procedure related to the selected instruments, assignments of the selected instrumentsIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 to different roles, explanations for the selected instruments, tasks, roles, etc. and / or the text, audio, or video, instructive examples as described in more detail herein in connection with FIGs. 3A-3E). The persistent analysis criteria may include, but are not limited to, directives to optimize patient outcomes or reduce waste.

[0061] As shown in FIG. 2A, the instrument guidance machine learning model 202 may include embedding or projection layers 218, additional layers 220, and an output layer 222.

[0062] The embedding or projection layers 218 may be configured to receive and process different portions of the pre-operative multi-modal data 204 for further processing by the additional layers 220 and the output layer 222. In some embodiments, the embedding or projection layers 218 may be configured to process different modalities of data (text, audio, video, system event data, sensor data, etc.) into a single input format (e.g., natural language text) for the additional layers 220. For example, image or video data (e.g., external video data 204D and historical procedure video data 204E shown in FIG. 2B) in the pre-operative multimodal data 204 may be converted to natural language descriptions of what is depicted in the image or video data and historical procedure log data from the computer-assisted system 100 may be converted into natural language descriptions of the operations performed and instruments used.

[0063] The image or video data may be input into a VLM potion (not depicted) of the embedding or projection layers 218 to convert the image or video data into the natural language descriptions understood by the additional layers 220 and the output layer 222. The VLM may be configured to generate text descriptions of features within the image or video data and a location thereof in the image data. These text descriptions may include natural language descriptions of a scene depicted by the input image data, such as image data output by an endoscopic instrument. Furthermore, the VLM may be configured to identify objects, such as surgical instruments and devices (e.g., surgical beds or tables, medical devices, display devices, etc.), individuals and personnel (e.g., clinicians, doctors, medical technicians, etc.) depicted by image data generated by an image sensor having a field of view of an operating room. The parameters of the VLM may be independently trained or tuned from the other parameters of instrument guidance machine learning model 202.

[0064] The historical procedure log data may be maintained in the data store 212 as part of the pre-operative multi-modal data 204. The historical procedure log data may include system event data (e.g., historical event data 204B shown in FIG. 2B), kinematic data (e.g., historical kinematic data 204C shown in FIG. 2B), or force-sensing data generated by the computer-Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 assisted system 100 of FIG. 1 (or other computer-assisted system) when carrying out the historical medical procedure. In particular, the event data may include data indicating events such as instrument installs, insertions, removals, etc. for the computer-assisted system 100 of FIG. 1 during the historical procedures. The kinematic data may include data indicating movements of the instruments 122 and / or the repositionable structures 120 of the computer- assisted system 100 during the historical procedures. The force-sensing data may include data indicating forces exerted on the instruments 122 and / or the repositionable structures 120, etc. during the historical procedures.

[0065] The historical procedure log data may be input into a transformer model, LMM, or similar portion of the embedding or projection layers 218 to convert the past procedure log data into the natural language descriptions understood by the additional layers 220 and the output layer 222. In particular, the transformer model, LMM, etc. may be configured to convert the event data, kinematic data, force- sensing data or other aspect of the historical procedure log data into natural language descriptions of the operations performed, instruments used, complications encountered, etc. during performance of the past medical procedure. In some embodiments, parameter values of the transformer model, LMM, etc. may be independently trained or tuned from the other parameters of instrument guidance machine learning model 202.

[0066] In some embodiments, portions of the pre-operative multi-modal data 204 that are already in a natural language text form may be passed without modification to the additional layers 220 and the output layer 222. However, in some embodiments, the embedding or projection layers 218 may be configured to summarize or extract key details of these portions of the pre-operative multi-modal data 204. Summarizing and / or extracting key details before further processing by the additional layers 220 and the output layer 222 may help to confine the set of data received by the additional layers 220 to a predefined context window for the additional layers 220.

[0067] The additional layers 220 may comprise various Al or ML type layers known in the art. Such layers include multiheaded self- attention layers, cross-attention layers, multi-layer perceptrons, feed forward layers, softmax layers, etc. In some embodiments, the additional layers 220 and may comprise layers of a pretrained Al model such as an LLM or similar. This pretrained Al model can include third party provided models that are either fine-tuned to process the pre-operative multi-modal data 204 or that are used as is without any additional tunning or training (e.g., via a call to a third party provided application programmingIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 interface or via a call to a private self-hosted instance of the model). In particular, the pretrained Al model may be configured to receive inputs in the natural language text format output from the embedding or projection layers 218. However, it should be appreciated that other variations of the additional layers 220 with different input formats are also possible.

[0068] As shown in FIG. 2A, the instrument guidance constitution 206 may be input into the additional layers 220 to direct how the additional layers 220 process the natural language text inputs generated by the embedding or projection layers 218. However, it should be appreciated that in other cases, the instrument guidance constitution 206 may be input to an associated one of the embedding or projection layers 218 before proceeding to the additional layers 220.

[0069] The output layer 222 aggregates outputs from each preceding layer of the instrument guidance machine learning model 202 (e.g., the embedding or projection layers 218 and the additional layers 220) to generate the output instrument guidance 208.

[0070] The instrument guidance 208 output from the instrument guidance machine learning model 202 includes an indication of the set of instruments to use when performing the planned medical procedure. The set of instruments are selected by the instrument guidance machine learning model 202 from analysis of the pre-operative multi-modal data 204 using the rules included in the instrument guidance constitution 206 (e.g., the instrument selection rules, the set of fixed rules, and / or other sets of rules as described herein). In embodiments where the care team provided an instrument and / or procedure plan, the instrument guidance 208 may format the guidance as a set of recommended adjustments to the provided plans. For example, the guidance may recommend utilizing additional or alternative instruments to those included in the provided plans.

[0071] In some embodiments, the instrument guidance 208 may also include portions that describe a set of roles that are to perform the planned medical procedures and assignments of subsets of the set of instruments to each role of the set of roles. The roles may include one or more of a surgeon, a first assistant, a scrub tech, and a nurse. Furthermore, in these embodiments, the instrument guidance constitution 206 includes role rules that direct the instrument guidance machine learning model 202 to adapt a presentation of the instrument guidance 208 based on the set of roles and the corresponding instrument assignments.

[0072] As discussed above, in some embodiments the instrument guidance constitution 206 may be adapted based on the planned procedure or other user inputs. Accordingly, the system 200 includes a constitution generator 210 to generate or instantiate the instrumentIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 guidance constitution 206 by dynamically selecting the plurality of rules to include in the instrument guidance constitution 206 based on user input received by the user interface 216 (e.g., user input data 204F shown in FIG. 2B) and / or from data present in the pre-operative multi-modal data 204. In particular, the constitution generator 210 may be configured to identify and select the plurality of rules to include in the instrument guidance 208 from the data store 212 or other similar data storage devices or systems. The constitution generator 210 may include an algorithmic software or hardware module executable by the processing unit 201 according to instructions stored in the memory unit 203. However, in some embodiments the constitution generator 210 may include a machine learning model with trained or tuned parameter values for selecting the rules for the instrument guidance constitution 206 and other constitutions as described herein.

[0073] For example, the data store 212 may store a personnel profile that indicates instrument preferences for personnel performing the medical procedure. Additionally or alternatively, the constitution generator 210 may analyze user input received via the user interface 216 to select rules to include in the instrument guidance constitution 206. For example, a user may interact with a chatbot interface, an intraoperative imaging interface, etc. to provide the user inputs and / or instructions to the constitution generator 210 indicative of, for example, user instrument preferences.

[0074] After the instrument guidance machine learning model 202 generates the instrument guidance 208, the processing unit 201 provides the instrument guidance 208 to the display unit 214 and / or the user interface 216 to direct pre-operative preparation for performing the planned medical procedure. The display unit 214 may include the display device 112 included in the workstation 102 of the computer-assisted system 100 of FIG. 1. Additionally, or alternatively, in some embodiments the display unit 214 may include one or more display devices (computer screen, tv screen, phone screen, tablet screen, etc.) either located at the location where the planned medical procedure is set to occur or associated with the personnel assigned to perform the planned medical procedure. For example, in some embodiments the processing unit 201 may transmit (e.g., via email, text, push notification, on device software application, web application, etc.) the instrument guidance 208 to a computer, phone, tablet, etc. of the personnel assigned to perform the planned medical procedure.

[0075] The user interface 216 may be configured to detect user input data from a user indicative of user feedback on the instrument guidance 208. The user interface 216 may include a touch screen, keyboard, mouse, microphone, or other suitable device for receivingIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 the feedback. In some embodiments, the user interface 216 may be a part of the workstation 102 of the computer-assisted system 100 of FIG. 1. Additionally or alternatively, in some embodiments, the user interface 216 is a part of the device that includes the display unit 214 on which the instrument guidance 208 is displayed. In some embodiments, the user interface 216 may be one of the data sources that supply the pre-operative multi-modal data 204.

[0076] In some embodiments, the user feedback received by the user interface 216 is an acceptance of the instrument guidance 208. In response to receiving the acceptance, the processing unit 201 may provide a fulfillment request for the personnel assigned to the planned procedure to prepare the set of instruments indicated in the instrument guidance 208. In some embodiments, the indication of the set of instruments included in the instrument guidance 208 includes an indication of a first set of instruments that are required to perform the planned medical procedure and a second set of instruments that are to be on-hand should a need arise. In these embodiments, the fulfillment request provided to the personnel assigned to perform the planned procedure may indicate that (i) instruments included in the first set of instruments are to be opened or otherwise prepared for usage, and (ii) instruments included in the second set of instruments are to be placed in an operating room in an unopened state. In this way the system 200 may avoid spoiling or wasting the second set of instruments in the event they are not ultimately needed during the planned medical procedure.

[0077] In some embodiments, the user feedback received by the user interface 216 is an indication of a modification 223 of the instrument guidance 208. In particular, the modification 223 may indicate a new or revised user preference as to types of instruments or methods of performing the planned medical procedure, a requested substitution of one of the set of instruments in the instrument guidance 208, a request for inclusion of an additional instrument or instrument type in the set of instruments in the instrument guidance 208, a request to remove an instrument form the set of instruments in the instrument guidance 208, etc.

[0078] Additionally, the processing unit 201 may input user input data indicative of the modification 223 into the instrument guidance machine learning model 202 to generate a modified pre-operative instrument guidance in accordance with the modification 223. After the instrument guidance machine learning model 202 generates the modified pre-operative instrument guidance, the processing unit 201 may provide it to the display unit 214 for presentation to the user. As shown in FIG. 2A, in some embodiments, the modification 223 and the originally output instrument guidance 208 may be input into the embedding orIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 projection layers 218 of the instrument guidance machine learning model 202. After the user accepts the modified instrument guidance, the processing unit 201 may process the user feedback to associate any indicated preferences with a personnel profile maintained in the data store 212.

[0079] Furthermore, the user input data indicative of the modification 223 may be input into the constitution generator 210 to modify the set of rules included in the instrument guidance constitution 206 based on the modification 223. For example, the constitution generator 210 may select different rules based on the new or revised user preference as to types of instruments or methods of performing the planned medical procedure, the requested substitution of one of the set of instruments in the instrument guidance 208, the request for inclusion of an additional instrument or instrument type in the set of instruments in the instrument guidance 208, the request to remove an instrument form the set of instruments in the instrument guidance 208, etc. Furthermore, the constitution generator 210 may select a set of rules that instruct the instrument guidance machine learning model 202 on how to analyze the instrument guidance 208 and the user input data indicative of the modification 223 that may be input into the embedding or projection layers 218.

[0080] It should be appreciated that in some embodiments, the instrument guidance machine learning model 202 may generate the modified pre-operative instrument guidance without directly inputting the instrument guidance 208 and the user input data indicative of the modification 223 into the instrument guidance machine learning model 202. For example, the instrument guidance machine learning model 202 may generate the modified preoperative instrument guidance based on the revised instrument guidance constitution 206 without needing to directly consider the instrument guidance 208 and the user input data indicative of the modification 223 as inputs to the instrument guidance machine learning model 202.

[0081] In some embodiments, in response to receiving the modification 223, the processing unit 201 may first validate that the modification 223 was received from a user with a role that has a permission level that permits the modification 223 to be performed. When the user does have a permission level that permits the modification 223 to be performed, the processing unit 201 may proceed to process the modification 223 in the various ways described herein. However, when the user does not have a permission level that permits the modification 223 to be performed, the processing unit 201 may ignore the modification 223.Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519

[0082] In some embodiments, in response to the modification 223, the processing unit 201 may obtain the inventory data from the data store 212 or another data store associated with an instrument inventory system that is operatively coupled to the processing unit 201. The inventory data may indicate the availability of substitute or additional instruments indicated in the modification 223.

[0083] In some embodiments, the processing unit 201 may include the inventory data in the pre-operative multi-modal data 204 or otherwise input the inventory data into the instrument guidance machine learning model 202 when instructing the instrument guidance machine learning model 202 to generate the modified pre-operative instrument guidance. In these embodiments, the constitution generator 210 may include, in the revised instrument guidance constitution 206, instructions that direct the instrument guidance machine learning model 202 to determine whether the inventory data indicates that proposed changes to the set of instruments (e.g., instruments additions or substitutions) in the modification 223 can be fulfilled. The revised rules in the instrument guidance constitution 206 may further direct the instrument guidance machine learning model 202 to include a modification response in the modified pre-operative instrument guidance when the instrument guidance machine learning model 202 determines that the modification 223 cannot be fulfilled. In particular, the modification response may include one or more of: (i) alternative instruments to the instruments substations or additions in the modification 223 that are selected by the instrument guidance machine learning model 202 and are different from the set of instruments included in the instrument guidance 208; (ii) a rationale generated by the instrument guidance machine learning model 202 describing why the modification 223 cannot be fulfilled, and (iii) a solicitation of further input from the user as to alternatives to the modification 223.

[0084] It should also be appreciated that in some embodiments, the processing unit 201 may use the instrument guidance machine learning model 202 to determine if the modification 223 can be fulfilled separately from generating the modified pre-operative instrument guidance. For example, the processing unit 201 may first determine the ability of the modification 223 to be fulfilled in embodiments where other sections of the modified preoperative instrument guidance would be dependent on the particular instruments ultimately selected as part of the modified pre-operative instrument guidance. These dependent sections may include sections as described herein related to tasks, roles, explanations, instructive examples, etc.Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519

[0085] Additionally or alternatively, the user feedback received from the user interface 216 may include a request to provide a rationale for the instrument guidance 208. In response to receiving this request, the processing unit 201 may input user input data indicative of the request to provide the rationale into the instrument guidance machine learning model 202 to obtain the requested rationale. In these embodiments, the instrument guidance 208 may also be input to the instrument guidance machine learning model 202. It should be appreciated that in some of these embodiments the request to provide the rationale for the instrument guidance 208 directly provides the instructions to the instrument guidance machine learning model 202 such that the constitution generator 210 does not need to generate an updated instrument guidance constitution 206. However, in some embodiments, the constitution generator 210 may generate an updated instrument guidance constitution 206 to provide further details or instructions to the instrument guidance machine learning model 202 in relation to the request to provide the rationale for the instrument guidance 208. After the rational is output form the instrument guidance machine learning model 202, the processing unit 201 may update the instrument guidance 208 to include the obtained rationale and provide the updated instrument guidance 208 to the display unit 214.

[0086] With particular reference now to FIG. 2B, operation of the system 200 to segment the planned medical procedure into one or more tasks, assign those tasks to different roles, and incorporating the segmented tasks and assigned roles with instrument selections will be described in more detail.

[0087] As shown in FIG. 2B, the preoperative multimodal data 204 may in include, but is not limited to, the EMRs 204A for the patient undergoing the planned procedure and / or previous patient having undergone historical procedures, the historical event data 204B from the computer-assisted system 100 of FIG. 1, the historical kinematic data 204C from the computer-assisted system 100 of FIG. 1, the external video data 204D of operating areas associated with the planned or historical procedures, the historical procedure video data 204E, and user input data 204F. It should be appreciated that the pre-operative multi-modal data 204 used the embodiment of the system 200 shown in FIG. 2B may also include other multi-modal data as described herein, including the data described above in connection with the embodiment of the system 200 shown in FIG. 2A.

[0088] As shown in FIG. 2B, the system 200 may include a task segmentation machine learning module 224 and a task assignment machine learning model 226. The task segmentation machine learning module 224 is configured to segment the planned medicalIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 procedure into discrete tasks 228 based on rules in a task segmentation constitution 230 generated by the constitution generator 210. In particular, the task segmentation machine learning module 224 may include a scene recognition LMM 232 that receives the preoperative multi-modal data 204 and outputs natural language descriptions of personnel and objects present at the location associated with the planned medical procedure. These natural language descriptions (along with the task segmentation constitution 230) are then input into a task segmentation LLM 227 to output discrete tasks 228 based on the rules included in the task segmentation constitution 230 by the constitution generator 210.

[0089] In some embodiments, the rules in the task segmentation constitution 230 may include a set of procedure rules associated with a procedure type of the planned medical procedure indicated by the pre-operative multi-modal data 204. The set of procedure rules may control how the task segmentation LLM 227 determines particular tasks that need to be performed for a particular procedure type. The rules in the task segmentation constitution 230 may also include a set of patient rules associated with the patient conditions. The set of patient rules may instruct the task segmentation LLM 227 to determine which tasks need to be performed as part of the planned medical procedure based on the one or more patient conditions indicated by the pre-operative multi-modal data 204 (e.g., in the EMRs 204A).

[0090] The rules in the task segmentation constitution 230 may also include a set of personnel rules associated with different possible levels of experience for personnel assigned to the planned medical procedure. The personnel rules may a control how the task segmentation LLM 227 segments the planned procedure into the discrete tasks 228 and determines a granularity, duration, or sequence of the discrete tasks 228 based on the levels of experience of the personnel assigned to the planned medical procedure. The rules in the task segmentation constitution 230 may also include rules that direct the task segmentation LLM 227 to generate the discrete tasks 228 based on historical tasks segmentations for historical medical procedures that are contained in the pre-operative multi-modal data 204.

[0091] As shown in FIG. 2B, the task assignment machine learning model 226 may include an LLM and may be configured receive the discrete tasks 228 from the task segmentation machine learning module 224, generate task assignments 233 between roles associated with the planned medical procedure indicated in the pre-operative multi-modal data 204 and the discrete tasks 228, and outputs a combination of the discrete tasks 228 and the task assignments 233 as valid tasks 234 for input into the instrument guidance machine learning model 202. These operations of the task assignment machine learning model 226Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 may be controlled by rules included in a task assignment constitution 236 generated by the constitution generator 210.

[0092] The rules in the task assignment constitution 236 may include a set of personnel rules associated with different possible levels of experience for personnel assigned to the planned medical procedure. The personnel rules may direct the task assignment machine learning model 226 to assign the discrete tasks 228 to different roles based on the level of experience of the personnel assigned to the planned medical procedure. In some embodiments, the rules in the task assignment constitution 236 may also include rules that direct the task assignment machine learning model 226 to generate the task assignments 233 based on historical task assignments included in the pre-operative multi-modal data 204. The set of procedure rules may also control how the instrument guidance machine learning model 202 analyzes historical procedure data in the pre-operative multi-modal data 204 to determine the set of instruments based on instruments that have historically been used to perform the discrete tasks 228 of the planned medical procedure.

[0093] As shown in FIG. 2B, the instrument guidance machine learning model 202 receives the valid tasks 234 from the task assignment machine learning model 226. The instrument guidance machine learning model 202 may then process the valid tasks 234 along with the pre-operative multi-modal data 204 to generate the instrument guidance 208. In these embodiments, the instrument guidance 208 may include the selected set of instruments and task data. The task data may relate to the valid tasks 234 and may indicate one or more of (i) instruments in the set of instruments to be used to perform specific ones of the discrete tasks 228, (ii) task assignments 233 for a task along with instruments in the set of instruments to perform the task, and (iii) a rationale for usage of the instruments in the set of instruments in relation to each of the discrete tasks 228.

[0094] In some embodiments, the instrument guidance constitution 206 may include additional rules that control how the instrument guidance machine learning model 202 generates the task data portions of the instrument guidance 208. In particular, the instrument guidance constitution 206 may include the set of personnel rules, the set of procedure rules, and / or the set of patient rules. The set of personnel rules may control how the instrument guidance machine learning model 202 determines one or more task presentation characteristics for the task data included in the instrument guidance 208 based on the levels of experience for personnel assigned to the planned medical procedure. The task presentation characteristics may determine how the discrete tasks 228 and the task assignments 233 areIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 displayed (e.g., the order of the tasks, level of granularity, etc.) in the instrument guidance 208. The set of patient rules may also control how the instrument guidance machine learning model 202 determines which instruments in the set of instruments are to be used to perform different ones of the discrete tasks 228 based on one or more patient conditions indicated by the pre-operative multi-modal data 204.

[0095] In some embodiments, the task segmentation techniques, the task assignment techniques, and the techniques that generate the instrument guidance 208 described above are separately performed by the task segmentation machine learning model 224, the task assignment machine learning model 226, and the instrument guidance machine learning model 202. In other embodiments, these techniques may instead be implemented by different combinations of machine learning models. For example, in some embodiments, the instrument guidance machine learning model 202 may also perform the task segmentation and task assignment technique described herein. In these embodiments, the instrument guidance constitution 206 may include all of the iterations of the personnel rules, procedure rules, and patient rules described herein. Furthermore, in some embodiments, the task segmentation machine learning model 224 and the task assignment machine learning model 226 may be combined into a single model that both generates the discrete tasks 228 and the task assignments 233 for output to the instrument guidance machine learning model 202.

[0096] With reference now to FIG. 2C, in some embodiments, the instrument guidance 208 may include a video snippet 237 of an example performance of one of the discrete tasks 228. In some embodiments, the video snippet 237 may include a synthetic video generated by a generative machine learning model 238. As shown in FIG. 2C, the generative machine learning model 238 may generate the video snippet 237 based on an experience level 240 for a person assigned to an associated task and the instruments and task data 242 output from the instrument guidance machine learning model 202. Once generated, the processing unit 201 may combine the video snippet 237 with the task data 242 into the instrument guidance 208 and present the instrument guidance 208 with the video snippet 237 for display on the display unit 214.

[0097] With reference now to FIGS. 3A-3E, embodiments of the instrument guidance 208 that include different possible timeline plots 300A, 300B, 300C, 300D, and 300E will be discussed in more detail. In general, the timeline plots 300A, 300B, 300C, 300D, and 300E are provided for display on the display unit 214 and may indicate timing for when one or more tasks 302A, 302B, 302C, 302D, and 302E are to be performed. In particular, each ofIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 one or more tasks 302A, 302B, 302C, 302D, and 302E may be presented in respective sequential timelines 304A, 304B, 304C, 304D, and 304E. It should be appreciated that the tasks 302A, 302B, 302C, 302D, and 302E on the timeline plots 300A, 300B, 300C, 300D, and 300E indicate the task data generated by the instrument guidance machine learning model 202 and / or the discrete tasks 228 and task assignments 233 generated by the task segmentation machine learning model 224 and task assignment machine learning model 226, respectively. In some embodiments, the detailed portions of the instrument guidance 208 described in more detail below and relating to each of the one or more tasks 302A, 302B, 302C, 302D, and 302E may be displayed based on user input selecting a corresponding portion of the sequential timelines 304A, 304B, 304C, 304D, and 304E.

[0098] As shown in FIG. 3A, the timeline plot 300A is configured for an inexperienced person assigned to perform the planned medical procedure and may include tasks 302A ordered in the sequential timeline 304A. As shown in FIG. 3A, the tasks 302A may be projected to have a total completion time of 1 hour and 45 minutes. Furthermore, each of the tasks 302A may include an example video snippet 306A (e.g., a video snippet 237 generated by the generative machine learning model 238 of FIG. 2C) for the corresponding task 302A and a corresponding instrument assignment 308 A (e.g., the instrument assigned to the task by the instrument guidance machine learning model 202).

[0099] As shown in FIG. 3B, the timeline plot 300B of FIG. 3B is configured for an experienced person assigned to perform the planned medical procedure. In contrast to the timeline plot 300A of FIG. 3A and because of the differing experience levels for the persons assigned to the planned procedure, the tasks 302B may project to have a shorter total completion time of 1 hour and 10 minutes vs the 1 hour and 45 minutes for the tasks 302A. Furthermore, a total number of the tasks 302B may be less than a total number of the tasks 302A because of the differences in experience levels between the persons assigned to perform the planned medical procedure with respect the timeline plot 300A of FIG. 3A and the timeline plot 300B of FIG. 3B. Furthermore, the timeline plot 300B may include an example video snippet 306B (e.g., a video snippet 237 generated by the generative machine learning model 238 of FIG. 2C) for the corresponding task 302B and a corresponding instrument assignment 308B (e.g., the instrument assigned to the task by the instrument guidance machine learning model 202). In some embodiments, the example video snippets 306B for the tasks 302B may be different from the example video snippets 306A for the tasks 302A. to reflect the differences in experience levels between the persons assigned to performIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 the planned medical procedure in each case. In particular, the example video snippets 306B and 306A will be different when they include video snippets generated by the generative machine learning model 238 because the generative machine learning model 238 uses the experience level 240 when generating the video snippet 237.

[0100] As shown in FIG. 3C, the timeline plot 300C may include tasks 302C ordered in the sequential timeline 304C. Each of the tasks 302C may include an example surgical task video snippet 306C (e.g., a video snippet 237 generated by the generative machine learning model 238 of FIG. 2C) for the corresponding task 302C and a corresponding instrument assignment 308C (e.g., the instrument assigned to the task by the instrument guidance machine learning model 202). Furthermore, each of the tasks 302C may include OR staff role video snippets 310C (e.g., another video snippet 237 generated by the generative machine learning model 238 of FIG. 2C) and a corresponding OR staff instrument assignment 312C (e.g., the instrument assigned to the task by the instrument guidance machine learning model 202).

[0101] As shown in FIG. 3D, the timeline plot 300D may include tasks 302D ordered in the sequential timeline 304D. Each of the tasks 302D may include an example surgical task video snippet 306D (e.g., a video snippet 237 generated by the generative machine learning model 238 of FIG. 2C) for the corresponding task 302D and a corresponding instrument assignment 308D (e.g., the instrument assigned to the task by the instrument guidance machine learning model 202). Furthermore, each of the tasks 302D may include OR staff role video snippets 310D (e.g., another video snippet 237 generated by the generative machine learning model 238 of FIG. 2C) and a corresponding OR staff instrument assignment 312D (e.g., the instrument assigned to the task by the instrument guidance machine learning model 202). Further still, each of the tasks 302D may include a surgical instrument explanation 314D that provides a natural language explanation for the corresponding instrument assignment 308D and an OR staff instrument explanation 316D that provides a natural language explanation for the corresponding OR staff instrument assignment 312D.

[0102] As shown in FIG. 3E, the timeline plot 300E may include tasks 302E ordered in the sequential timeline 304E. Each of the tasks 302E may include an example surgical task video snippet 306E (e.g., a video snippet 237 generated by the generative machine learning model 238 of FIG. 2C) for the corresponding task 302E and a corresponding instrument assignment 308E (e.g., the instrument assigned to the task by the instrument guidance machine learning model 202). Furthermore, each of the tasks 302E may include OR staff role video snippets 310E (e.g., another video snippet 237 generated by the generative machine learning modelIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519238 of FIG. 2C) and a corresponding OR staff instrument assignment 312E (e.g., the instrument assigned to the task by the instrument guidance machine learning model 202). Further still, each of the tasks 302E may include a surgical instrument explanation 314E that provides a natural language explanation for the corresponding instrument assignment 308E and an OR staff instrument explanation 316E that provides a natural language explanation for the corresponding OR staff instrument assignment 312E. As shown in FIG. 3E, the tasks 302E may include surgeon role assignments 318E linked to the surgical task video snippet 306E, the corresponding instrument assignment 308E, and the surgical instrument explanation 314E. Furthermore, the tasks 302E may include OR staff role assignments 320E linked to the OR staff role video snippets 310E, the corresponding OR staff instrument assignment 312E, and the OR staff instrument explanation 316E. AS shown in FIG. 3E, the OR staff role assignments 320E may also breakdown particular groupings of OR staff assigned to each task 302E (e.g., a first assistant, nurse and scrub tech for task 3 and a first assistant and scrub tech for task 10).

[0103] FIG. 4A is a schematic diagram of a system 400 for using adaptive Al for generating intra-operative guidance in relation to the planned medical procedure. The system 400 includes a plurality of software or hardware modules that may be executed by a processing unit 401 to generate the pre-operative guidance. The processing unit 401 may include the control system 140 and the Al assist module 180 of FIG. 1, the processing unit 201 of the system 200 of FIG. 2A, or other processing components known in the art. For example, the processing unit 401 may include one or more processors, each of which may be a programmable microprocessor or the like that executes software instructions stored in a memory unit 403 (e.g., memory 160 of FIG. 1, memory unit 203 of FIG. 2A, or another suitable memory unit not directly associated with the control system 140 of FIG. 1) to execute some or all of the functions of the system 400 as described herein. The processing unit 401 may include one or more graphics processing units (GPUs) and / or one or more central processing units (CPUs), language processing units (LPUs), for example. Alternatively, or in addition, one or more processors in the processing unit 401 may be other types of processors (e.g., application- specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.), and some of the functionality of the system 400 as described herein may instead be implemented in hardware.

[0104] As shown in FIG. 4A, the system 400 includes a compliance machine learning model 402 which is executable by the processing unit 401. The compliance machine learningIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 model 402 is configured to generate an intra-operative guidance 408 based on inputs of intraoperative multi-modal data 404 and the instrument guidance 208 as directed by the compliance constitution 406.

[0105] The compliance machine learning model 402 may comprise a set of interconnected nodes, layers, trained parameter values (e.g., multiplicative weights, additive bias, etc.), etc. The trained parameters may be set via backpropagation techniques in a training process that uses historical data inputs. In some embodiments, the compliance machine learning model 402 is configured using a multi-modal transformer type architecture that can receive inputs in different modalities relating to the planned medical procedure. However, it should be appreciated that this architecture may be substituted or supplemented with other Al architectures including, but not limited to, convolutional neural network (CNN) architectures, recurrent / recursive neural network (RNN) architectures, sorting / clustering architectures, etc.

[0106] The intra-operative multi-modal data 404 may generally include data obtained from the computer-assisted system 100 during execution of the planned medical procedure. The data may include one or more of real time kinematic data, event data, and force-sensing data from the computer-assisted system 100. The intra-operative multi-modal data 404 may also include real time image or video data of the location associated with the planned medical procedure (e.g., image or video data from an operating room image sensor) or procedure image or video data (e.g., image or video data the is generated by an endoscope such as the imaging device of the instruments 122 of the computer-assisted system 100 of FIG. 1). The intra-operative multi-modal data 404 may also include some or all of the pre-operative multimodal data 204 as described herein and intra-operative user input received by the user interface 216. The intra-operative multi-modal data 404 may be obtained from one or more data sources such as a plurality of data streams from the computer-assisted system 100, user input to the system 400 (e.g., audio input, text inputs, touch interaction with the user interface 216, etc.), and / or the data store 212.

[0107] The compliance constitution 406 directs the compliance machine learning model 402 on how to analyze the intra-operative multi-modal data 404 to generate the intraoperative guidance 408. In particular, the compliance constitution 406 includes a plurality of rules that control or otherwise instruct the compliance machine learning model 402 on how to generate the intra-operative guidance 408. For example, the plurality of rules may include compliance checking rules that control how the compliance machine learning model 402Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 determines that the intra-operative multi-modal data 404 indicates compliance or non- compliance with the instrument guidance 208.

[0108] The compliance checking rules may direct the compliance machine learning model 402 to compare instruments actually used, tasks actually performed, and roles that performed the tasks or used the instruments to the instruments, tasks, role assignments etc. in the instrument guidance 208 and determine conformance to or deviation from the instrument guidance 208 based on the comparison. Furthermore, the compliance checking rules may also direct the compliance machine learning model 402 to generate reminders, suggestions, or instructions in the intra-operative guidance 408 for bringing the operation of the planned medical procedure back into alignment with the instrument guidance 208 when the compliance machine learning model 402 detects a deviation.

[0109] In some embodiments, the plurality of rules in the compliance constitution 406 may also include task monitoring rules. The task monitoring rules may direct the compliance machine learning model 402 to identify when tasks in instrument guidance 208 have been completed and provided instructions, in the intra-operative guidance 408, to personnel participating in the planned medical procedure to prepare a subset of the set of instruments listed in the instrument guidance 208 in preparation for performance of one or more subsequent tasks of the instrument guidance 208. For example, the compliance machine learning model 402 may direct the personnel to open a subset of the set of instruments that had originally been placed in the operating area in an unopened state.

[0110] In some embodiments, the plurality of rules in the compliance constitution 406 may also include unplanned event detection rules that direct the compliance machine learning model 402 to analyze the intra-operative multi-modal data 404 to detect unplanned events (e.g., events not accounted for in the instrument guidance 208, events not typical for the type of procedure begin performed, breaches of sanitation protocols, unplanned interactions with patient anatomy, etc.) and provide, in the intra-operative guidance 408, instruction to personnel to prepare a subset of the set of instruments to address the occurrence of the unplanned event.

[0111] In some embodiments, the plurality of rules in the compliance constitution 406 may also include unplanned event prediction rules that direct the compliance machine learning model 402 to analyze the intra-operative multi-modal data 404 to detect time-based patterns that indicate occurrence of a future unplanned event as part of the medical procedure. The unplanned event prediction rules may also direct the compliance machine learning model 402Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 to provide, in the intra-operative guidance 408, instruction to personnel to prevent the occurrence of the future unplanned event.

[0112] It should also be appreciated that the plurality of rules in the compliance constitution 406 may also include modified or unmodified versions of the set of fixed rules, the set of personnel rules, the set of patient rules, and / or the set of procedure rules described herein. In general, the plurality of rules in the compliance constitution 406 (and / or the other constitutions described herein) may be defined and selected based on the procedure configuration information (e.g., the type of computer-assisted system 100), the type of procedure being performed, and the type of workflow being followed). As described herein the various constitutions described herein may include modifiable rules that can be changed based on user input (e.g., based on user preferences, a user profile, user inputs into a user interface, etc.) and fixed rules that are set do not change based on the user input.

[0113] As shown in FIG. 4A, the compliance machine learning model 402 may include embedding or projection layers 410, additional layers 412, and an output layer 414. The embedding or projection layers 410 may be configured to receive and process different portions of the intra-operative multi-modal data 404 for further processing by the additional layers 412 and the output layer 414. In some embodiments, the embedding or projection layers 410 may be configured to process different modalities of data (text, audio, video, etc.) into a single input format (e.g., natural language text) for the additional layers 412. For example, the real time image or video data of the location associated with the planned medical procedure and the procedure image or video data in the intra-operative multi-modal data 404 may be converted to natural language descriptions of what is depicted in the image or video data and the real time data obtained from the computer-assisted system 100 may be converted into natural language descriptions of the operations being performed and instruments being used.

[0114] The image or video data may be input into a VLM potion (not pictured) of the embedding or projection layers 410 to convert the image or video data into the natural language descriptions understood by the additional layers 412 and the output layer 414. The VLM may be configured to generate text descriptions of features within the image or video data and a location thereof in the image data. These text descriptions may include natural language descriptions of a scene depicted by the image data, such as image data output by an endoscopic instrument. Furthermore, the VLM may be configured to identify objects, such as surgical instruments and devices (e.g., surgical beds or tables, medical devices, displayIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 devices, etc.), individuals and personnel (e.g., clinicians, doctors, medical technicians, etc.) depicted by image data generated by an image sensor having a field of view of an operating room. The parameters of the VLM may be independently trained or tuned from the other parameters of compliance machine learning model 402. In some embodiments, the VLM portion of the embedding or projection layers 410 may be the same as the VLM portion of the embedding or projection layers 218 of the instrument guidance machine learning model 202.

[0115] The real time data obtained from the computer-assisted system 100 may be input into a transformer model, LMM, or similar portion of the embedding or projection layers 410 to convert the real time data into the natural language descriptions understood by the additional layers 412 and the output layer 414. In particular, the transformer model, LMM, etc. may be configured to convert the real time stream of event data, kinematic data, forcesensing data into natural language descriptions of the operations performed, instruments used, complications encountered, etc. during performance of the planned medical procedure. In some embodiments, parameter values of the transformer model, LMM, etc. may be independently trained or tuned from the other parameters of compliance machine learning model 402. In some embodiments, the transformer model, LMM, etc. portion of the embedding or projection layers 410 may be the same as the transformer model, LMM, etc. portion of the embedding or projection layers 218 of the instrument guidance machine learning model 202.

[0116] In some embodiments, portions of the intra-operative multi-modal data 404 that are already in a natural language text form may be passed without modification to the additional layers 412 and the output layer 414. However, in some embodiments, the embedding or projection layers 410 may be configured to summarize or extract key details of these portions of the intra-operative multi-modal data 404. Summarizing and / or extracting key details before further processing by the additional layers 412 and the output layer 414 may help to confine the set of data received by the additional layers 412 to a predefined context window for the additional layers 412.

[0117] The additional layers 412 may comprise various Al or ML type layers known in the art. Such layers include multiheaded self- attention layers, cross-attention layers, multi-layer perceptrons, feed forward layers, softmax layers, etc. In some embodiments, the additional layers 412 and may comprise layers of a pretrained Al model such as an LLM or similar as described herein. This pretrained Al model can include the same or different models from that of the additional layers 220 of the instrument guidance machine learning model 202.Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519

[0118] As shown in FIG. 4A, the compliance constitution 406 may be input into the additional layers 412 to direct how the additional layers 412 process the natural language text inputs generated by the embedding or projection layers 410. However, it should be appreciated that in other cases, the compliance constitution 406 may be input an associated one of the embedding or projection layers 410 before proceeding to the additional layers 412.

[0119] The output layer 414 aggregates outputs from each proceeding layer of the compliance machine learning model 402 (e.g., the embedding or projection layers 410 and the additional layers 412) to generate the output intra-operative guidance 408.

[0120] In general, the intra-operative guidance 408 output from the compliance machine learning model 402 includes material configured to assist, guide, or otherwise instruct the personal presently performing the planned procedure. In particular, the intra-operative guidance 408 may include the reminders, suggestions, or instructions for bringing the operation of the planned medical procedure back into alignment with the instrument guidance 208 when the compliance machine learning model 402 detects the deviation from the instrument guidance 208. The intra-operative guidance 408 may also include an indication of conformance to the instrument guidance 208 when the compliance machine learning model 402 detects conformance to the instrument guidance 208.

[0121] Once the compliance machine learning model 402 generates the intra-operative guidance 408, the processing unit 401 provides the intra-operative guidance 408 to the display unit 214 and / or the user interface 216 to provide the various intra-operative directions and instructions described herein to the personnel performing the planned medical procedure.

[0122] The constitution generator 210 may dynamically populate the compliance constitution 406 with the plurality of rules described herein based on user input received by the user interface 216 and / or from data present in the intra-operative multi-modal data 404. In particular, the constitution generator 210 may be configured to identify and select the plurality of rules to include in the intra-operative guidance 408 from the data store 212 or other similar data storage devices or systems based on the intra-operative multi-modal data 404 and / or the received user input on the user interface 216 or other input module of the system 400. For example, a user may interact with a chatbot interface, an intraoperative imaging interface, etc. to provide the user inputs and / or instructions to the constitution generator 210. Furthermore, in some embodiments, the constitution generator 210 may select the plurality of rules to include in the compliance constitution 406 based on the instrumentIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 guidance 208. For example, the constitution generator 210 may select rules relating to the instruments, roles, tasks, etc. noted in the instrument guidance 208.

[0123] FIG. 4B is a schematic diagram of the system 400 for intra- operatively generating updated instrument guidance 416 using the instrument guidance machine learning model 202. For example, in some embodiments, the plurality of rules in the compliance constitution 406 may direct the compliance machine learning model 402 to generate the intra-operative guidance 408 at least in part as an input into the instrument guidance machine learning model 202 for use in generating the updated instrument guidance 416.

[0124] As shown in FIG. 4B, in some embodiments, the intra-operative guidance 408 may also be input into the constitution generator 210, which generates a revised instrument guidance constitution 418 that controls how the instrument guidance machine learning model 202 generates the updated instrument guidance 416 based on inputs of the instrument guidance 208 and the intra-operative guidance 408. The constitution generator 210 may select the rules to be included in the revised instrument guidance constitution 418 from the data store 212 based on the intra-operative guidance 408. The constitution generator 210 may also generate or select the rules for the revised instrument guidance constitution 418 using some or all of the pre-operative multi-modal data 204 as described herein in connection with FIG. 2A. For example, the constitution generator 210 may use details in the intra-operative guidance 408 (e.g. descriptions on how the procedure has been performed so far, details on any unplanned events that have transpired or are predicted, etc.) and historical procedure data in the pre-operative multi-modal data 204 to select the rules for the revised instrument guidance constitution 418.

[0125] Additionally or alternatively, in some embodiments, the intra-operative guidance 408 generated by the compliance machine learning model 402 may include rules and instructions that control the operation of the instrument guidance machine learning model 202. In these embodiments, the processing unit 401 may not utilize the constitution generator 210 to generate the revised instrument guidance constitution 418.

[0126] The updated instrument guidance 416 may include updates to the task data associated with the one or more subsequent tasks to indicate updated timing information for those tasks based on the current progress of the planned procedure. The updated instrument guidance 416 may also include revisions to the instruments assigned to the subsequent tasks to account for unplanned use of those assigned instruments at earlier portions of the procedure. For example, the updated instrument guidance 416 may direct the personnelIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 assigned to the procedure to procure an additional or substitute instrument not original noted on the instrument guidance 208.

[0127] In some embodiments, the revisions to the task data and instruments may be updates that account for an unplanned event that has occurred or is predicted to occur. Furthermore, in some embodiments, the updated instrument guidance 416 may include revised role assignments for particular tasks and instruments. For example, in cases where the intra-operative guidance 408 indicates that one of the roles noted in the instrument guidance 208 had to unexpectedly leave the operating room, the updated instrument guidance 416 may reassign tasks or instruments assigned to that role to other roles still present. Furthermore, in some embodiments, the instrument guidance machine learning model 202 may also alter the tasks originally included in the instrument guidance 208 to reflect the now missing role or personnel and / or any changes in experience level between the missing personnel and any provided replacement.

[0128] After generated by the instrument guidance machine learning model 202, the processing unit 401 may provide the updated instrument guidance 416 on the display unit 214 and / or the user interface 216. The updated instrument guidance 416 may have a format similar to the timeline plots 300A, 300B, 300C, 300D, and 300E as shown in FIGS. 3A-3E except with revised tasks, instrument assignments, video snippets, role assignments etc. that reflect the data in the intra-operative guidance 408.

[0129] However, in some embodiments, the updated instrument guidance 416 may also include text or other indicators that specifically call attention to the revisions in the updated instrument guidance 416, For example, in some embodiments, the updated instrument guidance 416 may show the tasks, instruments, video snippets, role assignments etc. of the instrument guidance 208 next to the revised tasks, instruments, video snippets, role assignments etc. of the updated instrument guidance 416 with instructions that direct the personnel performing the planned medical procedure to perform the revised tasks, instruments, video snippets, role assignments etc. and not the tasks, instruments, video snippets, role assignments etc. of the instrument guidance 208. In some embodiments, the updated instrument guidance 416 may trigger an alert or warning signal (e.g., a visual, audio, haptic, etc.) to be broadcast at the location where the planned medical procedure is taking place.

[0130] In some embodiments, the user interface 216 may receive feedback on then updated instrument guidance 416. This feedback may include a modification to the updatedIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 instrument guidance 416. In these embodiments, the processing unit 401 is configured to handle the modification to the updated instrument guidance 416 in a manner similar to how the processing unit 201 handles the modification 223 of the instrument guidance 208 as discussed above in connection with the system 200 of FIG. 2A.

[0131] After the planned medical procedure is complete the processing unit 401 or other processing components described herein (e.g., the control system 140, the processing unit 201, etc.) may obtain post-operative data that documents the execution of the planned medical procedure. For example, the post-operative data may document which instruments were utilized during the planned medical procedure, which tasks were performed, which roles performed which tasks etc. The processing unit 401 may then store the post-operative data in the data store 212 or other procedure database for use when generating future pre-operative or intra-operative guidance using the instrument guidance machine learning model 202 or the compliance machine learning model 402 as described herein. For example, the post-operative data in the data store 212 may be used to tune the instrument guidance machine learning model 202 based on any differences between the set of instruments included in the instrument guidance 208 pre-operatively and a set of instruments that were utilized during the execution of the planned medical procedure.

[0132] FIG. 5 is a flow diagram of a computer-implemented method 500 for generating pre-operative instrument guidance using adaptive Al. The method 500 may be performed by a processor system or a control system (such as the processor system 150 and control system 140 of FIG. 1, the processing unit 201 of FIG. 2A, or the processing unit 401 of FIG. 4A). In some embodiments, the control system may implement an Al-assist module (such as the AI- assist module 180) to perform the functionality described with respect to the machine learning models.

[0133] At block 510, the method 500 includes obtaining pre-operative multi-modal data (e.g., pre-operative multi-modal data 204) relating to a planned medical procedure from one or more data sources

[0134] At block 520, the method 500 includes analyzing, via an instrument guidance machine learning model (e.g., instrument guidance machine learning model 202), the preoperative multi-modal data to generate an instrument guidance (e.g., instrument guidance 208), wherein an instrument guidance constitution (e.g., instrument guidance constitution 206) is input into the instrument guidance machine learning model. The instrument guidance constitution includes rules that control how the instrument guidance machine learning modelIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 generates the instrument guidance. The instrument guidance constitution may include a set of fixed rules including one or more rules associated with directing the instrument guidance machine learning model to optimize patient outcomes or reduce waste. The instrument guidance machine learning model may include a large language model (LLM) or a large multi-modal model (LMM).

[0135] At block 530, the method 500 includes receiving the instrument guidance as an output of the instrument guidance machine learning model. The instrument guidance includes an indication of a set of instruments to use when performing the planned medical procedure.

[0136] At block 540, the method 500 includes providing the instrument guidance to a display unit (e.g., display unit 214) to direct preparation for performing the planned medical procedure. The preparations may be performed pre- operatively for an upcoming medical procedure or intra-operatively for an upcoming task or phase of a current medical procedure.

[0137] In some embodiments, providing the instrument guidance includes determining a set of roles that are to perform the planned medical procedures; assigning each role in the set of roles a subset of the set of instruments for use during the planned medical procedure; and providing respective portions of the instrument guidance associated with the subsets of the set of instruments to a display unit associated with the respective role. In these embodiments, the roles may include one or more of a surgeon, a first assistant, a scrub tech, and a nurse. The instrument guidance constitution may also include role rules to adapt a presentation of the respective portions of the instrument guidance based on the respective role.

[0138] In some embodiments, the method 500 may include providing a user interface configured to detect user input data from a user indicative of user feedback. The user interface may be a data source of the one or more data sources. The user feedback may be an acceptance of the instrument guidance, and directing pre-operative preparation by providing the instrument guidance may include providing a fulfillment request for personnel to prepare the set of instruments. The indication of the set of instruments may include indication of a first set of instruments that are required to perform the planned medical procedure and a second set of instruments that are to be on-hand should a need arise. The fulfillment request may indicate that (i) instruments included in the first set of instruments are to be opened or otherwise prepared for usage, and (ii) instruments included in the second set of instruments are to be placed in an operating room in an unopened state. The user feedback may be a request to provide a rationale for the instrument guidance. In these embodiments, the method 500 may include inputting user input data indicative of the request to provide the rationaleIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 into the instrument guidance machine learning model to obtain the requested rationale and updating the instrument guidance to include the obtained rationale.

[0139] In some embodiments, the user feedback may be an indication of a modification of the instrument guidance. In these embodiments, the method 500 may include inputting user input data indicative of the modification into the instrument guidance machine learning model to generate a modified instrument guidance in accordance with the indicated modification. The method 500 may also include providing the modified instrument guidance to the display unit for presentation to the user. The modification may indicate a user preference, a requested substitution, an inclusion of an additional instrument or instrument type. The method 500 may additionally include validating that the user has a role having a permission level that permits the modification to be performed. Similar techniques may be applied to provide modified instrument guidance starting with an instrument guidance provided by a care-team, as opposed to instrument guidance generated by the instrument guidance machine learning model.

[0140] The method 500 may include modifying a set of rules included in the instrument guidance constitution based on the modification. Inputting the user input data may include obtaining inventory data from an instrument inventory system. The instrument inventory system may be a data source of the one or more data sources. The method 500 may also include inputting the inventory data and the user input data into the instrument guidance machine learning model to generate a determination of whether the modification can be fulfilled. The method 500 may additionally include performing, in response to determining that the modification cannot be fulfilled, one or more of (i) generating an alternative instrument guidance, (ii) generating a rationale for the modification not being fulfilled, and (iii) soliciting input from the user as to alternative modification.

[0141] In some embodiments, providing the instrument guidance may include segmenting, via the instrument guidance machine learning model, the planned medical procedure into one or more tasks. In these embodiments, providing the instrument guidance may also include generating, via the instrument guidance machine learning model, task data indicative of one or more of (i) one or more instruments to be used to perform the task, (ii) a person assigned to perform the task or use the one or more instrument to perform the task, and (iii) a rationale for usage of the one or more instruments. The task data may include a video snippet of an example performance of the task. The video snippet may be a synthetic video generated by a generative machine learning model. The method 500 may include providing, to the generativeIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 machine learning model, at least one of an experience level of the person assigned to perform the task and an indication of the one or more instruments to be used to perform the task.

[0142] In some embodiments, providing the instrument guidance may include generating a timeline plot indicative of respective timing for when the one or more tasks are to be performed, wherein the timeline plot indicates the task data. In these embodiments, the method 500 may include providing a user interface that includes the timeline plot. The method 500 may also include obtaining personnel data associated with personnel performing the planned medical procedure from one or more personnel profiles maintained in a personnel database. The personnel database is a data source of the one or more data sources, and the personnel profiles indicate respective levels of experience of the personnel. The method 500 may include obtaining a set of personnel rules associated with the levels of experience to control how the instrument guidance machine learning model determines one or more task presentation characteristics. The task presentation characteristics may include one or more of a task selection, a task granularity, a task duration, a task sequencing, or a task assignment of the one or more tasks included in the timeline plot. The method 500 may also include updating the instrument guidance constitution to include the obtained set of personnel rules. The respective levels of experience may indicate a level of experience associated with performing the planned medical procedure or using one or more instruments available to be used to perform the planned medical procedure.

[0143] The method 500 may also include obtaining procedure data indicative of a procedure type for the planned medical procedure via a user interface or from a procedure scheduling system; obtaining a set of procedure rules associated with the procedure type to control how the instrument guidance machine learning model determines which tasks need to be performed for procedures of the procedure type; and updating the instrument guidance constitution to include the obtained set of procedure rules. The method 500 may include obtaining historical procedure data from a procedure database, wherein the set of procedure rules control how the instrument guidance machine learning model analyzes the historical procedure data to determine which instruments have historically been used to perform the tasks.

[0144] The method 500 may also include obtaining, via a user interface or from a patient medical record, patient data indicative of one or more patient conditions for a patient operated upon during the planned medical procedure; obtaining a set of patient rules associated with the patient conditions to control how the instrument guidance machineIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 learning model determines (i) which tasks need to be performed or (ii) which instruments are to be used to perform the tasks based on the one or more patient conditions; and updating the instrument guidance constitution to include the obtained set of patient rules.

[0145] In some embodiments, the instrument guidance may include pre-operative instrument guidance. Pre-operative instrument guidance as used herein is intended to indicate instrument guidance that was generated from pre-operative multi-modal data (e.g., preoperative multi-modal data 204) and not as temporal indication of when the instrument guidance was generated (even though the pre-operative instrument guidance may still be generated temporally pre-operatively in some embodiments). That is, pre-operative instrument guidance is used to distinguish from intra-operative guidance that is generated based in part on intra-operative multi-modal data (e.g., intra-operative multi-modal data 404).

[0146] The method 500 may include obtaining intra-operative multi-modal data (e.g., intra-operative multi-modal data 404) relating to execution of the planned medical procedure. The intra-operative multi-modal data comprises a plurality of data streams from one or more data sources. The method 500 may also include inputting the intra-operative multi-modal data and the pre-operative instrument guidance into a compliance machine learning model to provide intra-operative guidance for compliance with the pre-operative instrument guidance. The intra-operative multi-modal data may include robotic data obtained from a computer- assisted system assisting in the execution of the planned medical procedure. The robotic data may include one or more of kinematic data, event data, and force-sensing data.

[0147] In some embodiments, providing the intra-operative guidance includes: analyzing, via the compliance machine learning model, the intra-operative multi-modal data to detect a completion of a task; and performing one or more of (i) providing intra-operative guidance to personnel to prepare a subset of the set of instruments in preparation for performance of one or more subsequent tasks and (ii) updating, via the instrument guidance machine learning model, task data associated with the one or more subsequent tasks to indicate updated timing information. Providing the intra-operative guidance may also include: analyzing, via the compliance machine learning model, the intra-operative multi-modal data to detect timebased patterns indicative of an unplanned event that is predicted to occur; and providing intra-operative guidance to personnel to prevent occurrence of the unplanned event.

[0148] The method 500 may include analyzing, via the compliance machine learning model, the intra-operative multi-modal data to detect that an unplanned event has occurred; and providing intra-operative guidance to personnel to prepare a subset of the set ofIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 instruments to address occurrence of the unplanned event. The unplanned event may be a breach of sanitation protocols, or an unplanned interaction with patient anatomy.

[0149] The method 500 may include analyzing, via the compliance machine learning model, the intra-operative multi-modal data to detect that an unplanned event has occurred; and generating an input to the instrument guidance machine learning model to update the preoperative instrument guidance in view of the unplanned event and provide an updated instrument guidance to the display unit.

[0150] In some embodiments, the intra-operative multi-modal data includes image data, and a projection layer of the compliance machine learning model associated with the image data includes a visual-language model (VLM). The VLM may be configured to generate text descriptions of features within the image data and a location thereof in the image data. In these embodiments, the method 500 may include generating the image data by at least one of an endoscope or an operating room image sensor.

[0151] The method 500 may include obtaining post-operative data on execution of the planned medical procedure. The post-operative data may indicate which instruments were utilized during the planned medical procedure. The method 500 may also include storing the post-operative data in a procedure database. The post-operative data in the procedure database may be used to tune the instrument guidance machine learning model based on any differences between the set of instruments included in the instrument guidance pre- operatively and a set of instruments that were utilized during the execution of the planned medical procedure.

[0152] One or more components of the examples discussed in this disclosure, may be implemented in software for execution on one or more processors of a computer system. The software may include code that when executed by the one or more processors, configures the one or more processors to perform various functionalities as discussed herein. The code may be stored in a non-transitory computer readable storage medium (e.g., a memory, magnetic storage, optical storage, solid-state storage, etc.). The computer readable storage medium may be part of a computer readable storage device, such as an electronic circuit, a semiconductor device, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device. The code may be downloaded via computer networks such as the Internet, Intranet, etc. for storage on the computer readable storage medium. The code may be executed by any of a wide variety of centralized or distributedIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 data processing architectures. The programmed instructions of the code may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the systems described herein. The components of the computing systems discussed herein may be connected using wired and / or wireless connections. In some examples, the wireless connections may use wireless communication protocols such as Bluetooth, near-field communication (NFC), Infrared Data Association (IrDA), home radio frequency (HomeRF), IEEE 502.11, Digital Enhanced Cordless Telecommunications (DECT), and wireless medical telemetry service (WMTS).

[0153] Various general-purpose computer systems may be used to perform one or more processes, methods, or functionalities described herein. Additionally or alternatively, various specialized computer systems may be used to perform one or more processes, methods, or functionalities described herein. In addition, a variety of programming languages may be used to implement one or more of the processes, methods, or functionalities described herein.

[0154] While certain examples and examples have been described above and shown in the accompanying drawings, it is to be understood that such examples and examples are merely illustrative and are not limited to the specific constructions and arrangements shown and described, since various other alternatives, modifications, and equivalents will be appreciated by those with ordinary skill in the art.

Claims

Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519What is claimed is:

1. A computer system comprising: one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the computer system to: obtain pre-operative multi-modal data relating to a planned medical procedure from one or more data sources; analyze, via an instrument guidance machine learning model, the pre-operative multi-modal data to generate an instrument guidance, wherein: an instrument guidance constitution is input into the instrument guidance machine learning model, and the instrument guidance constitution includes rules that control how the instrument guidance machine learning model generates the instrument guidance, receive the instrument guidance as an output of the instrument guidance machine learning model, wherein the instrument guidance includes an indication of a set of instruments to use when performing the planned medical procedure; and provide the instrument guidance to a display unit to direct preparation for performing the planned medical procedure.

2. The computer system of claim 1, wherein to provide the instrument guidance, the instructions, when executed, cause the system to: determine a set of roles that are to perform the planned medical procedure; assign each role in the set of roles a subset of the set of instruments for use during the planned medical procedure; and provide respective portions of the instrument guidance associated with the subsets of the set of instruments to a display unit associated with the respective role.

3. The computer system of claim 2 wherein the roles include one or more of a surgeon, a first assistant, a scrub tech, and a nurse.Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 705194. The computer system of claim 2 wherein the instrument guidance constitution includes role rules to adapt a presentation of the respective portions of the instrument guidance based on the respective role.

5. The computer system of claim 1 wherein the instructions, when executed, cause the computer system to: provide a user interface configured to detect user input data from a user indicative of user feedback, wherein the user interface is a data source of the one or more data sources.

6. The computer system of claim 5, wherein the user feedback is an acceptance of the instrument guidance, and to direct pre-operative preparation, the instructions, when executed, cause the computer system to: provide a fulfillment request for personnel to prepare the set of instruments.

7. The computer system of claim 6, wherein:The indication of the set of instruments includes indication of a first set of instruments that are required to perform the planned medical procedure and a second set of instruments that are to be on-hand should a need arise, and the fulfillment request indicates that (i) instruments included in the first set of instruments are to be opened or otherwise prepared for usage, and (ii) instruments included in the second set of instruments are to be placed in an operating room in an unopened state.

8. The computer system of claim 5, wherein the user feedback is an indication of a modification of the instrument guidance and the instructions, when executed, cause the computer system to: input user input data indicative of the modification into the instrument guidance machine learning model to generate a modified instrument guidance in accordance with the indicated modification; and provide the modified instrument guidance to the display unit for presentation to the user.

9. The computer system of claim 8, wherein the modification indicates a user preference, a requested substitution, or an inclusion of an additional instrument or instrument type.Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 7051910. The computer system of claim 8, wherein the instructions, when executed, cause the computer system to: modify a set of rules included in the instrument guidance constitution based on the modification.

11. The computer system of claim 8, wherein to input the user input data, the instructions, when executed, cause the computer system to: obtain inventory data from an instrument inventory system, wherein the instrument inventory system is a data source of the one or more data sources; input the inventory data and the user input data into the instrument guidance machine learning model to generate a determination of whether the modification can be fulfilled; and in response to determining that the modification cannot be fulfilled, perform one or more of (i) generating an alternative instrument guidance, (ii) generating a rationale for the modification not being fulfilled, and (iii) soliciting input from the user as to alternative modification.

12. The computer system of claim 8, wherein the instructions, when executed, cause the computer system to: validate that the user has a role having a permission level that permits the modification to be performed.

13. The computer system of claim 5, wherein the user feedback is a request to provide a rationale for the instrument guidance and the instructions, when executed, cause the computer system to: input user input data indicative of the request to provide the rationale into the instrument guidance machine learning model to obtain the requested rationale; and update the instrument guidance to include the obtained rationale.

14. The computer system of claim 1, wherein to provide the instrument guidance, the instructions, when executed, cause the computer system to: segment, via the instrument guidance machine learning model, the planned medical procedure into one or more tasks; andIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 generate, via the instrument guidance machine learning model, task data indicative of one or more of (i) one or more instruments to be used to perform the task, (ii) a person assigned to perform the task or use the one or more instruments to perform the task, and (iii) a rationale for usage of the one or more instruments.

15. The computer system of claim 14 wherein the task data includes a video snippet of an example performance of the task.

16. The computer system of claim 15, wherein the video snippet is a synthetic video generated by a generative machine learning model.

17. The computer system of claim 16, wherein to generate the video snippet, the computer system is configured to provide to the generative machine learning model at least one of an experience level of the person assigned to perform the task and an indication of the one or more instruments to be used to perform the task.

18. The computer system of claim 14, wherein to provide the instrument guidance, the instructions, when executed, cause the computer system to: generate a timeline plot indicative of respective timing for when the one or more tasks are to be performed, wherein the timeline plot indicates the task data.

19. The computer system of claim 18, wherein the instructions, when executed, cause the computer system to: provide a user interface that includes the timeline plot.

20. The computer system of claim 18, wherein the instructions, when executed, cause the computer system to: obtain personnel data associated with personnel performing the planned medical procedure from one or more personnel profiles maintained in a personnel database, wherein: the personnel database is a data source of the one or more data sources, and the personnel profiles indicate respective levels of experience of the personnel; obtain a set of personnel rules associated with the levels of experience to control how the instrument guidance machine learning model determines one or more task presentationIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 characteristics, wherein the task presentation characteristics include one or more of a task selection, a task granularity, a task duration, a task sequencing, or a task assignment of the one or more tasks included in the timeline plot; and update the instrument guidance constitution to include the obtained set of personnel rules.

21. The computer system of claim 20, wherein the respective levels of experience indicate a level of experience associated with performing the planned medical procedure or using one or more instruments available to be used to perform the planned medical procedure.

22. The computer system of claim 1, wherein the instructions, when executed, cause the computer system to: obtain procedure data indicative of a procedure type for the planned medical procedure via a user interface or from a procedure scheduling system; obtain a set of procedure rules associated with the procedure type to control how the instrument guidance machine learning model determines which tasks need to be performed for procedures of the procedure type; and update the instrument guidance constitution to include the obtained set of procedure rules.

23. The computer system of claim 22, wherein the instructions, when executed, cause the computer system to: obtain historical procedure data from a procedure database, wherein the set of procedure rules control how the instrument guidance machine learning model analyzes the historical procedure data to determine which instruments have historically been used to perform the tasks.

24. The computer system of claim 1, wherein the instructions, when executed, cause the computer system to: obtain, via a user interface or from a patient medical record, patient data indicative of one or more patient conditions for a patient operated upon during the planned medical procedure;Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 obtain a set of patient rules associated with the patient conditions to control how the instrument guidance machine learning model determines (i) which tasks need to be performed or (ii) which instruments are to be used to perform the tasks based on the one or more patient conditions; and update the instrument guidance constitution to include the obtained set of patient rules.

25. The computer system of claim 1, wherein the instrument guidance constitution includes a set of fixed rules including one or more rules associated with directing the instrument guidance machine learning model to optimize patient outcomes or reduce waste.

26. The computer system of claim 1, wherein the instrument guidance machine learning model includes a large language model (LLM) or a large multi-modal model (LMM).

27. The computer system of any one of claims 1-26 wherein the instrument guidance is pre-operative instrument guidance and the instructions, when executed by the one or more processors, cause the computer system to: obtain intra-operative multi-modal data relating to execution of the planned medical procedure, wherein the intra-operative multi-modal data comprises a plurality of data streams from one or more data sources, and input the intra-operative multi-modal data and the pre-operative instrument guidance into a compliance machine learning model to provide intra-operative guidance for compliance with the pre-operative instrument guidance.

28. The computer system of claim 27, wherein to provide intra-operative guidance, the computer system is configured to: analyze, via the compliance machine learning model, the intra-operative multi-modal data to detect a completion of a task; and perform one or more of (i) providing intra-operative guidance to personnel to prepare a subset of the set of instruments in preparation for performance of one or more subsequent tasks and (ii) updating, via the instrument guidance machine learning model, task data associated with the one or more subsequent tasks to indicate updated timing information.Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 7051929. The computer system of claim 27, wherein to provide intra-operative guidance, the computer system is configured to: analyze, via the compliance machine learning model, the intra-operative multi-modal data to detect time-based patterns indicative of an unplanned event that is predicted to occur; and provide intra-operative guidance to personnel to prevent an occurrence of the unplanned event.

30. The computer system of claim 27, wherein to provide intra-operative guidance, the computer system is configured to: analyze, via the compliance machine learning model, the intra-operative multi-modal data to detect that an unplanned event has occurred; and provide intra-operative guidance to personnel to prepare a subset of the set of instruments to address an occurrence of the unplanned event.

31. The computer system of claim 30, wherein the unplanned event is a breach of sanitation protocols, or an unplanned interaction with patient anatomy.

32. The computer system of claim 27, wherein the instructions, when executed, cause the computer system to: analyze, via the compliance machine learning model, the intra-operative multi-modal data to detect that an unplanned event has occurred; and generate an input to the instrument guidance machine learning model to update the pre-operative instrument guidance in view of the unplanned event and provide an updated instrument guidance to the display unit.

33. The computer system of claim 27, wherein: the intra-operative multi-modal data includes image data, and a projection layer of the compliance machine learning model associated with the image data includes a visual-language model (VLM) configured to generate text descriptions of features within the image data and a location thereof in the image data.Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 7051934. The computer system of claim 33, wherein the image data is generated by at least one of an endoscope or an operating room image sensor.

35. The computer system of claim 27, wherein: the intra-operative multi-modal data includes robotic data obtained from a computer- assisted system assisting in the execution of the planned medical procedure, and the robotic data includes one or more of kinematic data, event data, and force-sensing data.

36. The computer system of any one of claims 1-35, wherein the instructions, when executed by the one or more processors, cause the computer system to: obtain post-operative data on execution of the planned medical procedure, the postoperative data indicating which instruments were utilized during the planned medical procedure; and storing the post-operative data in a procedure database.

37. The computer system of claim 36, wherein the post-operative data in the procedure database is used to tune the instrument guidance machine learning model based on any differences between the set of instruments included in the pre-operative instrument guidance and a set of instruments that were utilized during the execution of the planned medical procedure.

38. A computer-implemented method comprising : obtaining pre-operative multi-modal data relating to a planned medical procedure from one or more data sources; analyzing, via an instrument guidance machine learning model, the pre-operative multi-modal data to generate an instrument guidance, wherein: an instrument guidance constitution is input into the instrument guidance machine learning model, and the instrument guidance constitution includes rules that control how the instrument guidance machine learning model generates the instrument guidance,Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 receiving the instrument guidance as an output of the instrument guidance machine learning model, wherein the instrument guidance includes an indication of a set of instruments to use when performing the planned medical procedure; and providing the instrument guidance to a display unit to direct preparation for performing the planned medical procedure.

39. The computer-implemented method of claim 38, wherein providing the instrument guidance includes: determining a set of roles that are to perform the planned medical procedures; assigning each role in the set of roles a subset of the set of instruments for use during the planned medical procedure; and providing respective portions of the instrument guidance associated with the subsets of the set of instruments to a display unit associated with the respective role.

40. The computer-implemented method of claim 39 wherein the roles include one or more of a surgeon, a first assistant, a scrub tech, and a nurse.

41. The computer- implemented method of claim 39 wherein the instrument guidance constitution includes role rules to adapt a presentation of the respective portions of the instrument guidance based on the respective role.

42. The computer-implemented method of claim 38 further comprising: providing a user interface configured to detect user input data from a user indicative of user feedback, wherein the user interface is a data source of the one or more data sources.

43. The computer-implemented method of claim 42, wherein the user feedback is an acceptance of the instrument guidance, and directing pre-operative preparation by providing the instrument guidance includes: providing a fulfillment request for personnel to prepare the set of instruments.

44. The computer-implemented method of claim 43, wherein:Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 the indication of the set of instruments includes indication of a first set of instruments that are required to perform the planned medical procedure and a second set of instruments that are to be on-hand should a need arise, and the fulfillment request indicates that (i) instruments included in the first set of instruments are to be opened or otherwise prepared for usage, and (ii) instruments included in the second set of instruments are to be placed in an operating room in an unopened state.

45. The computer-implemented method of claim 42, wherein the user feedback is an indication of a modification of the instrument guidance and further comprising: inputting user input data indicative of the modification into the instrument guidance machine learning model to generate a modified instrument guidance in accordance with the indicated modification; and providing the modified instrument guidance to the display unit for presentation to the user.

46. The computer-implemented method of claim 45, wherein the modification indicates a user preference, a requested substitution, an inclusion of an additional instrument or instrument type.

47. The computer- implemented method of claim 45, further comprising: modifying a set of rules included in the instrument guidance constitution based on the modification.

48. The computer- implemented method of claim 45, wherein inputting the user input data includes: obtaining inventory data from an instrument inventory system, wherein the instrument inventory system is a data source of the one or more data sources; inputting the inventory data and the user input data into the instrument guidance machine learning model to generate a determination of whether the modification can be fulfilled; and performing, in response to determining that the modification cannot be fulfilled, one or more of (i) generating an alternative instrument guidance, (ii) generating a rationale for theIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 modification not being fulfilled, and (iii) soliciting input from the user as to alternative modification.

49. The computer-implemented method of claim 45, further comprising: validating that the user has a role having a permission level that permits the modification to be performed.

50. The computer-implemented method of claim 42, wherein the user feedback is a request to provide a rationale for the instrument guidance and further comprising: inputting user input data indicative of the request to provide the rationale into the instrument guidance machine learning model to obtain the requested rationale; and updating the instrument guidance to include the obtained rationale.

51. The computer- implemented method of claim 38, wherein providing the instrument guidance includes: segmenting, via the instrument guidance machine learning model, the planned medical procedure into one or more tasks; and generating, via the instrument guidance machine learning model, task data indicative of one or more of (i) one or more instruments to be used to perform the task, (ii) a person assigned to perform the task or use the one or more instrument to perform the task, and (iii) a rationale for usage of the one or more instruments.

52. The computer- implemented method of claim 51 wherein the task data includes a video snippet of an example performance of the task.

53. The computer-implemented method of claim 52, wherein the video snippet is a synthetic video generated by a generative machine learning model.

54. The computer-implemented method of claim 53, further comprising: providing, to the generative machine learning model, at least one of an experience level of the person assigned to perform the task and an indication of the one or more instruments to be used to perform the task.Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 7051955. The computer- implemented method of claim 51, wherein providing the instrument guidance includes: generating a timeline plot indicative of respective timing for when the one or more tasks are to be performed, wherein the timeline plot indicates the task data.

56. The computer-implemented method of claim 55, further comprising: providing a user interface that includes the timeline plot.

57. The computer-implemented method of claim 55, further comprising: obtaining personnel data associated with personnel performing the planned medical procedure from one or more personnel profiles maintained in a personnel database, wherein: the personnel database is a data source of the one or more data sources, and the personnel profiles indicate respective levels of experience of the personnel; obtaining a set of personnel rules associated with the levels of experience to control how the instrument guidance machine learning model determines one or more task presentation characteristics, wherein the task presentation characteristics include one or more of a task selection, a task granularity, a task duration, a task sequencing, or a task assignment of the one or more tasks included in the timeline plot; and updating the instrument guidance constitution to include the obtained set of personnel rules.

58. The computer- implemented method of claim 57, wherein the respective levels of experience indicate a level of experience associated with performing the planned medical procedure or using one or more instruments available to be used to perform the planned medical procedure.

59. The computer-implemented method of claim 38, further comprising: obtaining procedure data indicative of a procedure type for the planned medical procedure via a user interface or from a procedure scheduling system; obtaining a set of procedure rules associated with the procedure type to control how the instrument guidance machine learning model determines which tasks need to be performed for procedures of the procedure type; andIntuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 updating the instrument guidance constitution to include the obtained set of procedure rules.

60. The computer-implemented method of claim 59, further comprising: obtaining historical procedure data from a procedure database, wherein the set of procedure rules control how the instrument guidance machine learning model analyzes the historical procedure data to determine which instruments have historically been used to perform the tasks.

61. The computer- implemented method of claim 38, further comprising: obtaining, via a user interface or from a patient medical record, patient data indicative of one or more patient conditions for a patient operated upon during the planned medical procedure; obtaining a set of patient rules associated with the patient conditions to control how the instrument guidance machine learning model determines (i) which tasks need to be performed or (ii) which instruments are to be used to perform the tasks based on the one or more patient conditions; and updating the instrument guidance constitution to include the obtained set of patient rules.

62. The computer-implemented method of claim 38, wherein the instrument guidance constitution includes a set of fixed rules including one or more rules associated with directing the instrument guidance machine learning model to optimize patient outcomes or reduce waste.

63. The computer- implemented method of claim 38, wherein the instrument guidance machine learning model includes a large language model (LLM) or a large multi-modal model (LMM).

64. The computer-implemented method of any one of claims 38-63 wherein the instrument guidance is pre-operative instrument guidance and further comprising:Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 obtaining intra-operative multi-modal data relating to execution of the planned medical procedure, wherein the intra-operative multi-modal data comprises a plurality of data streams from one or more data sources, and inputting the intra-operative multi-modal data and the instrument guidance into a compliance machine learning model to provide intra-operative guidance for compliance with the pre-operative instrument guidance.

65. The computer- implemented method of claim 64, wherein providing the intraoperative guidance includes: analyzing, via the compliance machine learning model, the intra-operative multimodal data to detect a completion of a task; and performing one or more of (i) providing intra-operative guidance to personnel to prepare a subset of the set of instruments in preparation for performance of one or more subsequent tasks and (ii) updating, via the instrument guidance machine learning model, task data associated with the one or more subsequent tasks to indicate updated timing information.

66. The computer-implemented method of claim 64, wherein providing the intraoperative guidance includes: analyzing, via the compliance machine learning model, the intra-operative multimodal data to detect time-based patterns indicative of an unplanned event that is predicted to occur; and providing intra-operative guidance to personnel to prevent an occurrence of the unplanned event.

67. The computer- implemented method of claim 64, further comprising: analyzing, via the compliance machine learning model, the intra-operative multimodal data to detect that an unplanned event has occurred; and providing intra-operative guidance to personnel to prepare a subset of the set of instruments to address an occurrence of the unplanned event.

68. The computer-implemented method of claim 67, wherein the unplanned event is a breach of sanitation protocols, or an unplanned interaction with patient anatomy.Intuitive Docket No.: P06952-WO Attorney Docket No.: 33685 / 7051969. The computer-implemented method of claim 64, further comprising: analyzing, via the compliance machine learning model, the intra-operative multimodal data to detect that an unplanned event has occurred; and generating an input to the instrument guidance machine learning model to update the pre-operative instrument guidance in view of the unplanned event and provide an updated instrument guidance to the display unit.

70. The computer-implemented method of claim 64, wherein: the intra-operative multi-modal data includes image data, and a projection layer of the compliance machine learning model associated with the image data includes a visual-language models (VLM) configured to generate text descriptions of features within the image data and a location thereof in the image data.

71. The computer- implemented method of claim 70, further comprising: generating the image data by at least one of an endoscope or an operating room image sensor.

72. The computer-implemented method of claim 64, wherein: the intra-operative multi-modal data includes robotic data obtained from a computer- assisted system assisting in the execution of the planned medical procedure, and the robotic data includes one or more of kinematic data, event data, and force-sensing data.

73. The computer-implemented method of any one of claims 38-72, further comprising: obtaining post-operative data on execution of the planned medical procedure, the post-operative data indicating which instruments were utilized during the planned medical procedure; and storing the post-operative data in a procedure database.

74. The computer-implemented method of claim 73, wherein the post-operative data in the procedure database is used to tune the instrument guidance machine learning model based on any differences between the set of instruments included in the instrument guidance pre-Intuitive Docket No.: P06952-WOAttorney Docket No.: 33685 / 70519 operatively and a set of instruments that were utilized during the execution of the planned medical procedure.

75. A non-transitory machine-readable medium comprising a plurality of machine- readable instructions that when executed by one or more processors are adapted to cause the one or more processors to perform the method of any one of claims 38-74.

Images