Backtable optimization using real-time surgical activity recognition

Abstract

Methods and apparatus (e.g., systems, particularly including software) for assisting a technician (e.g., surgeon, surgical technician, nurse, assistant, etc.) in preparing one or more tools for use in one or more surgical procedures in order to efficiently support medical (e.g., surgical) procedures.

Description

Technical Field

[0001] Priority Claim

[0001] This patent application claims priority to U.S. Provisional Patent Application No. 63 / 508,873, titled "BACK TABLE OPTIMIZATION USING REAL TIME SURGICAL ACTIVITY RECOGNITION", filed on Jun. 16, 2023, which is hereby incorporated by reference in its entirety.

Background Art

[0002]

[0002] An operating room has a sterile environment and a non-sterile environment. The sterile environment is where surgeons, surgical assistants, and surgical technicians (e.g., scrub technicians) work. The main duty of a scrub technician in an operating room is to prepare and maintain an area called the back table within the sterile area. The scrub technician lays out surgical instruments that the surgeon may need and passes the correct instrument to the surgeon at the correct time.

[0003]

[0003] Appropriate coordination between the scrub technician and the surgeon is essential for an efficient surgical procedure. A surgical operation can take two to three times longer if the coordination breaks down. Usually, a scrub technician is a nurse who supports multiple surgeons who have their own preferences regarding surgical tools and techniques. Furthermore, in case of unexpected pathology, additional interventions and surgical tasks may start that require additional tools.

Summary of the Invention

Means for Solving the Problems

[0004]

[0004] Methods and apparatus (e.g., systems including software in particular) for assisting a technician (e.g., a surgeon, surgical technician, nurse, assistant, etc.) in preparing one or more tools for use in one or more surgical procedures in order to efficiently support medical (e.g., surgical) procedures are described herein.

[0005]

[0005] In some embodiments, the present invention includes hardware, software, and / or firmware configured to perform automated or semi-automated procedures, the automated or semi-automated procedures including processing video input from one or more surgical fields in real time (e.g., forming one or more video streams), optionally processing video input from one or more cameras imaging a back table (e.g., positioned on a back table) (e.g., forming one or more video streams), analyzing the video streams in real time using deep learning and computer vision techniques, and recognizing surgical actions and surgical context, i.e., surgical procedures / subprocedures in which these actions occur. The surgical context may include overall situational awareness, which may include recognizing anatomical structures and / or pathologies encountered in the surgical field, preceding actions, the patient's specific medical history, etc. Generally, these methods and devices may include anticipating upcoming surgical subprocedures to be performed. In some examples, these methods and devices may include mapping upcoming subprocedures to a specific sequence of surgical actions. The sequence may be based on a predetermined schedule (e.g., corresponding to the procedures to be performed) and / or on the surgeon's preferences. These methods and devices may be configured to anticipate the surgical tools that will be needed by the surgeon at a particular point in time based on their recognition of the surgical context within the field of view.

[0006]

[0006] In any of these examples, the methods and apparatus described herein may (1) instruct a scrub technician to provide (e.g., lay out) one or more sets of tools that will be required for a particular surgical subprocedure, (2) instruct a scrub technician to hand a particular tool / implant to the surgeon at one or more specific time points, (3) optionally, the method or apparatus performing the method may also recognize the layout of instruments on the back table by analyzing a video feed from an external camera overlooking the back table and confirm that the layout of instruments matches the expected surgical activity, subject to the surgeon's preferences for technique and tools, and (4) the method and apparatus may notify a scrub technician to change the surgeon's tool needs in the event of an emergency in the field of view. The tools may be changed by replacing, adding or omitting one or more particular tools. For example, if the method or apparatus performing the method detects excessive bleeding in the field of view, it may prompt the surgeon to pause the current surgical activity and take a cauter to seal the wound before resuming the task at hand.

[0007]

[0007] For example, a method for providing surgical guidance to a scrub technician during a surgical procedure may include using a real-time surgical context recognition module to identify one or more surgical procedures to be performed on a patient in a sterile field, the real-time surgical context recognition module receiving and identifying one or more video streams of the surgical procedures to be performed and one or more video streams of the back table in the sterile field, determining a sequence of surgical tools that will be required to perform the identified one or more surgical procedures using a back table instruction processor module including a trained machine learning agent, and outputting each of the surgical tools in the sequence to a monitor that can be viewed in the sterile field, the surgical tools being presented sequentially for placement on the back table in the sterile field, receiving input from a back table camera observing the back table, and the back table instruction processor module verifying that the surgical tools in the sequence have been presented on the back table.

[0008]

[0008] Systems for carrying out any of these methods are also described herein. Generally, these systems may include one or more processors and memory coupled to one or more processors. The memory may hold computer program instructions, which, when executed by one or more processors, carry out the method. For example, the memory may hold computer program instructions, which, when executed by one or more processors, carry out the method, using a real-time surgical context recognition module to identify one or more surgical procedures to be performed on a patient in a sterile field, the real-time surgical context recognition module receiving one or more video streams of the surgical procedures to be performed and one or more video streams of the back table in the sterile field, and using a back table instruction processor including a trained machine learning agent to determine a sequence of surgical tools that will be required to carry out the identified one or more surgical procedures, and outputting each of the surgical tools in the sequence to a monitor that can be viewed in the sterile field, the surgical tools being presented sequentially for placement on the back table in the sterile field.

[0009]

[0009] All of the methods and apparatus described herein can be used in any combination to achieve the benefits intended and described herein.

[0010] A better understanding of the features and advantages of the methods and apparatus described herein can be obtained by referring to the following detailed description and accompanying drawings that illustrate illustrative embodiments. [Brief explanation of the drawing]

[0010] [Figure 1]

[0011] This figure schematically illustrates an example of the apparatus described herein. [Figure 2]

[0012] This figure shows an example of a visual output for scrub technicians. [Figure 3]

[0013] This figure shows an example of a visual output for scrub technicians. [Figure 4]

[0014] This figure shows an example of a visual output for scrub technicians. [Modes for carrying out the invention]

[0011]

[0015] Surgical activity often involves performing a specific action using a certain instrument on a given set of anatomical structures within a particular region. In this sense, surgical activity may be described as a more abstract concept compared to surgical task. At times, the activity may consist of several individual surgical actions, and in other cases, the activity may consist of fewer individual steps. Methods and apparatus may refer to an abstract form of surgical activity that includes a surgical procedure comprising a configuration of several activities performed in a given sequence.

[0012]

[0016] For example, the methods and apparatus described herein may each include a real-time surgical context recognition (SCR) module and a back-table instruction processor (BTIP), which are described in more detail herein. Generally, these methods and apparatus, in particular the real-time surgical context recognition module and back-table instruction processor, may address problems that have been difficult or impossible to successfully solve using existing techniques. The CR and BTIP may be part of a single system or subsystem. These methods and apparatus may also enable rapid and effective (e.g., real-time) assistance during surgical procedures by more efficiently processing data from one or more video streams, extracting essential information from the correct video stream, and providing information or output regarding actions to support (often time-sensitive) surgical procedures. Real-time surgical context recognition module

[0017] A real-time surgical context recognition module, also referred to herein as SCR (or Real-time SCR), may be implemented using a real-time video processing system which may employ machine learning (e.g., artificial intelligence, AI). The SCR may also be referred to as a system for providing back-table instructions to, for example, nurses or technicians, particularly medical support staff such as scrub technicians. This system may be part of or used with another surgical assistance / guidance system or subsystem, which may organize video streams and data extracted from video streams among several submodules performing specific tasks. These submodules may be implemented using machine learning (e.g., deep learning) and computer vision techniques, which may be modified and implemented as described herein. The SCR may also employ temporary internal storage labeled as Temporary Surgical Context Storage (TSC). Figure 1 shows an example of a circuit diagram of a system (e.g., SCR) including a TSC. In the diagram of Figure 1, the SCR including a TSC is configured to recognize surgical activities and procedures and interpret them in a context that can provide medical support staff with meaningful and useful context (output).

[0013]

[0018] In Figure 1, the system is schematically shown as a plurality of interconnected modules that may provide direct signals, i.e., outputs into and from the TSC. In this example, the system (real-time SCR system 101) may include an anatomical structure recognition module 112, a pathology recognition module 116, an anatomical structure region recognition module 114, and a tool recognition module 110. These modules may utilize the context provided by the inputs (e.g., surgical video input 122 and / or patient clinical data 120) and / or outputs of the Temporary Surgical Context Storage Module (TSC) 118 to increase the reliability of the recognition tasks performed by the modules. For example, in the case of the tool recognition module 110, the TSC may be used as shown below.

[0014]

[0019] For example, while a surgeon is manipulating one or more surgical tools, the endoscope camera (providing surgical video camera input 122) may temporarily lose sight of the tool or may see an unidentifiable portion of the tool. The tool recognition module 110 may use the TSC 118 to make its final prediction by utilizing temporary storage and confidently determine by rule of thumb whether the tool may have been altered since it was last recognized. To increase the reliability of recognition, the tool recognition module may apply various techniques, including object tracking, tool hue changes, and smoothing of the output over time.

[0015]

[0020] Similarly, the anatomical structure 112, pathology 116, and anatomical structure region 114 recognition modules may also clarify structures by utilizing context, for example, a field of view from a few seconds ago providing information about where the camera is currently positioned.

[0016]

[0021] High-level modules may utilize the context provided by the TSC to perform those tasks. surgical activity recognition

[0022] In some cases, for example, a machine learning agent for recognizing surgical activity may be configured as a transformer network. One or more transformer networks (e.g., as an example of a certain type of deep learning technique that may be used herein) may be used to output a sequence of tokens based on a sequence of input tokens. Examples may include transformers used for language translation and sentence completion tasks. Transformers have also been used to recognize sports activity from video streams. The methods and apparatus described herein may provide a particular variation of a transformer network, referred to herein as videoBERT, which represents a video bidirectional transformer. In a general sense, these networks may be trained to output a text description of a video stream, and these networks model the joint distribution of text and video data.

[0017]

[0023] The methods and apparatus described herein may represent improvements to conventional implementations of transformer-based networks that may be used for surgical activity recognition that might otherwise fail in the surgical domain. Described herein are approaches to the use of transformer-based surgical activity recognition that may utilize real-time tool, anatomical structure, pathology, and anatomical conception domain recognition models. These methods and apparatus may use feature extraction performed on the outputs of these models, and feature vectors may be assembled for each frame in the input video stream. In some examples, subject experts may provide extensive text descriptions of various activities, tools, anatomical structures, and pathologies observed in the video stream (e.g., for training). These text descriptions and features may be extracted from the video stream as described above and used to construct a joint distribution across a feature matrix and a sequence of surgical semantic tokens generated by the subject expert. Surgical Procedure Recognition

[0024] The methods and apparatuses described herein may also recognize surgical activities in a more abstract form than a surgical procedure, which is a configuration of several activities performed in a given sequence. For that purpose, any of these systems may include a surgical activity recognition module 106. The surgical recognition module may include, for example, a trained machine learning agent that may recognize the surgical activities being performed from video input 122 and / or from patient clinical data. Backtable instruction processor

[0025] Any of these methods and apparatuses may include a real-time surgical context recognition module 101 and a backtable instruction processor 130 that may receive inputs from the same or different iterations of the tool recognition module 110' and from a particular backtable camera 115. The backtable instruction processor may also access one or more databases including a surgical preference database 132 and / or a tool / implant database 134. The backtable instruction processor may synthesize this information to prepare instructions for display to the medical support staff (e.g., graphical and / or text output), and based on the SCR output, may proactively and intelligently provide instructions regarding which tools the surgeon may need, and may verify that the correct tools are being provided / prepared within the sterile field (e.g., from the backtable camera / video input and the tool recognition module 110'). The backtable instruction processor may also provide context-specific and appropriate text output in cooperation with the physician (surgeon) performing the procedure.

[0018]

[0026] Generally, this output may be provided to medical support staff (e.g., scrub technicians) within the sterile field on a dedicated display. This display (user interface) may be interactive or non - interactive with respect to the medical support staff. The output may respond in real - time to an ongoing surgical procedure as well as to actions by medical support staff on the back - table area when preparing tools for access by the surgeon.

[0019]

[0027] These systems (e.g., back - table instruction processor modules, BTIP) may be dynamic and may respond to changes in the procedure based on the SCR module and / or the back - table camera and tool recognition module.

[0020]

[0028] Thus, when surgical activity is recognized in real - time, the BTIP module may translate this into specific instructions for the scrub technician. This module may decouple the preferences of the surgeon, i.e., the specific tools and techniques used by a given surgeon, from the recognition of the activity itself. In this way, the system can be easily configured to meet the needs of different surgeons simply by changing the surgeon preferences and the tool / implant database. BTIP may map the surgical activities predicted by the SAR module to the sequence of tools required by the surgeon.

[0021]

[0029] For example, Figures 2-4 show examples of outputs that may be provided by the backtable instruction processor module 130, as described herein. Figures 2-4 show screenshots from the output of the backtable instruction processor, illustrating how a support technician can assist a surgeon. For example, a system including the backtable instruction processor module may recognize surgical activities (e.g., "Suture Delivery," "Femoral Drilling Preparation," etc.) from the real-time surgical context recognition module 101 within a corresponding surgical procedure.

[0022]

[0030] In some cases, the backtable instruction processor may use a trained machine learning agent to synthesize inputs from the real-time SCR101 and backtable camera (and / or tool recognition module 110'), and in some cases from the surgical preference database 132 and / or tool / implant database 134, and provide an output. For example, the system may be configured to display video and / or image highlights of surgical procedures in the application and / or apply useful anatomical labels, as shown. In some cases, the system may display a live video stream of surgical procedures in the application, with automatic labeling on the live stream.

[0023]

[0031] As shown in Figure 2, the system and / or method may display the current and / or anticipated next steps in the procedure, e.g., suturing, suture delivery. The system and / or method may then identify which tool 201 to provide (e.g., “FirstPass Mini” suture passer). In Figure 2, the output may include an image of the procedure (or a similar procedure) and a note (“passing note” 202) regarding physician and / or facility preferences concerning considerations for the scrub technician preparing the tool on the back table, the note including prompts to remind the scrub technician how to prepare the tool and / or what to ask the surgeon to help prepare the tool.

[0024]

[0032] Figure 3 shows another example, illustrating the tools 301 and memo 302 to be prepared during an identified stage of a procedure (preparation for femoral drilling). The system may also detect a procedure (e.g., ACL reconstruction), identify the stage, and determine which tools the surgeon will need based on input from the surgical context recognition module 101 and / or a database.

[0025]

[0033] These systems may also use input from, for example, a backtable camera and a tool recognition module, as described above, to verify that the appropriate tool is prepared. In some cases, the system may provide further information and / or details to recognize the tool being offered, as shown in Figure 4. In this example, the user (e.g., a scrub technician) may request further details about the instrument being offered (e.g., an arthroscopic knot cutter) as part of the procedure. Enhanced options

[0034] As described above, the BTIP module may optionally include feedback to support staff (e.g., scrub technicians) that the prepared tools are correct and / or properly prepared. For example, in some cases, the BTIP module may process a second video stream (e.g., a back table video stream) to provide the scrub technicians with an additional level of guidance. Such an arrangement is shown in the diagram in Figure 1. The tool recognition module may be specifically trained to recognize static tools against a properly neutral background of the back table and may be used to process a video camera corresponding to back table video observing the back table. The main output of the BTIP module may be analyzed on the back table in the context of the tools recognized by the BTIP. More precise and visual feedback can thus be provided to the scrub technicians.

[0026]

[0035] As described herein, the various modules described herein may include one or more machine learning agents for analyzing surgical video input 122 and / or backtable video input in real time. For example, the real-time SCR module 101, the backtable instruction processor module 130, the tool recognition modules 110, 110', the pathology recognition module 116, the anatomical structure recognition module 108, and / or the anatomical structure region recognition module 114 may use one or more trained machine learning agents. In some cases, a single trained machine learning agent may be used for multiple modules. These trained machine learning agents may be of different types or similar, and may be trained on the same training data or different training data.

[0027]

[0036] Any suitable type of machine learning, including but not limited to supervised machine learning, unsupervised machine learning, semi-supervised machine learning, and reinforcement learning, may be used with the methods and apparatus described herein. In supervised learning, a machine learning agent (e.g., a model) may be trained on a labeled dataset. The labeled dataset has both input and output parameters. In supervised learning, an algorithm learns to map points between inputs and correct outputs. The algorithm has both a labeled training dataset and a validation dataset. For example, annotated video footage of a surgical procedure, including patient clinical data, in which all of the tools, anatomical structures, and parts of pathology are shown, may be used to train a machine learning model for one or more of the modules described herein (e.g., tool recognition 110, anatomical structure recognition 112, pathology recognition 116, anatomical structure region recognition 114, surgical procedure recognition 104, surgical activity recognition 106, etc.).

[0028]

[0037] A trained machine learning agent may be an artificial intelligence agent. A machine learning agent may be a deep learning agent. In some examples, a trained pattern matching agent may be a trained neural network. Any suitable type of neural network may be used, including generative neural networks. The neural network may be one or more of the following: perceptron, feedforward neural network, multilayer perceptron, convolutional neural network, radial basis function neural network, recurrent neural network, long short-term memory (LSTM), sequence-to-sequence model, modular neural network, etc.

[0029]

[0038] In some cases, the machine learning agent may be supervised learning and may include building image identifiers to distinguish between various surgical tools. A thetaset of various surgical tools may be used to train the machine learning agent to identify and / or classify the surgical tools from these labeled images (videos). Supervised learning categories may include classification and regression. Classification deals with predicting a categorical target variable that indicates a discrete class or label. Classification algorithms learn to map input features to one of a predefined class and may include one or more such as logistic regression, support vector machine, random forest, decision tree, K-Nearest Neighbors (KNN), and naive Bayes. Regression deals with predicting a continuous target variable that indicates a numerical value. Regression may include linear regression, polynomial regression, ridge regression, lasso regression, decision tree, and random forest regression.

[0030]

[0039] Unsupervised machine learning is a type of machine learning technique in which a machine learning agent discovers patterns and relationships using unlabeled data. Unlike supervised learning, unsupervised learning does not involve providing a labeled target output to the algorithm. The primary goal of unsupervised learning is often to discover hidden patterns, similarities, or clusters within the data, which can then be used for a variety of purposes, such as data exploration, visualization, dimensionality reduction, and much more. There are two main categories of unsupervised learning: clustering and association. Clustering is the process of grouping data points into clusters based on their similarity. This technique is useful for identifying patterns and relationships within data without the need for labeled examples. Examples of clustering algorithms that may be used may include the K-means clustering algorithm, the mean-shift algorithm, the DBSCAN algorithm, principal component analysis, and independent component analysis. Association rule learning is a technique for discovering relationships between items within a dataset. Association rule learning may identify rules that indicate the presence of one item suggests the presence of another item with a certain probability. Specific types of association rule learning algorithms may include the A priori algorithm, Eclat, and FP-growth algorithms.

[0031]

[0040] Any machine learning agent described herein may, similarly or alternatively, be a semi-supervised learning agent. Semi-supervised learning is a machine learning technique that works between supervised and unsupervised learning and may use both labeled and unlabeled data. Semi-supervised learning is particularly useful when obtaining labeled data is expensive, time-consuming, or resource-intensive. This approach is useful when datasets are expensive and time-consuming. Semi-supervised learning is chosen when labeled data requires skill and associated resources to train on it or learn from it. In some cases, methods and apparatus may use semi-supervised learning, for example, when an image (video) training dataset is not fully labeled. There are several different semi-supervised learning methods, including graph-based semi-supervised learning, label propagation, co-training, self-training, and generative adversarial networks (GANs).

[0032]

[0041] Any machine learning agent described herein may, similarly or alternatively, be configured for reinforcement machine learning. Reinforcement machine learning algorithms are learning methods that interact with the environment by generating actions and discovering errors. Examples of reinforcement learning techniques that may be used include Q-learning, SARSA (State-Action-Reward-State-Action), and deep Q-learning.

[0033]

[0042] Any of these methods and apparatus may include a processor. A processor includes hardware that executes computer program code. In particular, the term “processor” may include controllers and may also include dedicated circuits such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), signal processing devices, and other devices, as well as computers having different architectures such as single / multiple processor architectures and sequential (von Neumann) / parallel architectures.

[0034]

[0043] Modules of the systems and methods described herein may include one or more engines and datastores. A computer system may be implemented as an engine, as part of an engine, or through multiple engines. As used herein, an engine includes one or more processors or a portion of one or more processors. A portion of one or more processors may include a portion of hardware smaller than all of the hardware comprising any given one or more processors, such as a subset of registers, a portion of a processor dedicated to one or more threads of a multithreaded processor, a time slice to which the processor is dedicated, all or partly, during which time to performing a portion of the engine's functionality, or the like. Thus, a first engine and a second engine may have one or more dedicated processors, or a first engine and a second engine may share one or more processors with one or more other engines. Depending on the implementation-specific or other considerations, an engine may be centralized or its functionality may be distributed. An engine may include hardware, firmware, or software embodied in a computer-readable medium for execution by a processor. The processor transforms data into new data using the implemented data structures and methods, as described with reference to the figures in this specification.

[0035]

[0044] The engines described herein, or the engines through which the systems and devices described herein may be implemented, may be cloud-based engines. As described herein, a cloud-based engine is an engine that can run applications and / or functions using a cloud-based computing system. All or a given portion of an application and / or function may be distributed across multiple computing devices and do not need to be limited to a single computing device. In some embodiments, a cloud-based engine can run functions and / or modules that are accessed by a user through a web browser or container application, without having functions and / or modules locally installed on the end user's computing device.

[0036]

[0045] As used herein, a datastore is intended to include a repository having any applicable data integration, including tables, comma-separated value (CSV) files, conventional databases (e.g., SQL), or other applicable, known, or convenient integration formats. A datastore may be implemented, for example, in firmware, hardware, a combination thereof, or as software embodied in a physical computer-readable medium on a purpose-specific machine on an applicable, known, or convenient device or system. Datastore-related components, such as database interfaces, may be considered "part of" the datastore, part of some other system component, or a combination thereof, but the physical location and other characteristics of datastore-related components are not important to understanding the techniques described herein.

[0037]

[0046] A data store may include data structures. As used herein, a data structure relates to a specific method of storing and integrating data within a computer so that the data can be used efficiently within a given context. A data structure is generally based on a computer's ability to fetch and store data at any location in memory, specified by an address, a bit string, which is stored in memory itself and can be manipulated by a program. Thus, some data structures are based on calculating the addresses of data items by arithmetic operations, while others are based on storing the addresses of data items within the structure itself. Many data structures use both principles, sometimes in a non-trivial way. Implementations of data structures typically involve writing a set of procedures for creating and manipulating instances of the structure. The data stores described herein may be cloud-based data stores. A cloud-based data store is a data store that is compatible with cloud-based data computing systems and engines.

[0038]

[0047] All publications and patent applications described herein are incorporated by reference to the same extent as if each individual publication or patent application were specifically and individually indicated. Furthermore, it should be recognized that all combinations of the above concepts and further concepts discussed in more detail below (where such concepts are not contradictory) are intended to be part of the subject matter of the invention disclosed herein and may be used to achieve the benefits described herein.

[0039]

[0048] Any method of the methods described herein (including user interfaces) may be implemented as software, hardware, or firmware, and may be described as a non-temporary computer-readable storage medium storing a set of instructions that can be executed by a processor (e.g., a computer, tablet, smartphone, etc.), the set of instructions causing the processor to perform any step of the steps, the steps including, but not limited to, displaying, communicating with a user, analyzing, modifying parameters (including timing, frequency, intensity, etc.), determining, warning, or similar. For example, any method of the methods described herein may be at least partially implemented by an apparatus including one or more processors having memory storing a non-temporary computer-readable storage medium storing a set of instructions for the process of the method.

[0040]

[0049] While various embodiments have been described and / or shown herein in the context of a fully functional computing system, one or more of these exemplary embodiments may be delivered as a program product in various forms, regardless of the specific type of computer-readable medium used to actually carry out the delivery. Embodiments disclosed herein may also be implemented using software modules that perform a particular task. These software modules may include scripts, batches, or other executable files that may be stored on a computer-readable storage medium or within the computing system. In some embodiments, these software modules may be configured in the computing system to carry out one or more of the exemplary embodiments disclosed herein.

[0041]

[0050] As described herein, the computing devices and systems described and / or shown herein broadly represent any type or form of computing device or system capable of executing computer-readable instructions, such as those contained within the modules described herein. In their most basic configurations, each of these computing devices may comprise at least one memory device and at least one physical processor.

[0042]

[0051] As used herein, the terms “memory” or “memory device” generally refer to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, a memory device may store, load, and / or maintain one or more of the modules described herein. Examples of memory devices include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drives (HDD), solid-state drives (SSD), optical disk drives, caches, one or more variations or combinations thereof, or any other suitable storage memory.

[0043]

[0052] Furthermore, as used herein, the terms “processor” or “physical processor” generally refer to any type or form of hardware-implemented processing unit capable of interpreting and / or executing computer-readable instructions. In one example, a physical processor may access and / or modify one or more modules stored in the memory devices described above. Examples of physical processors include, but are not limited to, microprocessors, microcontrollers, central processing units (CPUs), field-programmable gate arrays (FPGAs) implementing soft-core processors, application-specific circuits (ASICs), one or more predetermined parts thereof, one or more variations or combinations thereof, or any other suitable physical processor.

[0044]

[0053] Although shown as separate elements, the method steps described and / or shown herein may represent a predetermined part of a single application. Furthermore, in some embodiments, one or more of these steps may represent or correspond to one or more software applications or programs that, when performed by a computing device, cause the computing device to perform one or more tasks, such as the method steps.

[0045]

[0054] Furthermore, one or more of the devices described herein may transform data, physical devices, and / or representations of physical devices from one form to another. Additionally or alternatively, one or more of the modules listed herein may transform processors, volatile memory, non-volatile memory, and / or any other parts of a physical computing device from one form of a computing device to another form of a computing device by performing on the computing device, storing data in the computing device, and / or otherwise interacting with the computing device.

[0046]

[0055] As used herein, the term “computer-readable medium” generally refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable mediums include, but are not limited to, transmission-type media such as carrier waves, and non-transient-type media such as magnetic storage media (e.g., hard disk drives, tape drives, and floppy disks), optical storage media (e.g., compact discs (CDs), digital video discs (DVDs), and Blu-ray discs), electronic storage media (e.g., solid-state drives and flash media), and other distribution systems.

[0047]

[0056] Those skilled in the art will recognize that any process or method disclosed herein can be modified in many ways. The process parameters or sequences of steps described and / or shown herein are given merely as examples and can be changed as desired. For example, the steps shown and / or described herein may be shown or discussed in a particular order, but these steps do not necessarily have to be performed in the order shown or discussed.

[0048]

[0057] Various exemplary methods described and / or shown herein may also omit one or more of the steps described or shown herein, or include additional steps in addition to the disclosed steps. Furthermore, any step of any method disclosed herein may be combined with any one or more steps of any other method disclosed herein.

[0049]

[0058] A processor described herein may be configured to perform one or more steps of any of the methods disclosed herein. Alternatively or in combination, a processor may be configured to combine one or more steps of one or more of the methods disclosed herein.

[0050]

[0059] When a feature or element is referred to herein as being "on" another feature or element, there may be other features and / or elements that are directly on or interposing to the other feature or element. In contrast, when a feature or element is referred to as being "directly on" another feature or element, there are no interposing features or elements. When a feature or element is referred to as being "connected," "attached," or "coupled" to another feature or element, it will also be understood that there may be other features or elements that can be directly connected, attached, or coupled to the other feature or element. In contrast, when a feature or element is referred to as being "directly connected," "directly attached," or "directly coupled" to another feature or element, there are no interposing features or elements. While a feature or element described or shown in reference to one embodiment may be applicable to other embodiments, it will also be recognized by those skilled in the art that a reference to a structure or feature positioned "adjacent" to another feature may have a predetermined portion that overlaps with or lies beneath the adjacent feature.

[0051]

[0060] The terms used herein are merely for the purpose of describing specific embodiments and are not intended to be limitations of the invention. For example, as used herein, the singular forms “a,” “an,” and “the” are intended to also include the plural form unless the context explicitly indicates otherwise. When the terms “comprises” and / or “comprising” are used herein, they specify the presence of the described features, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, actions, elements, components, and / or groups thereof. As used herein, the terms “and / or” include any and all combinations of one or more of the related listed items and may be abbreviated as “ / ”.

[0052]

[0061] Spatially relative terms such as "under," "below," "lower," "over," "upper," and similar may be used herein for ease of explanation to describe the relationship of one element or feature to another, as shown in the figure. It will be understood that spatially relative terms are intended to encompass different orientations of the device during use or operation, in addition to the orientation shown in the figure. For example, if the device in the figure is inverted, an element described as being "under" or "beneath" another element or feature will be oriented "over" the other element or feature. Thus, the exemplary term "under" can encompass both orientations of "over" and "below." The device may be oriented in other ways (rotated by 90 degrees or other orientations), and the spatially relative descriptors used herein shall be interpreted accordingly herein. Similarly, the terms “upwardly,” “downwardly,” “vertical,” “horizontal,” and similar terms are used herein solely for illustrative purposes unless otherwise specified.

[0053]

[0062] The terms “first” and “second” may be used herein to describe various features / elements (including steps), but these features / elements should not be limited by these terms unless the context indicates otherwise. These terms may be used solely to distinguish one feature / element from another. Thus, without departing from the teachings of the invention, the first feature / element discussed below may be referred to as the second feature / element, and similarly, the second feature / element discussed below may be referred to as the first feature / element.

[0054]

[0063] In general, any apparatus and method described herein should be understood as inclusive, although all or a subset of components and / or steps may be alternatively exclusive, and may be expressed as "consisting of" or alternatively "consisting essentially of" various components, steps, subcomponents, or substeps.

[0055]

[0064] As used in the examples and as used in this specification and in the claims, including, unless otherwise expressly specified, all numbers may be read as if they began with the word “about” or “approximately,” even if the term is not expressly present. The phrase “about” or “approximately” may be used when describing a magnitude and / or location to indicate that the value and / or location described is within a reasonable expected range of the value and / or location. For example, a number may have a value that is + / -0.1% of the stated value (or range of value), + / -1% of the stated value (or range of value), + / -2% of the stated value (or range of value), + / -5% of the stated value (or range of value), + / -10% of the stated value (or range of value), and so on. Any number given herein should also be understood to include about or approximately that value unless the context indicates otherwise. For example, if the value “10” is disclosed, “about 10” is also disclosed. Any numerical ranges cited herein are intended to include all subranges contained therein. When a value is disclosed, it is understood, as those skilled in the art will understand, that "less than or equal to the value," "greater than or equal to the value," and possible ranges between the values ​​are also disclosed. For example, if the value "X" is disclosed, then "less than or equal to X" and "greater than or equal to X" (for example, X is a number) are also disclosed. Throughout the application, it is understood that data is provided in multiple different formats, and that this data indicates ranges for any combination of endpoints, start points, and data points. For example, if a particular data point "10" and a particular data point "15" are disclosed, it is understood that for 10 and 15, greater than, greater than or equal to, less than, less than or equal to, and between 10 and 15 are disclosed. It is also understood that each unit between two particular units is also disclosed.For example, if 10 and 15 are disclosed, then 11, 12, 13, and 14 are also disclosed.

[0056]

[0065] Although various illustrative embodiments have been described above, any modification of any of the modifications may be made to various embodiments without departing from the scope of the invention as described by the claims. Optional features of various device and system embodiments may be included in some embodiments and not in others. Therefore, the above description is provided primarily for illustrative purposes and should not be construed as limiting the scope of the invention, as the scope of the invention is stated in the claims.

[0057]

[0066] The examples and illustrations included herein illustrate, not limit, specific embodiments in which the subject matter may be implemented. As stated, other embodiments may be utilized and derived from specific embodiments, thereby allowing structural and logical substitutions and modifications without departing from the scope of this disclosure. If two or more such embodiments of the subject matter of the present invention are actually disclosed, they may be referred to individually or collectively herein by the term “invention” simply for convenience and without the intention of voluntarily limiting the scope of this application to any single invention or concept of invention. Thus, any arrangement planned to achieve the same objective as a specific embodiment shown and described herein may replace the specific embodiment shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments. Combinations of the above embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art upon consideration of the above description.

Claims

1. A method for providing surgical guidance to a scrub technician during a surgical procedure, A step of using a real-time surgical context recognition module to identify one or more surgical procedures performed on a patient in a sterile field, wherein the real-time surgical context recognition module receives one or more video streams of the surgical procedures being performed and one or more video streams of the back table within the sterile field. A step of determining a sequence of surgical tools that will be required to perform the identified one or more surgical procedures using a backtable instruction processor that includes a trained machine learning agent, A step of outputting each of the surgical tools in the sequence to a monitor that can be viewed within the sterile field, wherein the surgical tools are presented sequentially for placement on the back table within the sterile field. Methods that include...

2. The method according to claim 1, wherein the one or more steps include two or more steps that share the same back table in the sterile field.

3. The method according to claim 1, wherein the one or more steps include a single step.

4. The method according to claim 1, wherein the output step includes sequentially outputting the surgical tools at a timely interval during the course of one or more surgical procedures.

5. The method according to claim 1, wherein the trained machine learning agent is trained to identify the sequence of surgical tools based on physician preferences.

6. The method according to claim 1, wherein the step of identifying one or more surgical procedures to be performed includes using a second trained machine learning agent.

7. The method according to claim 1, wherein the steps of identifying and determining are performed in real time.

8. The method according to claim 1, wherein the step of identifying using the real-time surgical context recognition module includes identifying one or more tools already used in the surgical procedure.

9. The method according to claim 8, wherein the step of determining using the backtable instruction processor includes identifying one or more tools present on the backtable.

10. The method according to any one of claims 8 and 9, further comprising using a tool recognition module.

11. The method according to claim 1, wherein the step of identifying using the real-time surgical context recognition module includes receiving patient clinical data in addition to the one or more video streams of the back table and the one or more video streams of the surgical procedure in the sterile field.

12. A method for providing surgical guidance to a scrub technician during a surgical procedure, A step of using a real-time surgical context recognition module to identify one or more surgical procedures performed on a patient in a sterile field, wherein the real-time surgical context recognition module receives one or more video streams of the surgical procedures being performed and one or more video streams of the back table within the sterile field. A step of determining a sequence of surgical tools that will be required to perform the identified one or more surgical procedures using a backtable instruction processor module that includes a trained machine learning agent, A step of outputting each of the surgical tools in the sequence to a monitor that can be viewed within the sterile field, wherein the surgical tools are presented sequentially for placement on the back table within the sterile field. The steps include receiving input from a back table camera that observes the back table, The backtable instruction processor module verifies that the surgical tool in the sequence has been placed on the backtable, Methods that include...

13. One or more processors, The computer implementation method comprises a memory coupled to one or more processors, the memory storing computer program instructions, and when the computer program instructions are executed by one or more processors, the computer implementation method implements a computer implementation method, and the computer implementation method Identifying one or more surgical procedures performed on a patient in a sterile field using a real-time surgical context recognition module, wherein the real-time surgical context recognition module receives and identifies one or more video streams of the surgical procedures being performed and one or more video streams of the back table within the sterile field. Using a backtable instruction processor that includes a trained machine learning agent, determine the sequence of surgical tools that will be required to perform the identified one or more surgical procedures. Outputting each of the surgical tools in the sequence to a monitor visible within the sterile field, wherein the surgical tools are output to be sequentially presented for placement on the back table within the sterile field. A system that includes this.

14. The system according to claim 13, wherein output includes sequentially outputting the surgical tools at a timely interval during the course of one or more surgical procedures.

15. The system according to claim 13, wherein the trained machine learning agent is trained to identify the sequence of surgical tools based on physician preferences.

16. The system according to claim 13, wherein identifying the one or more surgical procedures to be performed includes using a second trained machine learning agent.

17. The system according to claim 13, wherein identification and determination are performed in real time.

18. The system according to claim 13, wherein identification using the real-time surgical context recognition module includes identifying one or more tools already used in the surgical procedure.

19. The system according to claim 18, wherein determining using the backtable instruction processor includes identifying one or more tools present on the backtable.

20. The system according to claim 19, further comprising using a tool recognition module.

21. The system according to claim 13, wherein identification using the real-time surgical context recognition module includes receiving patient clinical data in addition to the one or more video streams of the back table and the one or more video streams of the surgical procedure in the sterile field.

Images