Systems and methods for ai-assisted surgery

Abstract

To provide, in various embodiments, systems and methods to assist or guide an arthroscopic surgery or other surgical procedure e.g., surgery of the shoulder, knee or hip.SOLUTION: The method comprises steps of receiving an image from an interventional imaging device, identifying a feature in the image using an image recognition algorithm, overlaying the features on a video feed on a display device, and making recommendations or suggestions to an operator based on the identified feature in the image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This PCT application claims priority to Indian Provisional Patent Application No. 20204105990, filed April 13, 2020, and U.S. Provisional Application Nos. 63 / 030,695, filed May 27, 2020, and 63 / 143,367, filed January 29, 2021, the entire contents of which are hereby incorporated by reference in their entirety for all purposes.

[0002]

[0002] Embodiments of the present invention relate to systems, devices and methods for guiding diagnostic and surgical procedures, particularly using artificial intelligence (AI). [Background technology]

[0003]

[0003] In recent years, artificial intelligence has begun to be developed for use in processing images to recognize features of various anatomical structures in the human body, as well as human faces. These AI tools can be used to automatically recognize anatomical features to assist operators during medical procedures. Computational methods such as machine learning and deep learning algorithms can be used for image or language processing to collect and process information generated during medical procedures. Therefore, it is desirable to use AI algorithms that can be used to predict or improve surgical outcomes or to guide the education of new physicians through virtual or educational surgeries. Current AI-assisted surgical systems and methods, for example, are still less than ideal in many respects when used to guide surgical procedures. Therefore, improvements to AI-assisted surgical systems and methods are desirable. Summary of the Invention [Problem to be solved by the invention]

[0004] Various embodiments of the present invention provide computer-implemented medical systems, devices, and methods for guiding surgical or other medical procedures. Numerous embodiments are described herein that provide guidance to an operator and accomplish this by identifying and labeling one or more anatomical features, lesions, and other features in the surgical field in real time. Operator errors during the course of a surgical procedure can be costly. For example, it may be difficult or impossible for an operator to determine the exact location of a critical anatomical feature that is hidden from a camera (e.g., a camera used during arthroscopic or endoscopic surgery), or a portion of the lesion being removed may be outside the operator's or camera's field of view. Therefore, computer-implemented medical systems, devices, and methods, such as artificial intelligence (AI) tools, can be beneficial, particularly for guiding medical procedures. These AI tools may have limitations in accurately and reliably predicting the detection of instruments, anatomical structures, or procedures. In fast-paced surgical procedures, AI tools also need to perform predictions with low latency to provide real-time assistance to the operator. [Means for solving the problem]

[0005]

[0005] There is recognized herein a need for fast, accurate, and robust AI tools that assist operators in real time during the surgical procedure to improve surgical outcomes. Accordingly, aspects of the present invention provide a pipeline of machine learning algorithms that are versatile and well-trained for the unique needs of various medical procedures, including various minimally invasive procedures such as arthroscopic, endoscopic, laparoscopic, and cardioscopic procedures. Examples of such minimally invasive procedures may include one or more of arthroscopic procedures (e.g., repair of a torn rotator cuff in the shoulder, cruciate ligament surgery in the knee, repair of various non-arthritic hip disorders, repair of cartilage damage in the ankle, or removal of bone spurs in the ankle), gastrointestinal (GI) surgery (e.g., intestinal biopsy, polypectomy, bariatric surgery, gastric reduction surgery / vertical band gastroplasty), urological procedures (e.g., kidney stone removal, bladder repair), gynecological surgery (e.g., DNC, uterine myomectomy), and laparoscopic surgery (e.g., appendectomy, cholecystectomy, colectomy, hernia repair, fundoplication).

[0006] Various embodiments of the present invention provide systems, devices, and methods that can receive information (e.g., images, audio, user input) during a medical procedure (e.g., a surgical procedure), process the received information to identify features associated with the procedure, and provide recommendations based on the identified features. These features may include anatomical sites or devices, or various steps or results of the procedure. Based on the identification of various features in the procedure, the systems, devices, and methods described herein can assist a surgeon in the procedure by providing recommendations that may include instrument entry angles, measurements of anatomical features or lesions, and actions to take or avoid, to name a few.

[0007] Aspects of the present invention further assist surgeons (and associated medical personnel) during surgery by using images from the surgical field and applying artificial intelligence (AI) to provide guidance and assistance to the surgeon and other medical staff. The AI ​​modules / algorithms used during surgery are referred to as surgical AI.

[0008] In a first aspect, the present invention provides a system for guiding an arthroscopic procedure. In some embodiments, the system includes one or more computer processors and one or more non-transitory computer-readable storage media storing instructions operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations, including receiving at least one image captured by an interventional imaging device, identifying one or more image features in the received at least one image using an image recognition algorithm, labeling the identified one or more image features, and displaying the labeled one or more image features in the at least one image to an operator continuously during the course of the arthroscopic procedure. In some embodiments, the identified one or more image features include one or more of an anatomical structure, a surgical instrument, a surgical instrument element, a procedure or action, or a lesion such as torn or injured tissue. Application of system embodiments to the guidance of other medical procedures, including endoscopic procedures, laparoscopic procedures, and minimally invasive procedures such as interventional cardiovascular surgery, is also contemplated.

[0009] In various embodiments, the labeled image feature or features may be displayed in real time or concurrently with the arthroscopic procedure. In some embodiments, the arthroscopic procedure is arthroscopic surgery. In some embodiments, the image recognition algorithm includes a hierarchical organization of processing modules, also referred to as software modules or modules.

[0010] In some embodiments, the processing module includes a plurality of artificial intelligence (AI) modules, which in various embodiments may correspond to machine learning algorithms, deep learning algorithms, or a combination of both. In some embodiments, the machine learning algorithm includes an artificial neural network. In various embodiments, the processing module also includes at least one dataset, which may include at least one training dataset.

[0011] In various embodiments, the processing module includes an upstream module and a downstream module, the downstream module being more specialized than the upstream module. In some embodiments, the upstream module is configured to identify one or more of the anatomical structures manipulated by a procedure or action being performed during the arthroscopic procedure. In some embodiments, the downstream module is configured to recognize one or more of the anatomical features of the identified anatomical structures or treatment instrument features associated with the procedure or action being performed. In some embodiments, the processing module includes a plurality of upstream modules and a plurality of downstream modules. In some embodiments, at least one of the modules of the plurality of upstream modules is configured to select an individual downstream module from among the processing modules for use.

[0012] In some embodiments, the operational steps or actions are identified by one or more of the processing modules based at least in part on the identification of a surgical instrument, e.g., an arthroscope or endoscope, used during the procedure. In some embodiments, the interventional imaging device is an arthroscope. In some embodiments, the interventional imaging device is an endoscope. In some embodiments, the image recognition algorithm is configured to identify one or more of a surgical site or entry point for an arthroscopic or other medical procedure. In various embodiments, the surgical site may correspond to one or more of a shoulder, a knee, or a hip.

[0013] In some embodiments, at least one module from the processing modules is selected based on at least an identification of one or more of the surgical site (e.g., shoulder) or approach (e.g., capsular approach for shoulder, an anterior approach for knee). In some embodiments, the operations performed further include storing the at least one image in a memory device. In some embodiments, the operations further include discarding the at least one image after the step of displaying the sign element to optimize memory usage.

[0014] In various embodiments, the one or more labeled image features further include pixel-wise masked labeling, bounding box labeling, frame-wise labeling, or temporal labeling, which may be configured to be used for various purposes. For example, in some embodiments, pixel-wise masked labeling may be used to display labeled anatomical structures or surgical instruments. In one or more embodiments, bounding box labeling may further be used to display one or more labeled lesions, surgical instruments, or foreign bodies. Furthermore, according to one or more embodiments, frame-wise labeling may be used to display labeled anatomical sites, such as labeled shoulders, and temporal labeling may be used to display labeled operative procedures or actions (e.g., surgical actions such as tissue resection, ablation, or suturing).

[0015] In various embodiments, the operations further include providing a suggested action, which may be based on a number of factors, including results or other output from various modules. For example, in one or more embodiments, the suggested action may be based at least in part on one or more of the following: labeling of at least one image; results or other output from an upstream module; or results or other output from a downstream module. Also, in various embodiments, the suggested action may have a wide variety of purposes depending on the procedure and the situation. For example, the suggested action may be used to assist an operator during the course of a procedure, while also being used for educational or training purposes, such as when a surgeon is learning a new procedure and / or during a surgical simulation. In further embodiments, the suggested action may be based on important anatomical structures (e.g., arteries), surgical instruments, Providing a safety warning based at least on an action, a distance between two or more implants (e.g., whether they are above or below is a significant event), or identification of an adverse event (e.g., bleeding within the surgical field) or patient biomedical / physiological data (e.g., decreased blood pO2, blood pressure, respiratory rate, irregular heart rhythm (e.g., arrhythmia or other significant physiological parameter of the patient)).

[0016] Various embodiments of the present invention may also provide suggested surgical actions based on one or more of the dimensions, placement, and configuration of anatomical structures and structures, including implants, within a surgical field. For example, in some surgical situations in which embodiments of the present invention are used, the distance between two or more implants may pose a health risk. In particular, the actual or allowable distance between two or more placed implants in the surgical field may differ from a predefined distance between the two or more implants, which may make approach to the surgical field, including the approach for placing the implants, difficult and / or difficult to approach with the human eye, especially when performed on a screen in a two-dimensional view. Accordingly, in the above and related embodiments, the suggested action may include providing a suggested approach angle, in which a drill angle may be suggested for implant placement or other surgical action.

[0017] Various embodiments of the system may be configured for multiple joint surgeries, including, for example, one or more of shoulder surgery, knee surgery, hip surgery, ankle surgery, hand surgery, or elbow surgery. In these and related embodiments, one or more of the processing modules described herein include having data related to each particular site, which may be tailored to each particular site and may include a specialized training data set.

[0018] In some embodiments, the image recognition algorithm is trained using a database. In some embodiments, the database includes a plurality of training images including one or more surgical procedures, surgical instruments, surgical instrument elements, anatomical structures, or lesions. In some embodiments, augmentation methods are used to refine the training dataset to improve the robustness of the image recognition algorithm. The surgical images used for such training, including augmented training, may be selected from a wide variety of procedures, including one or more minimally invasive procedures such as arthroscopic, endoscopic, laparoscopic, and cardioscopic procedures, which may correspond to one or more of arthroscopic, bariatric, cardiovascular, bowel, gynecological, urological, or related procedures.

[0019] In some embodiments, the augmentation method includes rotating the training images to improve robustness to patient position or orientation during the arthroscopic procedure. In some embodiments, the augmentation method includes flipping the training images along a vertical axis to improve robustness to procedures performed on the patient's right or left side. In some embodiments, the augmentation method includes magnifying or cropping the training images to improve robustness to changes in depth of field.

[0020] In some embodiments, at least one image is generated from a surgical video stream. In some embodiments, the surgical video stream is an arthroscopic surgery video stream. In various embodiments, the surgical video stream may be monocular or stereoscopic. In these and related embodiments, system embodiments may be adapted to receive and recognize each type of view (monocular and stereoscopic) and switch back and forth between appropriate processing methods as the view is switched.

[0021] Another aspect of the present invention provides a computer-implemented method for guiding an arthroscopic procedure. In some embodiments, the method includes receiving at least one image captured by an interventional imaging device and using an image recognition algorithm to: The method includes identifying one or more image features in at least one received image, labeling the identified one or more image features, and displaying the labeled one or more image features in the at least one image to an operator continuously during the course of the arthroscopic procedure. In some embodiments, the identified one or more image features include one or more of an anatomical structure, a surgical instrument, a surgical instrument element, a procedure or action, or a lesion. In some embodiments, the labeled one or more image features are displayed in real time or concurrently with the arthroscopic procedure.

[0022] In some embodiments, the arthroscopic procedure is arthroscopic surgery. In some embodiments, the image recognition algorithm comprises a hierarchical organization of processing modules. In some embodiments, the processing module comprises a plurality of artificial intelligence (AI) modules. In some embodiments, the processing module comprises at least a machine learning algorithm, a deep learning algorithm, or a combination of both. In some embodiments, the machine learning algorithm comprises an artificial neural network.

[0023] In some embodiments, the processing module includes at least one dataset, which may include at least one training dataset. In some embodiments, the processing module includes an upstream module and a downstream module, the downstream module being more specialized than the upstream module. In some embodiments, the upstream module is configured to identify one or more of the anatomical structures manipulated by a procedure or action being performed during the arthroscopic procedure. In some embodiments, the downstream module is configured to do one or more of: recognize an anatomical feature of the identified anatomical structure; or recognize a feature of a treatment instrument associated with the procedure or action being performed. In some embodiments, the processing module includes multiple upstream modules or multiple downstream modules. In some embodiments, at least one of the modules of the multiple upstream modules is configured to select an individual downstream module from among the processing modules for use. In some embodiments, the procedure or action is identified based in part on identifying the surgical instrument.

[0024] In some embodiments, the interventional imaging device is an arthroscope. In some embodiments, the interventional imaging device is an endoscope. In some embodiments, the image recognition algorithm is configured to identify one or more of a surgical site or an approach for the arthroscopic procedure. In some embodiments, the surgical site is a shoulder. In some embodiments, the surgical site is a knee. In some embodiments, at least one of the processing modules is selected based at least on identifying one or more of the surgical site or an approach.

[0025] In some embodiments, the operations performed further include storing the at least one image in a memory device. In some embodiments, the operations further include discarding the at least one image after the step of displaying the sign element to optimize memory usage.

[0026] In some embodiments, the labeled one or more image features further include pixel-wise masked labeling, bounding box labeling, frame-wise labeling, or temporal labeling. In some embodiments, pixel-wise masked labeling is used to display labeled anatomical structures or surgical instruments. In some embodiments, bounding box labeling is used to display labeled lesions, surgical instruments, or foreign bodies. In some embodiments, frame-wise labeling is used to display labeled anatomical sites. In some embodiments In, temporal labeling is used to display steps that label an operating procedure or action.

[0027] In some embodiments, the operations further include providing a suggested action. In some embodiments, the suggested action is based at least in part on labeling of the at least one image. In some embodiments, the suggested action is based at least in part on an upstream module. In some embodiments, the suggested action is based at least in part on a downstream module. In some embodiments, the suggested action is to assist the operator during the course of a surgery. In some embodiments, the suggested action is provided for educational purposes. In some embodiments, the suggested action includes providing a safety warning based at least on identification of critical anatomical structures or distances between two or more implants.

[0028] In some embodiments, the distance between the two or more implants poses a health risk. In some embodiments, the distance between the two or more implants is different from a predefined distance between the two or more implants. In some embodiments, the suggested action includes providing a suggested approach angle. In some embodiments, the suggested approach angle may include a drilling angle.

[0029] In some embodiments, the method is configured for shoulder surgery. In some embodiments, the method is configured for knee surgery. In some embodiments, the image recognition algorithm is trained using a database. In some embodiments, the database includes a plurality of training images including one or more surgical procedures, surgical instruments, surgical instrument elements, anatomical structures, or lesions.

[0030] In some embodiments, multiple augmentation methods are used to refine the training dataset to improve the robustness of the image recognition algorithm. In some embodiments, the augmentation methods include rotating the training images to improve robustness to patient position or orientation during the arthroscopic procedure. In some embodiments, the augmentation methods include flipping the training images along a vertical axis to improve robustness to procedures performed on the patient's right or left side. In some embodiments, the augmentation methods include enlarging or cropping the training images to improve robustness to changes in depth of field.

[0031] In some embodiments, the at least one image is generated from a surgical video stream. In some embodiments, the surgical video stream is an arthroscopic surgery video stream. In some embodiments, the surgical video stream is monocular. In some embodiments, the surgical video stream is stereoscopic.

[0032] Another aspect of the present invention provides a method for training an algorithm for guiding an arthroscopic procedure. In some embodiments, the method includes receiving a set of image features based on one or more images related to the arthroscopic procedure, receiving a training dataset, recognizing one or more of the image features in the images of the training dataset, and constructing an image recognition algorithm based at least in part on the one or more image features and the received training dataset. In some embodiments, the training dataset includes one or more labeled images related to the arthroscopic procedure. In some embodiments, the training dataset includes one or more labeled images related to the arthroscopic procedure, wherein the one or more image features relate to one or more visual characteristics of an anatomical structure, a surgical instrument, a surgical instrument element, a procedure or action, or a lesion. In some embodiments, In this case, the image recognition algorithm is configured to identify and label one or more image features in an untagged image related to an arthroscopic procedure.

[0033] In some embodiments, the labeled one or more image features are displayed in real time or concurrently with the arthroscopic procedure. In some embodiments, the image recognition algorithm includes a hierarchical organization of processing modules. In some embodiments, the processing module includes multiple individual image processing modules. In some embodiments, the multiple individual image processing modules include a first module for identifying the arthroscopic procedure at a predetermined location, a second module for recognizing and labeling one or more surgical instruments and surgical instrument elements, a third module for recognizing and labeling one or more anatomical structures, or a combination thereof.

[0034] In some embodiments, the processing module includes multiple artificial intelligence (AI) modules. In some embodiments, the processing module includes at least a machine learning algorithm, a deep learning algorithm, or a combination of both. In some embodiments, the machine learning algorithm includes an artificial neural network.

[0035] In some embodiments, the processing module includes at least one data set. In some embodiments, the processing module includes at least one training data set.

[0036] In some embodiments, the processing module includes an upstream module and a downstream module, the downstream module being more specialized than the upstream module. In some embodiments, the upstream module is configured to identify one or more of the anatomical structures manipulated by a procedure or action being performed during the arthroscopic procedure. In some embodiments, the downstream module is configured to do one or more of: recognize anatomical features of the identified anatomical structures; or recognize treatment instrument features associated with the procedure or action being performed. In some embodiments, the processing module includes multiple upstream modules or multiple downstream modules. In some embodiments, at least one of the modules of the multiple upstream modules is configured to select an individual downstream module from among the processing modules for use.

[0037] In some embodiments, identifying one or more image features in at least one image further includes selecting one or more processing modules from a plurality of processing modules, the selection being based at least in part on a site and / or portal of the arthroscopic surgery. In some embodiments, the surgical procedure or action is identified based in part on identifying a surgical instrument. In some embodiments, the unlabeled image is captured by an interventional imaging device. In some embodiments, the interventional imaging device is an endoscope. In some embodiments, the unlabeled image is generated from a surgical video stream.

[0038] In some embodiments, the surgical video stream is an arthroscopic surgery video stream and the endoscopic surgery is arthroscopic surgery. In some embodiments, the surgical video stream is monocular. In some embodiments, the surgical video stream is stereoscopic.

[0039] In some embodiments, the image recognition algorithm is configured to identify one or more of a surgical site or an approach for an arthroscopic procedure. In some embodiments, the surgical site is a shoulder. In some embodiments, the surgical site is a knee. In an embodiment, at least one of the processing modules is selected based at least on identifying one or more of the surgical site or entry points.

[0040] In some embodiments, the image recognition algorithm stores the labeled images in a memory device. In some embodiments, the image recognition algorithm discards the labeled images to minimize memory usage.

[0041] In some embodiments, the labeled images include pixel-wise masked labeling, bounding box labeling, frame-wise labeling, or temporal labeling. In some embodiments, pixel-wise masked labeling is used to display labeled anatomical structures or surgical instruments. In some embodiments, bounding box labeling is used to display labeled lesions, surgical instruments, or foreign bodies. In some embodiments, frame-wise labeling is used to display labeled anatomical sites. In some embodiments, temporal labeling is used to display steps of an operating procedure or labeling an action.

[0042] In some embodiments, the training data set is configured for shoulder surgery. In some embodiments, the image recognition algorithm is trained for shoulder surgery using a training data set configured for shoulder surgery. In some embodiments, the training data set is configured for knee surgery. In some embodiments, the image recognition algorithm is trained for knee surgery using a training data set configured for knee surgery. In some embodiments, the training data set includes a plurality of training images including one or more surgical procedures, surgical instruments, surgical instrument elements, anatomical structures, or lesions.

[0043] In some embodiments, multiple augmentation methods are used to refine the training dataset to improve the robustness of the image recognition algorithm. In some embodiments, the augmentation method includes rotating the training images to improve robustness to patient position or orientation during the arthroscopic procedure. In some embodiments, the augmentation method includes flipping the training images along a vertical axis to improve robustness to procedures performed on the patient's right or left side. In some embodiments, the augmentation method includes enlarging or cropping the training images to improve robustness to changes in depth of field.

[0044] Another aspect of the present invention provides a method for implementing a hierarchical pipeline for guiding arthroscopic surgery. In some embodiments, the system includes one or more computer processors and one or more non-transitory computer-readable storage media storing instructions operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations including: (a) receiving at least one image captured by an interventional imaging device; (b) identifying one or more image features of a site of treatment or an entry point at the site based on at least one upstream module; (c) operating a first downstream module to identify one or more image features of an anatomical structure or lesion based at least in part on the one or more image features identified in step (b); (d) operating a second downstream module to identify one or more image features of a surgical instrument, surgical instrument element, procedure, or action related to the arthroscopic surgery based at least in part on the one or more image features identified in step (b); (e) labeling the identified one or more image features; and (f) displaying the labeled one or more image features in the at least one image to an operator continuously during the course of the arthroscopic surgery.

[0045] In some embodiments, at least one upstream module includes a first trained image processing algorithm. In some embodiments, at least one upstream module includes a first trained image processing algorithm, and the first downstream module includes a second trained image processing algorithm. In some embodiments, at least one upstream module includes a first trained image processing algorithm, the first downstream module includes a second trained image processing algorithm, and the second downstream module includes a third trained image processing algorithm. In some embodiments, steps (c) and (d) are independent of each other. In some embodiments, the first, second, or third trained image processing algorithm includes at least a machine learning algorithm, a deep learning algorithm, or a combination of both. In one or more embodiments, the machine learning algorithm includes an artificial neural network.

[0046] In some embodiments, the machine learning or deep learning algorithm is trained using at least one training data set. In some embodiments, the training data set is configured for shoulder surgery. In some embodiments, the training data set is configured for knee surgery. In some embodiments, the training data set includes a plurality of training images including one or more surgical procedures, surgical instruments, surgical instrument elements, anatomical structures, or lesions.

[0047] In some embodiments, multiple augmentation methods are used to refine the training dataset to improve the robustness of the image recognition algorithm. In some embodiments, the augmentation method includes rotating the training images to improve robustness to patient position or orientation during the arthroscopic procedure. In some embodiments, the augmentation method includes flipping the training images along a vertical axis to improve robustness to procedures performed on the patient's right or left side. In some embodiments, the augmentation method includes enlarging or cropping the training images to improve robustness to changes in depth of field.

[0048] In some embodiments, the first, second, or third trained image processing algorithm stores the displayed image having the labeled feature in a memory device. In some embodiments, the first, second, or third trained image processing algorithm discards the displayed image having the labeled feature to minimize memory usage.

[0049] In some embodiments, the arthroscopic procedure is arthroscopic surgery. In some embodiments, the at least one image is generated from a surgical video stream. In some embodiments, the surgical video stream is an endoscopic surgery video stream. In some embodiments, the surgical video stream is monocular. In some embodiments, the surgical video stream is stereoscopic.

[0050]

[0050] Another aspect of the present invention provides a non-transitory computer-readable medium containing machine-executable code that, when executed by one or more computer processors, performs any of the methods described above or elsewhere in this specification.

[0051] Another aspect of the present invention provides a system comprising one or more computer processors and a computer memory coupled thereto, the computer memory containing machine-executable code that, when executed by the one or more computer processors, performs any of the methods described above or elsewhere herein.

[0052]

[0052] Embodiments of the systems and methods of the present invention provide real-time display of anatomy, lesions, lesion repair, and selected measurements within the surgical field, thereby enabling visualization of related Such displays are particularly useful in assisting surgeons during procedures such as arthroscopic surgery. Such displays reduce errors due to misidentification of tissue structures and measurements, thus leading to improved surgical and patient outcomes. They also do so by reducing the cognitive load on the surgeon, allowing them to focus on the procedure and alert to potential adverse actions, conditions, or consequences during the surgical process, which may be hindered by focusing or overfocusing on important tasks. Such alerts prevent or reduce potential adverse events during surgery by alerting the surgeon before and / or as they occur, allowing the surgeon to take appropriate action to prevent or mitigate them. In use, such alerts lead to improved acute surgical and long-term patient outcomes, along with reduced morbidity and mortality, as surgical actions are performed with no or few errors and with greater precision and accuracy (e.g., accuracy in implant placement or removal of damaged or diseased tissue while preserving healthy tissue).

[0053]

[0053] Further aspects and advantages of the present invention will become readily apparent to those skilled in the art from the following detailed description, in which merely exemplary embodiments of the invention are shown and described. As will be realized, the invention is capable of other and different embodiments, and its several details are capable of modification in various obvious respects, all without departing from the present disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature and not as restrictive. Incorporation by Reference

[0054] All publications, patents, and patent applications mentioned herein are incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent that the publications and patents or patent applications incorporated by reference conflict with the disclosure contained herein, the present specification is intended to supersede and / or take precedence over such conflicting matter.

[0054]

[0055] The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be realized by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings (also referred to herein as "drawings" and "figures"), in which: [Brief explanation of the drawings]

[0055] [Figure 1]

[0056] 1 is a flowchart of an example of a hierarchical organization of modules in a system for AI-assisted surgery, according to some embodiments. [Figure 2]

[0057] FIG. 2A illustrates images of an arthroscopic surgical procedure, according to some embodiments.

[0058] FIG. 2B illustrates an example of marking features in an arthroscopic surgical procedure, according to some embodiments. [Figure 3A]

[0059] 10A-10C illustrate examples of marking instruments, actions, or procedures in images captured by an arthroscope, according to some embodiments. [Figure 3B]

[0060] 10A-10C illustrate other examples of marking instruments, actions, or procedures in images captured by an arthroscope, according to some embodiments. [Figure 3C]

[0061] FIG. 10 illustrates an example of recognizing any instrument in a surgical field, according to some embodiments. [Figure 4]

[0062] 1 is a flowchart of an example of training an image recognition algorithm, according to some embodiments. [Figure 5]

[0063] FIG. 1 illustrates a computer system that is programmed or otherwise configured to perform the methods presented herein, according to some embodiments. [Figure 6]

[0064] FIG. 1 illustrates an example of an AI pipeline used in simulating knee surgery, according to some embodiments. [Figure 7]

[0065] FIG. 7A is a diagram illustrating an example of recognizing and labeling anatomical structures, according to some embodiments.

[0066] FIG. 7B illustrates an example of recognizing and labeling a lesion, according to some embodiments. [Figure 8]

[0067] FIG. 8A illustrates example images of an arthroscopic surgical procedure, according to some embodiments.

[0068] FIG. 8B illustrates an example of an image of an arthroscopic surgical procedure that has been labeled using an image recognition algorithm, according to some embodiments.

[0069] FIG. 8C is a diagram illustrating an example of an anatomical structure in an image labeled by a subject matter expert, according to some embodiments. FIG. 8D is a diagram illustrating an example of an anatomical structure in an image labeled by a subject matter expert, according to some embodiments. FIG. 8E is a diagram illustrating an example of an anatomical structure in an image labeled by a subject matter expert, according to some embodiments. FIG. 8F is a diagram illustrating an example of an anatomical structure in an image labeled by a subject matter expert, according to some embodiments. FIG. 8G is a diagram illustrating an example of an anatomical structure in an image labeled by a subject matter expert, according to some embodiments. FIG. 8H is a diagram illustrating an example of an anatomical structure in an image labeled by a subject matter expert, according to some embodiments. FIG. 8I is a diagram illustrating an example of an anatomical structure in an image labeled by a subject matter expert, according to some embodiments. FIG. 8J is a diagram illustrating an example of an anatomical structure in an image labeled by a subject matter expert, according to some embodiments. FIG. 8K is a diagram illustrating an example of an anatomical structure in an image labeled by a subject matter expert, according to some embodiments. FIG. 8L is a diagram illustrating an example of an anatomical structure in an image labeled by a subject matter expert, according to some embodiments.

[0070] FIG. 8M is a diagram illustrating an example of an anatomical structure in an image labeled by an AI algorithm, according to some embodiments. FIG. 8N is a diagram illustrating an example of an anatomical structure in an image labeled by an AI algorithm, according to some embodiments. FIG. 8O is a diagram illustrating an example of an anatomical structure in an image labeled by an AI algorithm, according to some embodiments. FIG. 8P is a diagram illustrating an example of an anatomical structure in an image labeled by an AI algorithm, according to some embodiments. FIG. 8Q is a diagram illustrating an example of an anatomical structure in an image labeled by an AI algorithm, according to some embodiments. FIG. 8R is a diagram illustrating an example of an anatomical structure in an image labeled by an AI algorithm, according to some embodiments. FIG. 8S is a diagram illustrating an example of an anatomical structure in an image labeled by an AI algorithm, according to some embodiments. FIG. 8T is a diagram illustrating an example of an anatomical structure in an image labeled by an AI algorithm, according to some embodiments. FIG. 8U is a diagram illustrating an example of an anatomical structure in an image labeled by an AI algorithm, according to some embodiments. FIG. 8V is a diagram illustrating an example of an anatomical structure in an image labeled by an AI algorithm, according to some embodiments. [Figure 9]

[0071] FIG. 1 illustrates examples of actions recognized by a system for AI-assisted surgery, according to some embodiments. [Figure 10]

[0072] 1 is an exemplary flowchart of a process for identifying a surgical procedure, according to some embodiments. [Figure 11A]

[0073] FIG. 10 illustrates another example of a process for identifying a surgical procedure, according to some embodiments. [Figure 11B]

[0074] 10A-10C illustrate examples of outputs from an instrument detection model, an anatomy detection model, and an activity detection model over time, according to some embodiments. [Figure 12]

[0075] FIG. 1 is a schematic diagram of elements in an AI-assisted surgery system, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0056]

[0076] While various embodiments of the present invention are shown and described herein, those skilled in the art will appreciate that It will be apparent that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed, as described in more detail below.

[0057]

[0077] Various embodiments of the present invention provide computer-implemented medical systems, devices, and methods for using AI to assist surgeons in intraoperative situations. The systems, devices, and methods disclosed herein may improve existing methods of surgical assistance by providing improved classification (e.g., real-time) of various elements involved in a surgical procedure (e.g., surgical instruments, anatomical features, anatomical lesion / injury features, and procedure). One or more embodiments of the systems, devices, and methods presented herein may achieve this goal by using AI methods (e.g., machine learning, deep learning) to build classifiers that improve real-time classification of elements involved in a surgical procedure. Various embodiments of the AI ​​approaches and implementations described herein may leverage large datasets to gain new insights from the datasets. The classifier model can improve real-time characterization of various elements involved in a surgical procedure, which may lead to higher surgical success rates due to, for example, reduced errors due to misidentification of anatomical structures. The classifier model can provide operators (e.g., surgeons, operating room nurses, surgical technicians) with information to enable more accurate and timely decision-making (e.g., labeling important anatomical features in real time). This leads to fewer errors and improved performance through more accurate and precise surgical actions, such as placing an implant (e.g., an anchor) in a desired location or more completely removing damaged or diseased tissue (e.g., torn tendon or cartilage or tumorous tissue) while preserving healthy tissue from certain anatomical structures at the surgical site.

[0058]

[0078] The computer-implemented medical systems, devices, and methods disclosed herein can improve upon existing methods of clinical decision support systems by leveraging parameters related to various elements in the context of a surgical procedure to continuously achieve accurate real-time decision-making. A surgical procedure involves various elements, such as a patient's body with various parts and anatomical complexity, multiple instruments and devices, and actions based on the surgical procedure, along with actions based on atypical events that may occur during the procedure. The systems, devices, and methods disclosed herein can continuously operate even as the surgical environment changes to make classifications and suggestions based on multiple AI modules organized to make decisions hierarchically. For example, the classifiers described herein can classify anatomical features (e.g., shoulders, knees, organs, tissues, or lesions) even when the field of view of an endoscopic camera may change during surgery. Similarly, the systems, devices, and methods described herein can recognize surgical instruments as they appear in the field of view.

[0059]

[0079] The systems, devices, and methods as disclosed herein can be used to classify various elements involved in surgery. For example, the classifiers disclosed herein can identify and label anatomical structures (e.g., anatomical sites, organs, tissues), surgical instruments, or procedures being performed during surgery.

[0060]

[0080] The present invention can aid in the recognition of critical structures (e.g., nerves, arteries, veins, bones, cartilage, ligaments) or lesions (e.g., tissue requiring removal). Critical structures may be visible or invisible in the field of view. The systems, devices, and methods described herein can identify and mark (e.g., by color marking on the video stream) critical structures. Systems according to many embodiments can comprise multiple surgical AI modules organized to make decisions hierarchically. The surgical AI assistance modules disclosed herein can analyze video feeds of the surgical field and provide near-continuous decision-making. These modules communicate with message processors that can process the flow of video data and multiple outputs (e.g., decisions). The AI ​​system can react to changes in the surgical field. The disclosed method and system of the present invention can be applied to a variety of surgical procedures involving different anatomical sites and organs. The surgical AI assistance can come into play when the surgeon changes the field of view, i.e., when accessing the surgical field from a different portal.

[0061]

[0081] Reference will now be made in detail to various embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention and the described embodiments. However, embodiments of the invention may optionally be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments. In the drawings, like reference numbers indicate like or similar steps or components.

[0062]

[0082] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the scope of the claims. When used in describing the embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Also, the term "and / or," as used herein, should be understood to refer to and encompass any and all possible combinations of one or more of the associated listed items.

[0063]

[0083] As used herein, the word "if" is optionally interpreted to mean "when" or "when," or "in response to determining," or "in accordance with determining," or "in response to detecting," depending on the context, that the preceding stated condition is true. Similarly, the phrases "when it is determined that [the preceding stated condition is true]" or "when [the preceding stated condition is true]" or "when [the preceding stated condition is true]" are optionally interpreted to mean "upon determining," or "in response to determining," or "in accordance with determining," or "upon detecting," or "in response to detecting," that the preceding stated condition is true, depending on the context.

[0064]

[0084] As used herein, unless otherwise specified, the term "about" or "approximately" refers to an acceptable range of error for a particular value as determined by one of ordinary skill in the art, which depends in part on how the value is measured or determined. In certain embodiments, the term "about" or "approximately" means within 1, 2, 3, or 4 standard deviations. In certain embodiments, the term "about" or "approximately" means within 30%, 25%, 20%, 15%, 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, 0.5%, 0.1%, or 0.05% of a given value or range.

[0065]

[0085] As used herein, the words "comprises," "comprising," or any other variation thereof, are intended to cover non-exclusive inclusions, such that a process, method, article, or apparatus that comprises the recited elements not only includes those elements, but may also include other elements not expressly recited or that are inherent to such process, method, article, or apparatus.

[0066]

[0086] As used herein, the terms "subject" and "patient" are used interchangeably. As used herein, the term "subject" refers to a human being. In certain embodiments, the subject has undergone surgery. In certain embodiments, the subject is 0 to 6 months, 6 to 12 months, 1 to 5 years, 5 to 10 years, 10 to 15 years, 15 to 20 years, 20 to 25 years, 25 to 30 years, 30 to 35 years, 35 to 40 years, 40 to 45 years, 45 to 50 years, 50 to 55 years, 55 to 60 years, 60 to 65 years, 65 to 70 years, 70 to 75 years, 75 to 80 years, 80 to 85 years, 85 to 90 years, 90 to 95 years, or 95 to 100 years old.

[0067]

[0087] Whenever the phrase "at least," "greater than," or "greater than or equal to" is placed before the first number in a series of two or more numbers, the phrase "at least," "greater than," or "greater than or equal to" applies to each and every number in the series. For example, 1, 2, or 3 or more is equivalent to 1 or more, 2 or more, or 3 or more.

[0068]

[0088] Whenever the phrase "not greater than," "less than," or "less than or equal to" precedes the first number in a series of two or more numbers, the phrase "not greater than," "less than," or "less than or equal to" applies to each and every number in the series. For example, 3, 2, or 1 or less is equivalent to 3 or less, 2 or less, or 1 or less.

[0069]

[0089] The terms "surgical AI" or "surgical AI module," as used herein, generally refer to a system, device, or method that uses artificial intelligence algorithms to assist before, during, and / or after a surgical procedure. A surgical AI module may be defined as a combination of input data, machine learning or deep learning algorithms, training data sets, or other data sets.

[0070]

[0090] The term "machine learning," as used herein, may generally refer to computer algorithms that can automatically improve over time. Any description herein of machine learning may also apply to artificial intelligence, or vice versa, or any combination thereof.

[0071]

[0091] As used herein, the terms "continuous," "continuously," or any other variation thereof generally refer to a process that is nearly uninterrupted, or a process that has an acceptable time delay in the context of the process.

[0072]

[0092] The terms "video stream" or "video feed," as used herein, refer to data generated by a digital camera. A video feed can be a series of still images or a moving image.

[0073]

[0093] The terms "site," "organ," "tissue," and "structure," as used herein, may generally refer to anatomical features of the human body. A site may be larger than and may include an organ. An organ may include one or more tissue types and structures. A tissue may refer to a group of cells structurally joined to complete a common function. A structure can refer to a portion of a tissue. In some embodiments, a structure may refer to one or more portions of one or more tissues joined together to create an anatomical feature.

[0074]

[0094] The term "operative field" or "field of view," as used herein, refers to the range of vision captured by an interventional imaging device. Field of view may refer to the range of visual data observable by the human eye and captured by a digital camera.

[0075]

[0095] The term "decision," as described herein, may refer to the output from a machine learning or AI algorithm. Decisions may include labeling, classification, prediction, etc.

[0076]

[0096] The term "interventional imaging device," as used herein, generally refers to an imaging device used for medical purposes. Interventional imaging device may refer to an imaging device used in a surgical procedure, which may, in some embodiments, be a simulation of a surgery.

[0077]

[0097] As used herein, the term "operator" refers to a medical professional involved in a surgical procedure. An operator can be a surgeon, an operating room nurse, or a surgical technician.

[0078]

[0098] One aspect of the present invention provides a system for guiding an arthroscopic procedure. The system includes one or more computer processors and one or more non-transitory computer-readable storage media storing instructions operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations. The operations may include receiving at least one image captured by an interventional imaging device, identifying one or more image features in the received at least one image using an image recognition algorithm, labeling the identified one or more image features, where the identified one or more image features include one or more of an anatomical structure, a surgical instrument, a surgical instrument element, a procedure or action, or a lesion, and displaying the labeled one or more image features in the at least one image to an operator continuously during the course of the arthroscopic procedure. For example, in some embodiments, the procedure or action may be identified based in part on identifying the surgical instrument. In some embodiments, the interventional imaging device is an endoscope. In some embodiments, the interventional imaging device is an arthroscope. The operations may further include storing the images in a memory device. In some embodiments, the operations may further include discarding the image after the step of displaying the labeled image features to optimize memory usage.

[0079]

[0099] In some embodiments, the arthroscopic procedure can be arthroscopic surgery (or arthroscopy). Arthroscopic surgery, also known as arthroscopy or keyhole surgery, can be a minimally invasive surgical procedure on a joint (e.g., requiring only a small incision). Arthroscopy can include either a diagnostic procedure or a therapeutic procedure. Therapeutic aspects or types of arthroscopy can further include surgical repair, such as debridement or irrigation of the joint, for example, to remove small pieces of torn cartilage, ligament reconstruction, or synovectomy (removal of the joint lining). Arthroscopy can be performed using an arthroscope. The arthroscope can be inserted into a subject's body through a small incision to perform the arthroscopic procedure at or around the joint. The arthroscope can be an endoscope. The arthroscope can include a fiberscope. The arthroscope can be flexible or rigid. The arthroscope can include a camera (e.g., a digital camera), a light source, a lens that creates an image of the field of view, or a mechanism that conveys the image to a sensor. The sensor may include a global shutter (e.g., a CCD sensor) or a rolling shutter (e.g., a CMOS sensor). Images captured by the arthroscope may be displayed on a display (e.g., a monitor). An operator (e.g., a surgeon) may perform surgery using the displayed images (e.g., a video feed from the arthroscope).

[0080]

[0100] Arthroscopic surgery uses small incisions through which instruments and endoscopes are inserted to access the lesion. The arthroscope may access a joint for diagnosis or repair. Due to the minimally invasive nature of the procedure, patients may experience less pain, faster healing, or less bleeding than traditional surgery. However, arthroscopic surgery may be more technically demanding than traditional surgery (e.g., open surgery). In arthroscopy, the operator (e.g., surgeon) may operate with small instruments and a limited field of view due to a limited range of motion. Due to the limited field of view, the surgeon may need to pan the arthroscope to cover the target anatomical structure, for example. Non-limiting examples of challenges associated with arthroscopy include tracking target locations, such as intended repair locations, recognizing lesions (e.g., cancerous tissue), critical structures (e.g., nerves, arteries, veins, bones, cartilage, ligaments), and the like, visual orientation relative to anatomical structures, measuring dimensions (e.g., rotator cuff tear size) intraoperatively, tracking bone or tissue areas for implant placement when using grafts, and tracking areas of bone or tissue for implant placement when the procedure requires a larger field of view that is not sufficient for the operator to place anchors at predefined locations. The system and methods may include, for example, correlation of preoperative diagnostic imaging with intraoperative fields of view to identify critical fields of view. Critical fields of view may include lesions (e.g., tumors or cysts) or predefined implant or repair sites. In some embodiments, the systems and methods presented herein may be configured for shoulder surgery. In some embodiments, the systems and methods presented herein may be configured for knee surgery.

[0081]

[0101] In some embodiments, the imaging device (e.g., an arthroscope) The captured images may be received by a computer system. The computer system may include image recognition or other related algorithms. The image recognition algorithm may identify one or more features in the images received from the arthroscope. The image recognition algorithm may be configured to identify one or more of a surgical site or an approach (or portal) for an arthroscopic procedure (e.g., arthroscopic surgery). In some embodiments, the surgical site is a shoulder. In some embodiments, the surgical site is a knee.

[0082]

[0102] The image recognition algorithm may include a processing module. The image recognition algorithm may include an arrangement of processing modules. The arrangement of processing modules may be hierarchical. For example, a hierarchical arrangement of processing modules may include a first processing module that may be upstream of a second processing module and / or downstream of a third processing module. The image recognition algorithm may include at least two processing modules, an upstream module and a downstream module. In some embodiments, the downstream module may be more specialized than the upstream module (e.g., configured to identify features associated with a particular anatomical structure, lesion, tissue type, procedure, etc.). In some embodiments, the modules may be general-purpose modules. The specialized modules may include a portal recognition module, an anatomical structure recognition module, a module associated with a particular anatomical structure (e.g., a shoulder module, a knee module), a lesion recognition module, a module associated with a particular lesion (e.g., cancer, cartilage defects, rotator cuff tears, labral dislocation / injury, anterior cruciate ligament (ACL) injury, meniscus injury, biceps tendon rupture, synovial tissue inflammation, or femoroacetabular impingement (FAI)), or a lesion measurement module. The general-purpose module may include an instrument recognition module, an action recognition module (e.g., drilling, polishing, cleaning, etc.). In some embodiments, the processing module may include multiple upstream modules, multiple downstream modules, or a combination of both. In some embodiments, at least one of the multiple upstream modules may be configured to select at least one downstream module from multiple downstream modules of the processing module for further processing the image. In some embodiments, identifying one or more features identified in the image may further include selecting one or more processing modules from the multiple AI modules, the selection being based at least in part on output from the at least one upstream module. In some embodiments, the processing module is selected based at least on identifying one or more of the surgical site or entry point (or portal).

[0083]

[0103] In some embodiments, the upstream module is configured to execute during the arthroscopic procedure. In some embodiments, the downstream module may be configured to identify one or more of the anatomical structures manipulated by the procedure or action being performed. In some embodiments, the downstream module may be configured to recognize one or more anatomical features of the identified anatomical structures associated with the procedure or action being performed. In some embodiments, the downstream module may be configured to recognize a feature of the treatment instrument associated with the procedure or action being performed.

[0084]

[0104] The processing module may include an artificial intelligence (AI) module. In some embodiments, a processing module may include multiple AI modules. In some embodiments, a processing module or multiple AI modules may include at least a machine learning algorithm, a deep learning algorithm, or a combination of both. The machine learning algorithm may include a trained machine learning algorithm. The machine learning algorithm may include an artificial neural network. In some embodiments, a processing module or an AI in a processing module may include at least one dataset. In some embodiments, a processing module or an AI in a processing module may include at least a training dataset.

[0085]

[0105] FIG. 1 is a flow chart showing an example of a hierarchical structure 100 of modules. In some embodiments, the first module may include an area recognition module 110 that determines the surgical field. The area recognition module may receive images 101 (e.g., frames from a video feed) from an imaging device (e.g., an arthroscope). In some embodiments, the area recognition module 110 may not recognize the surgical field in the images. Therefore, the area recognition module 110 may stop forwarding the images to other modules and / or discard the frames. In some embodiments, the area recognition module 110 may recognize the surgical field in the images. The area recognition module 110 may then transmit the images to other modules downstream of the area recognition module 110, including general-purpose modules, specialized modules, or both. For example, modules downstream of the area recognition module 110 may include the portal recognition module 111, the instrument recognition module 120, or both. The images may be transmitted to two or more modules substantially simultaneously (e.g., in parallel). The images may be transmitted to two or more modules sequentially (e.g., sequentially). The image may be sent to the instrument recognition module 120, the action recognition module 121, or the procedure recognition module 122, in the order described herein. In some embodiments, the instrument recognition module 120 may recognize an instrument in the image. Module 121 may then determine an action based at least in part on the recognized instrument. Module 121 may then send the image to the procedure recognition module, which then determines a surgical procedure to be performed based at least in part on the determined action and / or the recognized instrument. The site recognition module 110 may also send the image to the portal recognition module 111 in parallel with the instrument recognition module 120. The portal recognition module 111 may then determine an entry point associated with the site recognized by the site recognition module 110. Based at least in part on the entry point recognized by the portal recognition module 111, the image may be sent to one or more specialization modules forming a specialized or customized pipeline.Such a specialized pipeline may be specialized for a particular anatomical site or location (e.g., shoulder, knee, or hip) and may include specialized modules (e.g., AI trained for a particular anatomical structure or pathology). One or more modules in this specialized pipeline may be activated upon recognition of an entry site (e.g., a saccular site) by the portal recognition module 111.

[0086]

[0106] In some embodiments, the step of labeling the image features comprises labeling the image features by pixel. In some embodiments, the method further includes pixel-wise masked labeling, bounding box labeling, frame-wise labeling, or temporal labeling. In some embodiments, pixel-wise masked labeling is used to display labeled anatomical structures or surgical instruments. In some embodiments, bounding box labeling is used to display labeled lesions, surgical instruments, or foreign bodies. In some embodiments, frame-wise labeling is used to display labeled anatomical sites. In some embodiments, temporal labeling is used to display steps in an operating procedure or to label recognized actions.

[0087]

[0107] Downstream of the part recognition module 110 and the portal recognition module 111 Thus, one or more images may be sent to one or more modules for recognizing and / or labeling one or more features in the images. Examples of labeled images are shown in FIGS. 7A-7B. The anatomy recognition module 130 shown in FIG. 1 may receive one or more images from the site recognition module 110 and the portal recognition module 111 and may recognize and label the anatomical structures (as shown in FIG. 7A). Different color masks 701a, 701b, 701c, and 701d, along with descriptive labels 702 and 703, may be used to distinguish the recognized anatomical structures. Downstream of the anatomy recognition module 130, the lesion recognition module 131 shown in FIG. 1 may receive images from the anatomy recognition module 131 and may recognize and label one or more lesions on the recognized anatomical structures (as shown in FIG. 7A). As described above, in various embodiments, the site recognition module 130, the anatomical structure recognition module, and subsequent recognition or measurement modules are specialized for a particular anatomical location (e.g., shoulder or knee) based on information identified by the portal recognition module 111 or a similar module or otherwise acquired. Because of this specialization, the computer system used during surgery to analyze and process images received from the imaging device may require less memory, programming, and processing capacity / resources. These reduced operational requirements (e.g., reduced memory, processing power) allow various image processing functions, such as one or more of anatomical structure recognition, lesion recognition, and related recognition, to be performed by devices in or near the operating room (also known as edge devices) rather than having to be performed remotely, e.g., via the cloud, thereby further accelerating the process and making it more reliable because it does not require sending or receiving data over the internet to a cloud or external computer device. Furthermore, in various embodiments, no internet or other network connection is required. In this manner, one or more of the reliability, speed, and cybersecurity of embodiments of the present invention are substantially improved.

[0088]

[0108] FIG. 7B uses a colored mask 701d with descriptive labels 705. 7A and 7B show examples of lesions identified and labeled using the labrum. For example, a tear has been detected and labeled in the labrum (FIG. 7B). Downstream of the lesion recognition module 131, the lesion measurement module 132 shown in FIG. 1 may receive images from the lesion recognition module 131 and may provide one or more measurements for the identified and labeled lesion.

[0089]

[0109] FIG. 2A shows an image of an arthroscopic surgical procedure. , are provided as input to the systems described herein. The images are processed using an image recognition model configured to analyze shoulder surgery images. The image recognition model may include a specialization module. For example, the specialization module may be used to analyze an intra-articular site (FIGS. 2A-2B). FIG. 2B illustrates an example of labeling features in the image presented in FIG. 2A using the methods and systems described herein. Features 210 (Bicep tendon (Bicep_Tndn)), 220 (Labrum), 230 (Glenoid fossa), 240 (Cora-pro), 250 (Middle glenohumeral ligament (MGH-Liga)), and 260 (Subscapularis muscle (Subscap)) may be given labels, such as a color mask and legend LG (which may also provide a color or pattern guide for other anatomical features that may be labeled and / or masked, such as Humeral head (Humrl Hd), superior glenoid process (Supra artclr), descending part of coracoid (Cora-dec), medial glenohumeral ligament (IGH-Liga), to name a few), as shown in FIG. 2B.

[0090]

[0110] 8A-8V show a surgical procedure using the systems and methods described herein. 8A and 8B show an example of identifying various features in a surgical procedure. Images (e.g., frames from a video stream) from the surgical field can be provided (FIG. 8A). This image recognition model Predictions can be generated for various classes of anatomical structures (e.g., humeral head, glenoid fossa, subscapularis muscle, biceps tendon, ligaments, tendons, etc.) that can be recognized in force images (Figure 8M (humeral head), Figure 8N (biceps tendon), Figure 8O (superior facet), Figure 8P (subscapularis muscle), Figure 8Q (labrum), Figure 8R (middle glenohumeral ligament), Figure 8S (glenoid fossa), Figure 8T (descending part of coracoid), Figure 8U (coracoid process), Figure 8V (internal glenohumeral ligament)). The predictions made by the recognition model are compared with a set of labels generated for the same class of anatomical structure by a subject matter expert (FIG. 8C (humeral head), FIG. 8D (bicep tendon), FIG. 8E (superior facet), FIG. 8F (subscapularis), FIG. 8G (labrum), FIG. 8H (middle glenohumeral ligament), FIG. 8I (glenoid fossa), FIG. 8J (descending part of coracoid), FIG. 8K (coracoid process), FIG. 8L (internal glenohumeral ligament)). After predicting the individual classes of anatomical structures, the predictions are combined into an output image (FIG. 8B), which is provided along with a prediction mask. This output image may include different color masks to distinguish different classes of identified anatomical structures. The classification process described herein may be applied to consecutive frames received from a video camera. This labeled output image may then be overlaid onto the video stream. This overlay may be performed in real time, or substantially near real time.

[0091]

[0111] FIG. 3A shows the instrument, action, or 3A illustrates an example of procedure labeling. An image recognition algorithm recognizes and labels an instrument 310 (e.g., a radiofrequency ablation instrument). Based on the recognized instrument 310, an instrument identification label 320 may be provided for the identified instrument type (e.g., a radiofrequency ablation instrument), and an action recognition module may identify the action being performed and provide an activity label 330 (e.g., indicating that irrigation is being performed). A procedure recognition module may determine the procedure being performed and provide a procedure label 340 indicating, for example, a coracoid decompression being performed.

[0092]

[0112] FIG. 3B shows the instrument, action, or 3 illustrates another example of procedure labeling. An image recognition algorithm recognizes and labels an arthroscopic burr instrument with label 350. The instrument may include a surgical probe, a shaver, a burr sharpener, a drilling instrument, an implant, a drill guide, a radiofrequency or other ablation instrument, an anchor, a grasper, a threader, or scissors. Based on the recognized instrument, an action recognition module determines, as label 360, that sharpening is the action being performed. The procedure recognition module determines, as label 370, that the procedure is a coracoid decompression. Arthroscopic procedures may include actions performed on areas including the shoulder, knee, or hip. Shoulder surgery may be performed at an intra-articular site and / or a capsular site. Anatomical structures that may be recognized at an intra-articular site may include the humeral head, labrum, glenoid fossa, supraspinatus, biceps tendon, glenohumeral ligament, and subscapularis. Anatomical structures that may be recognized at a capsular site may include the acromion, clavicle, humeral head, capsule, or rotator cuff. A surgical procedure may be initiated from one or more approaches into a surgical site (e.g., an intra-articular site adjacent to the knee, a bursa, an approach). Thus, the view of the surgical field may vary based at least on the approach or approach angle used in the surgery. Figure 3C illustrates an example of recognizing any instrument 380 in the surgical field.

[0093]

[0113] In some embodiments, the operation may involve presenting the proposed action to an operator (e.g., The proposed action may further include providing the proposed action to an operator (e.g., a surgeon). The operator may perform the procedure (e.g., arthroscopic surgery). The operator may be a person other than the surgeon. The operator may operate an imaging device (e.g., an arthroscope). In some embodiments, the proposed action is to assist the operator during the procedure (e.g., an arthroscope). For example, the surgeon may require measurements of tissue or a lesion, and the methods and systems described herein may provide the measurements to the surgeon during the procedure. In some embodiments, the proposed action may include a safety warning that makes the operator aware of potential safety issues. For example, the system may recognize that the distance of one or more implants implanted in the patient differs from a predefined safe distance. This distance may be the distance between two or more implants from each other. In some embodiments, the distance may be the distance of one or more implants from an anatomical structure, anatomical feature, or lesion. The suggested action may include avoiding critical anatomical features, such as veins, arteries, nerves, bone, cartilage, or ligaments. In some embodiments, the suggested action, including a safety warning, is based on at least the identified anatomical feature, the recognized instrument, the identified action, or a combination thereof. For example, an instrument (e.g., a burr sharpener) that may potentially damage tissue (e.g., cartilage) may be recognized. If the system recognizes an instrument, such as a burr sharpener, in close proximity to recognized tissue, e.g., cartilage, the system may issue a safety warning. The suggested action may include suggesting an approach angle for the device. This approach angle may be the approach angle of a drilling instrument. In some embodiments, the suggested action is provided for educational purposes. For example, a video stream of a medical procedure (e.g., an arthroscopy) can be used for educational purposes. The methods and systems described herein can be used to label features that are overlaid on the video stream (e.g., an image or frame of the video stream). Suggested actions may also be overlaid on the video stream for educational purposes.

[0094]

[0114] In some embodiments, the suggested action may involve labeling the image (e.g., The proposed action may be based at least in part on an output from one of a plurality of modules (e.g., an upstream module or a downstream module). In some embodiments, the proposed action is based at least in part on an output from at least one of a plurality of upstream modules. In some embodiments, the proposed action is based at least in part on an output from at least one of a plurality of downstream modules.

[0095]

[0115] In some embodiments, the images may be generated from a surgical video stream. In some embodiments, the surgical video stream is an arthroscopic surgery video stream. In some embodiments, the surgical video stream is monocular or stereoscopic. In some embodiments, the labeling of features on the images may be performed at a rate similar to the rate at which images are acquired from the imaging device. The arthroscope may generate continuous images (e.g., video feed) at a rate of at least about 10 frames per second (fps).

[0096]

[0116] In some embodiments, the image recognition algorithm uses a database (e.g., The database is trained using a training dataset. In some embodiments, this database may include multiple training images. The multiple training images may include one or more surgical procedures, surgical instruments, surgical instrument elements, anatomical structures, or lesions. In some embodiments, the training dataset may be generated using image editing techniques. The image editing techniques may include augmenting images of a site, portal, anatomical structure, or anatomical feature with images of a surgical instrument. The augmented images may then be used to train an image recognition algorithm to recognize the instrument within the context of the site, portal, anatomical structure, or anatomical feature. In some embodiments, the image editing or augmentation method may include rotating the training images to improve the robustness of the image recognition algorithm to patient position or orientation, for example, during arthroscopic surgery. In some embodiments, the image editing or augmentation method may include magnifying or cropping the training images to improve robustness to changes in depth of field. In some embodiments, the image editing or augmentation method may include flipping the training images along a vertical axis to improve robustness to procedures performed on the right or left side of the patient. The surgical images used for such training and extended training, including image editing, may be of one or more of arthroscopic surgery, bariatric surgery, cardiovascular surgery, bowel surgery, gynecological surgery, urological surgery or related procedures. The procedure may be selected from a wide variety of procedures, including one or more minimally invasive procedures such as arthroscopic, endoscopic, laparoscopic, and cardioscopic procedures that may correspond to the above.

[0097]

[0117] FIG. 4 is a flowchart of an example of training an image recognition algorithm. The AI ​​training method 400 may include a dataset 410. The dataset 410 may include images of surgical instruments, anatomical structures, anatomical features, surgical instrument elements, etc., obtained from a video feed of an arthroscope, a surgical portal, a surgical site, etc. The dataset may further include images edited or augmented using methods previously described herein. The images in the dataset 410 may be separated into at least a test dataset 420 and a training dataset 430. The dataset 410 may be divided into multiple test datasets and / or multiple training datasets. In a model training step 440, the training dataset may be used to train an image recognition algorithm. For example, multiple labeled images may be provided to an image recognition algorithm to train the image recognition algorithm, including a supervised learning algorithm (e.g., a supervised machine learning algorithm or a supervised deep learning algorithm). Unlabeled images may be used to build and train an image recognition algorithm, including an unsupervised learning algorithm (e.g., an unsupervised machine learning algorithm or an unsupervised deep learning algorithm). The trained model may be tested using the test dataset (or validation dataset). The test dataset may include unlabeled images (e.g., labeled images, but with the labels removed to test the trained model). The trained image recognition algorithm may be applied to the test dataset, and the prediction results may be compared to the actual labels associated with the removed data (e.g., images) to generate the test dataset in a test model prediction step 460. The model training step 440 and the test model prediction step 460 may be repeated using different training and / or test datasets until a predefined result is met. This predefined result may be an error rate. This error rate may be defined as one or more of accuracy, specificity, or responsiveness, or a combination thereof.The tested model 450 may then be used to generate predicted results 470 for labeling features in images from an imaging device (e.g., an arthroscope) used during a medical procedure (e.g., arthroscopy), which may include multiple predicted results 480 including the site of surgery, the surgical portal, anatomical structures, lesions, instruments, actions performed, procedures performed, etc.

[0098]

[0118] used in simulating knee surgery as described herein An example of an AI pipeline such as that described in is presented in Figure 6. In this example and related examples, a video feed of the surgery may be provided on a first screen 601. A second screen may provide AI inference of the surgical field 602, including an anatomical mask as previously described herein. The AI ​​feed may be overlaid on the surgical video feed in real time. The AI ​​feed and the surgical video feed may be provided simultaneously on the same screen.

[0099]

[0119] Another aspect of the present invention is a computer-implemented method for guiding an arthroscopic procedure. In some embodiments, the computer-implemented method may include receiving at least one image captured by an interventional imaging device, identifying one or more image features in the received at least one image using an image recognition algorithm, and labeling the identified one or more image features, wherein the identified one or more image features and the labeled one or more image features in the at least one image are displayed to an operator continuously during the course of the arthroscopic procedure. The labeled one or more image features may be displayed in real time or concurrently with the arthroscopic procedure. The identified one or more image features may be associated with an anatomical structure, a surgical instrument, a surgical instrument element, a surgical procedure or action, or may include one or more of the lesions.

[0100]

[0120] Another aspect of the present invention is to train an algorithm for guiding an arthroscopic procedure. The present invention provides a method for training an algorithm for identifying and labeling one or more image features in an anatomical structure, a surgical instrument, a surgical instrument element, a procedure or action, or a lesion. The method for training the algorithm may include receiving a set of image features based on one or more images related to an arthroscopic procedure, receiving a training dataset, the training dataset including one or more labeled images related to the arthroscopic procedure, recognizing one or more of the image features in the images of the training dataset, and constructing an image recognition algorithm based at least in part on the recognition of the one or more image features and the received training dataset. The one or more image features may relate to visual characteristics of one or more of an anatomical structure, a surgical instrument, a surgical instrument element, a procedure or action, or a lesion. The image recognition algorithm may be configured to identify and label one or more image features in untagged images related to the arthroscopic procedure.

[0101]

[0121] The image recognition algorithm may be trained using training data. The images may include images where subject matter experts track the contours of various anatomical structures, lesions, instruments, etc. The training process is similar to the example shown in Figures 8A-8V, as described elsewhere herein. These subject matter expert-labeled images may also be used to train an algorithm, for example, to recognize a given instrument performing a given action. A combination of instrument recognition, anatomical recognition, and action recognition may be used to predict a surgical procedure to be performed. For example, an anatomical feature or site (e.g., a capsule) may be recognized first, which may then trigger other recognition modules to recognize the instrument or action to be performed. Figure 3B illustrates an example in which, if the surgical site is recognized as a capsule, a burr instrument, identified by label 350, and a grinding action, identified by label 360, may be recognized. The combination of these prediction results may lead to the recognition of the procedure as a coracoid decompression, identified by label 370.

[0102]

[0122] In some embodiments, the instrument may not be present in the image. The system may still recognize activities based at least in part on recognized anatomical features, analyzed previous images, or a combination thereof. Previous images processed by the system may include recognizing instruments, actions, surgical procedures, or a combination thereof. FIG. 9 illustrates an example in which an instrument is not present in the image (NA or Not Applicable), as indicated by label 910, but is identified as label 920 and recognized by the system as an action (e.g., removal). The surgical procedure identified by label 930 may not be assigned to the image ( FIG. 9 ). For example, an operator may change instruments during the course of a surgical procedure. In some embodiments, changing instruments may take several seconds or minutes. The systems and methods described herein can maintain a memory of recognized instruments. In some other cases, instruments may be obscured or unrecognizable (e.g., camera movement may cause the instrument's orientation to become unrecognizable). To compensate for brief losses in instrument recognition, an AI architecture may be used. The AI ​​architecture may include a neural network architecture. The neural network architecture may include a long short-term memory (LSTM).

[0103]

[0123] FIG. 10 illustrates an example process for identifying a surgical procedure as described herein. 10 is an exemplary flowchart. An image frame 1001 with annotations may be received and segmented into one or more segments using one or more classifier models. The classifier models may include an instrument recognition model 1002, an anatomical structure detection model 1003, an activity detection model 1004, or a feature learning model 1005. The outputs of the one or more classifiers may be combined using a long short-term memory (LSTM) 1006. LSTM is an artificial recurrent neural network (RNN) classifier that can be used to make predictions based on image recognition at a given time compared to previous recognitions. That is, LSTMs can be used to generate a memory of the context of the image being processed, as described herein. The image context is then used to predict the stage of the operation, including the surgical procedure. Rule-based decision making that combines the classified segments into a single image can then be processed to identify / predict the surgical procedure 1007 to be performed.

[0104]

[0124] Another aspect of the present invention is a hierarchical pipeline for guiding arthroscopic procedures. A system for implementing an arthroscopic procedure is provided, which may include one or more computer processors and one or more non-transitory computer-readable storage media storing instructions operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations. The operations may include (a) receiving at least one image captured by an interventional imaging device, (b) identifying one or more image features of a site of treatment or an entry point at the site based on at least one upstream module, (c) operating a first downstream module to identify one or more image features of an anatomical structure or lesion based at least in part on the one or more image features identified in step (b), (d) operating a second downstream module to identify one or more image features of a surgical instrument, surgical instrument element, procedure, or action related to the arthroscopic procedure based at least in part on the one or more image features identified in step (b), and (e) labeling the identified one or more image features and displaying the labeled one or more image features in the at least one image to an operator continuously during the course of the arthroscopic procedure. At least one upstream module may include a first trained image processing algorithm, a downstream module may include a second trained image processing algorithm, and a second downstream module may include a third trained image processing algorithm.

[0105]

[0125] FIG. 11A illustrates the process of identifying a surgical procedure as described herein. FIG. 11 illustrates another example. An image frame 1101 with annotations may be received and segmented into one or more segments using one or more classifier models. The classifier models may include an instrument recognition model 1102, an anatomy detection model 1103, and / or an activity detection model 1104. Output from the one or more classifiers may be processed and combined using a rule-based decision-making algorithm 1106. A predicted surgical procedure 1107 may be provided as labels overlaid on the surgical video stream.

[0106]

[0126] FIG. 11B shows the results of the method described herein over time (e.g., minutes of procedure). 11B illustrates example outputs from each of the components involved. The outputs shown in FIG. 11B may include recognized instrument movement, anatomical structure movement, and / or changes in activity. Instrument movement may be due to camera movement or a change in the procedure if an instrument is changed. An anatomical structure movement may be due to a structural change in the anatomical structure due to the procedure or due to camera movement, for example. In some embodiments, the operator may pause or change an activity during a procedure and resume the action after pausing. An example output from the instrument recognition model 1102 (e.g., recognized instrument movement) is shown in graph 1112, an example output from the anatomical structure detection model 1103 (e.g., recognized anatomical structure movement) is shown in graph 1113, and an example output from the activity detection model 1104 (e.g., a pause or change in activity or surgical procedure) is shown in graph 1114. To achieve seamless labeling of predicted procedures, the instrument recognition model 1102, the anatomical structure detection model 1103, and the activity detection model 1104 are integrated into the instrument recognition model 1102, anatomical structure detection model 1103, and the activity detection model 1104. The outputs of the activity detection model 1104 and / or the activity detection model 1106 are combined (e.g., added) as shown in graph 1115. In some embodiments, the sum of those outputs is averaged over time to generate an averaged sum as shown in graph 1116. In some embodiments, the averaged sum is processed using a smoothing algorithm to generate a smoothed averaged sum. The smoothing algorithm may include a Gaussian smoothing function. This smoothing algorithm may include a convolutional neural network (e.g., a convolutional neural network) whose activity is shown in graph 1117.

[0107]

[0127] FIG. 12 illustrates an example of the AI-assisted surgical method and system described herein. 12 is a schematic diagram of illustrative elements. The system and method includes model development 1201, model repository 1202, and may include a label or mask providing (e.g., visualizing) 1203 element. The development of an AI / machine learning model or algorithm used to process surgical images may include a training and / or validation step 1210. Test data 1211 may be used to train and validate the AI / machine learning model. Features identified / recognized in images acquired from a surgery (e.g., frames of a surgical video) may be visualized (e.g., masked or labeled) and overlaid 1203 on a video or image of the surgical field. Computer Systems

[0128] Various embodiments of the present invention are directed to a computer programmed to carry out the methods of the present invention. A computer system is also provided. Accordingly, details of one or more embodiments of such a computer system are described below. FIG. 5 illustrates a computer system 501 that is programmed or otherwise configured to perform one or more functions or operations of the methods of the present invention. The computer system 501 can coordinate various aspects of the present invention, such as receiving images from an interventional imaging device, identifying features in the images using image recognition algorithms, overlaying the features on a video feed on a display device, and making recommendations or suggestions to an operator based on the identified features in the images. The computer system 501 can be a user's electronic device or a computer system located remotely relative to the electronic device. The electronic device can be a mobile electronic device.

[0108]

[0129] The computer system 501 may include a single-core or multi-core processor. The computer system 501 includes a central processing unit (CPU, also referred to herein as "processor" and "computer processor") 505, which may alternatively be multiple processors for parallel processing. The computer system 501 further includes memory or storage 510 (e.g., random access memory, read-only memory, flash memory), an electronic storage unit 515 (e.g., a hard disk), a communication interface 520 (e.g., a network adapter) for communicating with one or more other systems, and peripheral devices 525, such as cache, other memory, data storage, and / or electronic display adapters. The memory 510, storage unit 515, interface 520, and peripheral devices 525 are in communication with the CPU 505 via a communication bus (solid lines), such as a motherboard. The storage unit 515 may be a data storage unit (or data repository) for storing data. The computer system 501 may be operatively coupled to a computer network ("network") 530 using the communication interface 520. Network 530 may be the Internet, an Internet and / or extranet, or an intranet and / or extranet in communication with the Internet. In some embodiments, network 530 is a telecommunications network and / or a data network. Network 530 may include one or more computer servers that may enable distributed computing, such as cloud computing. Network 530 may include a network of computers connected to computer system 501 to act as a client or a server in some embodiments using computer system 501. A peer-to-peer network can be implemented that may allow for integrated devices.

[0109]

[0130] The CPU 505 may be embodied in a program or software, a machine-readable The CPU 505 may execute a sequence of instructions, which may be stored in a storage location such as memory 510. The instructions may be issued to the CPU 505, which may then be programmed or otherwise configured to perform the methods of the present invention. Examples of operations performed by the CPU 505 may include fetch, decode, execute, and writeback.

[0110]

[0131] The CPU 505 may be part of a circuit such as an integrated circuit. One or more other components may be included in the circuit, hi some embodiments, the circuit is an application specific integrated circuit (ASIC).

[0111]

[0132] The storage unit 515 stores drivers, libraries, and stored programs. Storage unit 515 may store files such as, for example, user preferences and user programs. Computer system 501 in some embodiments may include one or more additional data storage units external to computer system 501, such as a data storage unit located on a remote server in communication with computer system 501 via an intranet or the Internet.

[0112]

[0133] The computer system 501 communicates with one or more computers via a network 530. The computer system 501 may communicate with multiple remote computer systems. For example, the computer system 501 may communicate with a user's remote computer system (e.g., a portable computer, a tablet, a smart display device, a smart TV, etc.). Examples of remote computer systems include a personal computer (e.g., a portable PC), a slate or tablet PC (e.g., an Apple® iPad, a Samsung® Galaxy Tab), a phone, a smartphone (e.g., an Apple® iPhone, an Android-enabled device, a Blackberry®), or a personal digital assistant. A user can access the computer system 501 via the network 530.

[0113]

[0134] The methods described herein involve the use of electronic storage locations in a computer system 501, The instructions may be implemented via machine (e.g., computer processor) executable code stored, for example, on memory 510 or electronic storage unit 515. In one or more embodiments, the machine-executable or machine-readable code may be provided in the form of software. In use, the code may be executed by processor 505. In some embodiments, the code may be retrieved from storage unit 515 and stored in memory 510 for access by processor 505. In some cases, electronic storage unit 515 may be excluded, and the machine-executable instructions are stored in memory 510.

[0114]

[0135] The code can be used in conjunction with a machine having a processor adapted to execute the code. The code can be pre-compiled and configured for use in, or it can be compiled on the fly. The code can be supplied in a programming language that can be selected to allow it to run in a pre-compiled or compiled fashion.

[0115]

[0136] The systems and methods provided herein, such as computer system 501 Aspects of the technology may be embodied in programming. Various aspects of the technology are typically in the form of machine (or processor) executable code and / or associated data carried on or embodied in some kind of machine-readable medium. The software may be considered a "product" or "article of manufacture." The machine-executable code may be stored in an electronic storage unit, such as a memory (e.g., read-only memory, random-access memory, flash memory) or a hard disk. "Storage" type media may include any or all of the tangible memory of a computer, processor, etc., or its associated modules, such as various semiconductor memories, tape drives, disk drives, etc., which may provide non-transitory storage for the software programming at any given time. All or portions of the software may, in some cases, be communicated via the Internet or various other telecommunications networks. Such communication may, for example, enable loading of the software from one computer or processor to another, e.g., from an administrative server or host computer to an application server computer platform. Accordingly, other types of media that may bear software elements include light waves, radio waves, and electromagnetic waves, such as those used across physical interfaces between local devices over wired and optical cable networks, and over various airlinks. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., may also be considered software-bearing media. As used herein, unless limited to non-transitory tangible "storage" media, terms such as computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

[0116]

[0137] Thus, machine-readable media such as computer-executable code may be used in conjunction with tangible storage media, A carrier wave medium may take many forms, including, but not limited to, a physical transmission medium. Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices in any computer, such as those shown in the figures, that may be used to implement databases, etc. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables, i.e., copper wire and fiber optics, including the wiring that comprises a bus within a computer system. Carrier wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, common forms of computer readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card paper tape, any other physical storage media with perforated patterns, RAM, ROM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or memory cartridges, carrier waves carrying data or instructions, cables or links carrying such carrier waves, or any other medium from which a computer can read programming code and / or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0117]

[0138]

[0136] The computer system 501 may receive, for example, video from an arthroscope. An electronic display 535 may be included or in communication with the electronic display 535, which includes a user interface (UI) 540, to provide an overlay of identified features on the feed or to provide recommendations to the operator during the course of a surgical procedure. Examples of UIs include, but are not limited to, graphical user interfaces (GUIs) and web-based user interfaces.

[0118]

[0139] The methods and systems of the present invention are implemented via one or more algorithms. When executed by the central processing unit 505, an algorithm may be implemented via software. The algorithm may, for example, receive an image from an interventional imaging device, identify features in the image using an image recognition algorithm, and then process the features. , may include overlaying it on a video feed on a display device and making recommendations or suggestions to the operator based on identified features in the image.

[0119]

[0140] Although preferred embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that It should be apparent that such embodiments are provided by way of example only. The present invention is not intended to be limited by the specific examples provided herein. While the present invention has been described with reference to the foregoing specification, the description and illustration of the embodiments herein are not intended to be construed in a limiting sense. Numerous variations, changes, and substitutions may occur to those skilled in the art without departing from the present invention. Furthermore, it should be understood that all aspects of the present invention are not limited to the specific depictions, configurations, or relative proportions set forth herein, which depend upon a variety of conditions and variables. It should further be understood that various alternatives to the embodiments of the present invention described herein may be employed in practicing the present invention. Accordingly, it should be understood that the present invention encompasses various alternatives, modifications, variations, or equivalents of the embodiments of the present invention described herein.

[0120]

[0141] Also, certain implementations may be used to form many further embodiments within the scope of the present invention. Elements, features, or operations from the embodiments may be readily recombined or substituted for one or more elements, features, or operations from other embodiments. Furthermore, elements shown or described as being combined with other elements may exist as standalone elements in various embodiments. Furthermore, embodiments of the present invention specifically contemplate the exclusion of an element, operation, or feature, etc., if that element, operation, or feature is explicitly recited. Accordingly, the scope of the present invention is not limited to the details of the described embodiments, but rather is limited only by the appended claims.

Claims

1. 1. A system for guiding an arthroscopic procedure, comprising: one or more computer processors; and one or more non-transitory computer-readable storage media storing instructions operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations, the operations comprising: receiving at least one image captured by an interventional imaging device; using an image recognition algorithm to identify one or more image features in the at least one received image; labeling the identified one or more image features, wherein the identified one or more image features include one or more of an anatomical structure, a surgical instrument, a surgical instrument element, a procedure or action, or a pathology; displaying the labeled one or more image features in the at least one image to an operator continuously during the arthroscopic procedure; Including, the system.

2. The system of claim 1 , wherein the labeled one or more image features are displayed in real time or concurrently with the arthroscopic procedure.

3. The system of claim 1 , wherein the arthroscopic procedure is arthroscopic surgery.

4. The system of claim 1 , wherein the image recognition algorithm comprises a hierarchical arrangement of processing modules.

5. The system of claim 4 , wherein the processing module includes a plurality of artificial intelligence (AI) modules.

6. The system of claim 4 , wherein the processing module includes at least a machine learning algorithm, a deep learning algorithm, or a combination of both.

7. The system of claim 6 , wherein the machine learning algorithm comprises an artificial neural network.

8. The system of claim 4 , wherein the processing module includes at least one data set.

9. The system of claim 4 , wherein the processing module includes at least one training data set.

10. The system of claim 4 , wherein the processing modules include an upstream module and a downstream module, the downstream module being more specialized than the upstream module.

11. The system of claim 10 , wherein the upstream module is configured to identify one or more anatomical structures manipulated by a manipulation procedure or action being performed during the arthroscopic procedure.

12. The downstream module recognizes anatomical features of the identified anatomical structure or recognizes treatment tool features associated with the procedure or action being performed. The system of claim 10 configured to do one or more of the following:

13. The system of claim 10 , wherein the processing modules include a plurality of upstream modules or a plurality of downstream modules.

14. The system of claim 13 , wherein at least one of the plurality of upstream modules is configured to select an individual downstream module from among the processing modules for use.

15. The system of claim 1 , wherein the operational procedure or the action is identified based in part on the step of identifying the surgical instrument.

16. The system of claim 1 , wherein the interventional imaging device is an arthroscope.

17. The system of claim 1 , wherein the interventional imaging device is an endoscope.

18. The system of claim 1 , wherein the image recognition algorithm is configured to identify one or more of a surgical site or an entry point for the arthroscopic procedure.

19. The system of claim 18 , wherein the surgical site is a shoulder.

20. The system of claim 18 , wherein the surgical site is a knee.

21. The system of claim 4 , wherein at least one of the processing modules is selected based at least on the step of identifying the one or more of the surgical site or the entry points.

22. The system of claim 1 , wherein the actions performed further comprise storing the at least one image in a memory device.

23. The system of claim 1 , wherein the operations further comprise discarding the at least one image after the step of displaying a sign element to optimize memory usage.

24. The system of claim 1 , wherein labeling the one or more image features further comprises pixel-wise masked labeling, bounding box labeling, frame-wise labeling, or temporal labeling.

25. 25. The system of claim 24, wherein the pixel-wise masked labeling is used to display the labeled anatomical structure or the surgical instrument.

26. 25. The system of claim 24, wherein the bounding box labeling is used to display the marked lesion, the surgical instrument, or a foreign object.

27. 25. The system of claim 24, wherein the frame-by-frame labeling is used to display labeled anatomical locations.

28. 25. The system of claim 24, wherein the temporal labeling is used to display the steps that label the operating sequence or the actions.

29. The system of claim 1 , wherein the operation further comprises providing a suggested action.

30. 30. The system of claim 29, wherein the suggested action is based at least in part on the labeling of the at least one image.

31. 30. The system of claim 29, wherein the proposed action is based at least in part on the upstream module.

32. 30. The system of claim 29, wherein the proposed action is based at least in part on the downstream module.

33. 30. The system of claim 29, wherein the suggested action is to assist the operator in the course of the surgery.

34. 30. The system of claim 29, wherein the suggested actions are provided for educational purposes.

35. 30. The system of claim 29, wherein the suggested action includes providing a safety warning based at least on identification of a critical anatomical structure, the surgical instrument, the action, or a distance between two or more implants.

36. 36. The system of claim 35, wherein the distance between the two or more implants poses a health risk.

37. 36. The system of claim 35, wherein the distance between the two or more implants is different from a predefined distance between the two or more implants.

38. 30. The system of claim 29, wherein the suggested action includes providing a suggested approach angle.

39. 39. The system of claim 38, wherein the proposed approach angle can include a drilling angle.

40. The system of claim 1 configured for shoulder surgery.

41. The system of claim 1 configured for knee surgery.

42. The system of claim 1 , wherein the image recognition algorithm is trained using a database.

43. 43. The system of claim 42, wherein the database includes a plurality of training images comprising one or more surgical procedures, surgical instruments, surgical instrument elements, anatomical structures, or lesions.

44. 43. The system of claim 42, wherein multiple augmentation methods are used to refine the training dataset to improve the robustness of the image recognition algorithm.

45. 45. The system of claim 44, wherein the augmentation method includes rotating the training images to improve the robustness to patient position or orientation during the arthroscopic procedure.

46. 45. The system of claim 44, wherein the augmentation method includes flipping the training images along a vertical axis to improve the robustness to procedures performed on the right or left side of a patient.

47. 45. The system of claim 44, wherein the augmentation method comprises enlarging or cropping the training images to improve the robustness to changes in depth of field.

48. The system of claim 1 , wherein the at least one image is generated from a surgical video stream.

49. 49. The system of claim 48, wherein the surgical video stream is an arthroscopic surgery video stream.

50. 49. The system of claim 48, wherein the surgical video stream is monocular.

51. 49. The system of claim 48, wherein the surgical video stream is stereoscopic.

52. 1. A computer-implemented method for guiding an arthroscopic procedure, comprising the steps of receiving at least one image captured by an interventional imaging device; using an image recognition algorithm to identify one or more image features in the at least one received image; labeling the identified one or more image features, wherein the identified one or more image features include one or more of an anatomical structure, a surgical instrument, a surgical instrument element, a procedure or action, or a pathology; displaying the labeled one or more image features in the at least one image to an operator continuously during the arthroscopic procedure, the labeled one or more image features being displayed in real time or concurrently with the arthroscopic procedure; A method comprising:

53. 53. The method of claim 52, wherein the arthroscopic procedure is arthroscopic surgery.

54. 53. The method of claim 52, wherein the image recognition algorithm comprises a hierarchical organization of processing modules.

55. 55. The method of claim 54, wherein the processing modules include a plurality of artificial intelligence (AI) modules.

56. 55. The method of claim 54, wherein the processing module includes at least a machine learning algorithm, a deep learning algorithm, or a combination of both.

57. 57. The method of claim 56, wherein the machine learning algorithm comprises an artificial neural network.

58. 55. The method of claim 54, wherein the processing module includes at least one data set.

59. 55. The method of claim 54, wherein the processing module includes at least one training data set.

60. 55. The method of claim 54, wherein the processing modules include an upstream module and a downstream module, the downstream module being more specialized than the upstream module.

61. 61. The method of claim 60, wherein the upstream module is configured to identify one or more of anatomical structures manipulated by a manipulation procedure or action being performed during the arthroscopic procedure.

62. 61. The method of claim 60, wherein the downstream module is configured to do one or more of: recognize anatomical features of the identified anatomical structure; or recognize treatment instrument features associated with the procedure or action being performed.

63. 61. The method of claim 60, wherein the processing modules include a plurality of upstream modules or a plurality of downstream modules.

64. 64. The method of claim 63, wherein at least one of the plurality of upstream modules is configured to select an individual downstream module from among the processing modules for use.

65. 53. The method of claim 52, wherein the operational procedure or the action is identified based in part on the step of identifying the surgical instrument.

66. 53. The method of claim 52, wherein the interventional imaging device is an arthroscope.

67. 53. The method of claim 52, wherein the interventional imaging device is an endoscope.

68. 53. The method of claim 52, wherein the image recognition algorithm is configured to identify one or more of a surgical site or an entry point for the arthroscopic procedure.

69. 69. The method of claim 68, wherein the surgical site is a shoulder.

70. 69. The method of claim 68, wherein the surgical site is a knee.

71. 55. The method of claim 54, wherein at least one of the processing modules is selected based at least on the step of identifying the one or more of the surgical site or the entry points.

72. 53. The method of claim 52, wherein the actions performed further comprise storing the at least one image in a memory device.

73. 53. The method of claim 52, wherein the operations further comprise discarding the at least one image after the step of displaying a sign element to optimize memory usage.

74. 53. The method of claim 52, wherein labeling the one or more image features further comprises pixel-wise masked labeling, bounding box labeling, frame-wise labeling, or temporal labeling.

75. 75. The method of claim 74, wherein the pixel-by-pixel masked labeling is used to display the labeled anatomical structure or the surgical instrument.

76. 75. The method of claim 74, wherein the bounding box labeling is used to display the marked lesion, the surgical instrument, or a foreign body.

77. 75. The method of claim 74, wherein the frame-by-frame labeling is used to display labeled anatomical locations.

78. 75. The method of claim 74, wherein the temporal labeling is used to display the steps that label the operating sequence or the actions.

79. 53. The method of claim 52, wherein the actions further include providing a suggested action.

80. 80. The method of claim 79, wherein the suggested action is based at least in part on the labeling of the at least one image.

81. 80. The method of claim 79, wherein the proposed action is based at least in part on the upstream module.

82. 80. The method of claim 79, wherein the proposed action is based at least in part on the downstream module.

83. 80. The method of claim 79, wherein the suggested action is to assist the operator in the course of the surgery.

84. 80. The method of claim 79, wherein the suggested actions are provided for educational purposes.

85. 80. The method of claim 79, wherein the suggested action includes providing a safety warning based at least on identifying a critical anatomical structure or a distance between two or more implants.

86. 86. The method of claim 85, wherein the distance between the two or more implants poses a health risk.

87. 86. The method of claim 85, wherein the distance between the two or more implants is different from a predefined distance between the two or more implants.

88. 80. The method of claim 79, wherein the suggested action includes providing a suggested approach angle.

89. 89. The method of claim 88, wherein the proposed approach angle can include a drilling angle.

90. 53. The method of claim 52 configured for shoulder surgery.

91. 53. The method of claim 52 configured for knee surgery.

92. 53. The method of claim 52, wherein the image recognition algorithm is trained using a database.

93. The database may include one or more of a surgical procedure, a surgical instrument, a surgical instrument element, an anatomical 93. The method of claim 92, further comprising a plurality of training images including target structures or lesions.

94. 93. The method of claim 92, wherein multiple augmentation methods are used to refine the training dataset to improve the robustness of the image recognition algorithm.

95. 95. The method of claim 94, wherein the augmentation method includes rotating the training images to improve the robustness to patient position or orientation during the arthroscopic procedure.

96. 95. The method of claim 94, wherein the augmentation method includes flipping the training images along a vertical axis to improve the robustness to procedures performed on the right or left side of a patient.

97. 95. The method of claim 94, wherein the augmentation method comprises enlarging or cropping the training images to improve the robustness to changes in depth of field.

98. 53. The method of claim 52, wherein the at least one image is generated from a surgical video stream.

99. 99. The method of claim 98, wherein the surgical video stream is an arthroscopic surgery video stream.

100. 99. The method of claim 98, wherein the surgical video stream is monocular.

101. 99. The method of claim 98, wherein the surgical video stream is stereoscopic.

102. 1. A method of training an algorithm for guiding an arthroscopic procedure, comprising: receiving a set of image features based on one or more images related to the arthroscopic procedure; receiving a training dataset, the training dataset including one or more labeled images related to the arthroscopic procedure; Recognizing one or more image features in the images of the training dataset, the one or more image features relating to visual characteristics of one or more of an anatomical structure, a surgical instrument, a surgical instrument element, a procedure or action, or a pathology; constructing an image recognition algorithm based at least in part on the recognition of the one or more image features and the received training data set, the image recognition algorithm being configured to identify and label the one or more image features in unlabeled images related to the arthroscopic procedure; A method comprising:

103. 103. The method of claim 102, wherein the labeled one or more image features are displayed in real time or concurrently with the arthroscopic procedure.

104. 103. The method of claim 102, wherein the image recognition algorithm comprises a hierarchical arrangement of processing modules.

105. 105. The method of claim 104, wherein the processing module comprises a plurality of individual image processing modules.

106. The plurality of individual image processing modules are configured to identify the arthroscopic procedure at a predetermined location.

106. The method of claim 105, comprising a first module for recognizing and labeling one or more surgical instruments and surgical instrument elements, a second module for recognizing and labeling one or more anatomical structures, or a combination thereof.

107. 105. The method of claim 104, wherein the processing modules include a plurality of artificial intelligence (AI) modules.

108. 105. The method of claim 104, wherein the processing module includes at least a machine learning algorithm, a deep learning algorithm, or a combination of both.

109. 109. The method of claim 108, wherein the machine learning algorithm comprises an artificial neural network.

110. 105. The method of claim 104, wherein the processing module includes at least one data set.

111. 105. The method of claim 104, wherein the processing module includes at least one training data set.

112. 105. The method of claim 104, wherein the processing modules include an upstream module and a downstream module, the downstream module being more specialized than the upstream module.

113. 113. The method of claim 112, wherein the upstream module is configured to identify one or more of anatomical structures manipulated by an operating procedure or action being performed during the arthroscopic procedure.

114. 113. The method of claim 112, wherein the downstream module is configured to perform one or more of: recognizing anatomical features of the identified anatomical structure; or recognizing treatment instrument features associated with the procedure or action being performed.

115. 113. The method of claim 112, wherein the processing modules include a plurality of upstream modules or a plurality of downstream modules.

116. 116. The method of claim 115, wherein at least one of the plurality of upstream modules is configured to select an individual downstream module from among the processing modules for use.

117. 103. The method of claim 102, wherein identifying the one or more image features in the at least one image further comprises selecting one or more processing modules from a plurality of processing modules, the selection being based at least in part on the site and / or the entry point of the arthroscopic surgery.

118. 103. The method of claim 102, wherein the operational procedure or the action is identified based in part on the step of identifying the surgical instrument.

119. 103. The method of claim 102, wherein the untagged image is captured by an interventional imaging device.

120. 120. The method of claim 119, wherein the interventional imaging device is an endoscope.

121. 103. The method of claim 102, wherein the untagged images are generated from a surgical video stream.

122. 122. The method of claim 121, wherein the surgical video stream is an arthroscopic surgery video stream and the endoscopic surgery is arthroscopic surgery.

123. 122. The method of claim 121, wherein the surgical video stream is monocular.

124. 122. The method of claim 121, wherein the surgical video stream is stereoscopic.

125. 103. The method of claim 102, wherein the image recognition algorithm is configured to identify one or more of a surgical site or an entry point for the arthroscopic procedure.

126. 126. The method of claim 125, wherein the surgical site is a shoulder.

127. 126. The method of claim 125, wherein the surgical site is a knee.

128. 126. The method of claim 125, wherein at least one of the processing modules is selected based at least on the step of identifying the one or more of the surgical site or the entry points.

129. 103. The method of claim 102, wherein the image recognition algorithm stores the labeled image in a memory device.

130. 103. The method of claim 102, wherein the image recognition algorithm discards the tagged images to minimize memory usage.

131. 103. The method of claim 102, wherein the labeled images include pixel-wise masked labeling, bounding box labeling, frame-wise labeling, or temporal labeling.

132. 132. The method of claim 131, wherein the pixel-by-pixel masked labeling is used to display the labeled anatomical structure or the surgical instrument.

133. 132. The method of claim 131, wherein the bounding box labeling is used to display the marked lesion, the surgical instrument, or a foreign body.

134. 132. The method of claim 131, wherein the frame-by-frame labeling is used to display labeled anatomical locations.

135. 132. The method of claim 131, wherein the temporal labeling is used to display the steps that mark the procedure or action.

136. 103. The method of claim 102, wherein the training data set is configured for shoulder surgery.

137. 137. The method of claim 136, wherein the image recognition algorithm is trained for shoulder surgery using the training dataset configured for shoulder surgery.

138. 103. The method of claim 102, wherein the training data set is configured for knee surgery.

139. 139. The method of claim 138, wherein the image recognition algorithm is trained for knee surgery using the training dataset configured for knee surgery.

140. 103. The method of claim 102, wherein the training data set comprises a plurality of training images comprising one or more surgical procedures, surgical instruments, surgical instrument elements, anatomical structures, or lesions.

141. 103. The method of claim 102, wherein multiple augmentation methods are used to refine the training dataset to improve the robustness of the image recognition algorithm.

142. 142. The method of claim 141, wherein the augmentation method includes rotating the training images to improve the robustness to patient position or orientation during the arthroscopic procedure.

143. 142. The method of claim 141, wherein the augmentation method includes flipping the training images along a vertical axis to improve the robustness to procedures performed on the right or left side of a patient.

144. 142. The method of claim 141, wherein the augmentation method comprises enlarging or cropping the training images to improve the robustness to changes in depth of field.

145. 1. A method for implementing a hierarchical pipeline to guide a minimally invasive procedure, the system comprising: one or more computer processors; and one or more non-transitory computer-readable storage media storing instructions operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations, the operations comprising: (a) receiving at least one image captured by an interventional imaging device; (b) identifying one or more image features of a site of treatment or an entry point at said site based on at least one upstream module, said at least one upstream module including a first trained image processing algorithm; (c) operating a first downstream module to identify one or more image features of an anatomical structure or lesion based at least in part on the one or more image features identified in step (b), the first downstream module including a second trained image processing algorithm; (d) operating a second downstream module to identify one or more image features of a surgical instrument, surgical instrument element, operational sequence, or action related to the minimally invasive procedure based at least in part on the one or more image features identified in step (b), wherein the second downstream module includes a third trained image processing algorithm; (e) labeling the identified one or more image features; (f) displaying the labeled one or more image features in the at least one image to an operator continuously during the minimally invasive procedure; A method comprising:

146. 146. The system of claim 145, wherein the minimally invasive procedure is an arthroscopic procedure.

147. 146. The system of claim 145, wherein steps (c) and (d) are independent of each other.

148. 10. The method of claim 1, wherein the first, second, or third trained image processing algorithm comprises at least a machine learning algorithm, a deep learning algorithm, or a combination of both.

45. The system described in

149. 149. The system of claim 148, wherein the machine learning algorithm comprises an artificial neural network.

150. 149. The system of claim 148, wherein the machine learning algorithm or the deep learning algorithm is trained using at least one training dataset.

151. 151. The system of claim 150, wherein the training data set is configured for shoulder surgery.

152. 151. The system of claim 150, wherein the training data set is configured for knee surgery.

153. 151. The system of claim 150, wherein the training data set includes a plurality of training images comprising one or more surgical procedures, surgical instruments, surgical instrument elements, anatomical structures, or lesions.

154. 151. The system of claim 150, wherein multiple augmentation methods are used to refine the training dataset to improve the robustness of the image recognition algorithm.

155. 155. The system of claim 154, wherein the augmentation method includes rotating the training images to improve the robustness to patient position or orientation during the minimally invasive procedure.

156. 155. The system of claim 154, wherein the augmentation method includes flipping the training images along a vertical axis to improve the robustness to procedures performed on the right or left side of a patient.

157. 155. The system of claim 154, wherein the augmentation method includes enlarging or cropping the training images to improve the robustness to changes in depth of field.

158. 146. The system of claim 145, wherein the first, second, or third trained image processing algorithm stores the displayed image with the labeled features in a memory device.

159. 146. The system of claim 145, wherein the first, second, or third trained image processing algorithm discards the displayed images having labeled features to minimize memory usage.

160. 146. The system of claim 145, wherein the at least one image is generated from a surgical video stream.

161. 161. The system of claim 160, wherein the surgical video stream is an endoscopic surgery video stream.

162. 161. The system of claim 160, wherein the surgical video stream is monocular or stereoscopic.

Images