System and method for computer-assisted landmark or fiducial placement in video

Abstract

Various embodiments of the present invention provide systems and methods for assisting or guiding arthroscopic surgery (e.g., shoulder, knee, or hip surgery) or other surgical procedures by the placement of arbitrary markers at one or more locations in a surgical field of view. The systems and methods include receiving a video stream from an arthroscope or other imaging device, receiving one or more sets of coordinates of one or more markers, overlaying the one or more markers on the video stream, and displaying the overlay on one or more displays intra-operatively for use by an operator during the arthroscopic surgery or other medical procedure.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This PCT application claims priority to Indian Provisional Patent Application No. 202041015993, filed April 13, 2020, and U.S. Provisional Application Nos. 63 / 030,721, filed May 27, 2020, and 63 / 143,380, filed January 29, 2021, the entire contents of which are hereby incorporated by reference in their entirety.

[0002]

[0002] Embodiments of the present invention relate to systems, devices and methods for assisting surgical procedures, particularly using artificial intelligence (AI). [Background technology]

[0003]

[0003] In recent years, artificial intelligence has begun to be developed for use in processing images to recognize features of various anatomical structures in the human body, as well as human faces. These AI tools can be used to automatically recognize anatomical features to assist operators during medical procedures. Computational methods such as machine learning and deep learning algorithms can be used for image or language processing to collect and process information generated during medical procedures. Therefore, it would be desirable to use AI algorithms that can be used to improve surgical outcomes. Current AI-assisted surgical systems and methods remain less than ideal in many respects, for example, when used to guide surgical procedures. Therefore, improved AI-assisted surgical systems and methods are desired. Summary of the Invention [Problem to be solved by the invention]

[0004] Various embodiments of the present invention relate to computer-implemented medical systems, devices, and methods for guiding surgical procedures, such as by identifying and labeling anatomical features in real time and placing one or more markers on the identified anatomical features. Surgeons use physical markers to support a wide variety of cognitive tasks, such as tracking potential vascularity, staple lines, suture sites, and potential anatomical structures. The markers are typically placed using dyes, cauterization marks, and the like. In some embodiments, needles are inserted externally to mark the points. The placement of physical markers, which may require implanting objects into the patient's body, adds complexity to the surgery and may physically restrain the movement of surgical instruments during the surgical procedure. Other problems may include operator error during potentially expensive surgical procedures. For example, it may be difficult or impossible for an operator to determine the exact location of important anatomical features that are hidden from a camera (e.g., a camera used during arthroscopic or microscopic surgery), or changes in field of view may make it difficult for the operator to recognize the location of the markers. Therefore, computer-implemented medical systems, devices, and methods, such as artificial intelligence (AI) tools, for guiding medical procedures, particularly by adding virtual landmarks on a patient's body (e.g., on organs or anatomical features), may be beneficial. These AI tools may have limitations in accurately and reliably predicting the detection of instruments, anatomical structures, or procedures. In fast-paced surgical procedures, AI tools may also need to make predictions with low latency to provide real-time assistance to the operator.

[0005] There is a need for fast, accurate, and reliable AI tools that assist operators in real time during a surgical or other medical procedure by placing virtual markers at locations of interest to facilitate the surgical (or other medical) procedure for the operator (e.g., surgeon, interventional radiologist) and to improve the outcome of the surgical or other medical procedure. As recognized herein, various aspects and embodiments of the present invention accordingly provide a pipeline of machine learning algorithms that are versatile and well-trained for the unique labeling needs of various medical procedures. [Means for solving the problem]

[0006]

[0006] Various embodiments of the present invention described herein provide systems, devices, and methods that can receive information (e.g., images, audio, user input) before or during a medical procedure (e.g., a surgical procedure), process the received information to identify features associated with marker placement associated with the procedure, and place virtual markers at locations of interest in real time during the procedure.

[0007] Aspects of the present invention further assist surgeons in placing markers at locations of interest during surgery by using images acquired preoperatively using imaging modalities and related methods such as fluoroscopy, magnetic resonance imaging (MRI), or computed tomography (CT) scans. In one or more embodiments, the preoperative images can be preoperative images of the surgical field, and artificial intelligence (AI) can be applied to the preoperatively generated images to overlay images and / or locations of markers on a real-time video stream of the surgical procedure to provide guidance to the surgeon. This refers to AI modules / algorithms used intraoperatively, preoperatively, or postoperatively to assist in the surgical procedure or to improve the outcome of the procedure as surgical AI.

[0008] One aspect of the present invention provides a system for assisting arthroscopic surgery, such as shoulder, knee, hip, ankle, or other joint repair, by enabling computer-implemented arbitrary marker placement, the system comprising one or more computer processors and one or more non-transitory computer-readable storage media storing instructions operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations including receiving a video stream from an arthroscopic imaging device, receiving one or more sets of coordinates of one or more markers, overlaying the one or more markers on the video stream, and displaying the overlay on one or more displays intra-operatively for use by an operator during the arthroscopic surgery. Application of system embodiments to assistance in other medical procedures (e.g., by arbitrary marker placement), including minimally invasive procedures such as endoscopic surgery, laparoscopic surgery, and interventional cardiovascular surgery, is also contemplated. Examples of such minimally invasive procedures may include one or more of gastrointestinal (GI) surgery (e.g., intestinal biopsy, polypectomy, bariatric surgery, gastric reduction surgery / vertical band gastroplasty), urological procedures (e.g., kidney stone removal, bladder repair), gynecological surgery (e.g., dnc, uterine myomectomy), and laparoscopic surgery (e.g., appendectomy, cholecystectomy, colectomy, hernia repair, fundoplication).

[0009] In some embodiments, the operations further include identifying and labeling one or more elements in the video stream using at least one of the trained computer algorithms. In some embodiments, the one or more elements include one or more of an anatomical structure, a surgical instrument, a procedure or action, or a pathology. In some embodiments, identifying and labeling the one or more elements in the video stream includes using one or more software modules (herein modules). In some embodiments, the one or more modules may include modules for performing video stream decomposition, instrument recognition, anatomical structure recognition, instrument tracking, gesture recognition, landmark registration, or anatomical structure and landmark tracking. In some embodiments, the system recommends one or more landmarks based at least in part on the identified elements.

[0010] In some embodiments, the operations further include storing one or more sets of coordinates of the one or more markers, changing the view of the display to exclude the overlaid markers from being displayed, returning the view to the previous display, identifying one or more sets of coordinates of the one or more markers, and re-overlaying the one or more markers. In some embodiments, an operator activates the changing and restoring steps. In some embodiments, the changing of the view step is activated automatically based on a change in the identified anatomical structure or lesion.

[0011] In some embodiments, the one or more sets of coordinates of the one or more markers are provided by an operator (e.g., a surgeon, an interventional cardiologist, a radiologist, etc.) during surgery. In some embodiments, the one or more sets of coordinates of the one or more markers are provided by the operator pre-operatively. In some embodiments, the one or more sets of coordinates of the one or more markers are generated from one or more medical images of the subject. In some embodiments, the one or more medical images are radiological images of the subject. In some embodiments, the radiological image is of a joint or other bony structure of the subject. In some embodiments, the radiological image is associated with a shoulder, knee, hip, ankle, or elbow of the subject. In some embodiments, the radiological image is generated using fluoroscopy, magnetic resonance imaging (MRI), a computed tomography (CT) scan, a positron emission tomography (PET) scan, or ultrasound imaging.

[0012] In some embodiments, the video stream is provided by an arthroscope (or other imaging device) during arthroscopic surgery. In various embodiments, the arthroscopic surgery may correspond to one or more of the following types of procedures (and for which system and module embodiments may be configured to assist): cruciate ligament repair in knee surgery, graft placement used in superior capsule reconstruction of a torn rotator cuff, decompression, removal or resection of one or more inflamed tissues, or removal or resection of one or more torn tendons, where the video stream is monocular. In one or more of the above and other procedures, the video stream is stereoscopic or monocular unless otherwise specified in a particular procedure. Also, in various implementations, system embodiments of the present invention may be configured to toggle or switch back and forth between monocular or stereoscopic input video streams and associated output video overlays.

[0013] In some embodiments, one or more computer processors receive video streams from one or more camera control units using a wired media connection. In some embodiments, the latency between receiving input from the digital camera and outputting and overlaying the video stream is at most 40 milliseconds (ms) to accommodate a digital camera with approximately 24 frames per second (fps). In some embodiments, the latency between receiving input from the digital camera and outputting and overlaying the video stream does not exceed the time between two consecutive frames from the digital camera.

[0014] In some embodiments, one or more computer processors receive video streams from one or more camera control units using a network connection. In some embodiments, the interventional imaging device is a digital camera specialized for use in arthroscopes. In some embodiments, the digital camera is mounted on a rigid scope suitable for operation in an arthroscopic joint. In some embodiments, the camera control unit is configured to control a light source and capture digital information generated by the digital camera. In some embodiments, the camera control unit converts the digital information generated by the digital camera into a video stream. In some embodiments, the camera control unit The unit records digital information generated by the digital camera in a memory device. In some embodiments, the memory device is a local memory device, while it may also be a cloud-based memory device. In some embodiments, the digital camera is connected to a camera control unit, which in various embodiments may be configured to overlay output from one or more computer processors onto the video stream.

[0015] In some embodiments, the system further comprises a display monitor. In some embodiments, the one or more computer processors include a central processing unit or a graphics processing unit (also referred to as a GPU). In some embodiments, the system further comprises a mechanism for receiving input (to activate or deactivate the marking) from at least one operator during surgery. In various embodiments, the mechanism is configured to receive input via one or more of a push button, a touch screen device, a pointing device (e.g., a mouse or a head-mounted pointing device), a foot pedal, a gesture recognition system, or a voice recognition system. In some embodiments, the one or more marks are tracked during arthroscopic surgery or other medical procedure. In some embodiments, the tracking of the one or more marks is associated with a set of coordinates of the one or more marks related to at least one or more of an anatomical structure, an injury or lesion to the structure, an implant placed in the structure, or a repair to the structure.

[0016] In some embodiments, the display of the one or more markers is overlaid on a display of the video stream. In some embodiments, the one or more markers are displayed when the relevant anatomical structures are identified in the video stream. In some embodiments, the operator can choose to make the one or more markers invisible temporarily during or throughout the arthroscopic surgery or other medical procedure.

[0017] Another aspect of the present invention provides a system for assisting arthroscopic surgery or other medical procedures by enabling computer-implemented arbitrary marker placement using radiological imaging, the system comprising one or more computer processors and one or more non-transitory computer-readable storage media storing instructions operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations. In some embodiments, the operations include receiving at least one radiographic image of a subject, identifying one or more anatomical features in the at least one radiographic image using a trained machine learning algorithm, generating a 3D representation of the identified anatomical features, receiving the locations of the one or more markers from an operator, overlaying the one or more markers on the 3D representation of the anatomical structure, and displaying the overlay on a display device used by the operator. Again, application of embodiments of the above system to assisting in other medical procedures is contemplated, including minimally invasive procedures such as endoscopic surgery, laparoscopic surgery, and interventional cardiovascular surgery.

[0018] In some embodiments, the anatomical feature includes a bone structure or a tendon. In some embodiments, the at least one radiological image includes one or more of an MRI scan, a CT scan, a PET scan, an ultrasound image, or a combination thereof. In some embodiments, the at least one radiological image includes an image of a marker. In some embodiments, the operations further include identifying a location of the marker. In some embodiments, the operations further include recommending a location for the marker based at least in part on the identified location of the marker in the at least one radiological image.

[0019] In some embodiments, at least one radiographic image or one or more targets The image is blended with a video stream from an imaging device. In some embodiments, the blended image is displayed on a display device. In some embodiments, the step of displaying the blended image occurs during arthroscopic surgery or other medical procedure. In some embodiments, the imaging device is an interventional imaging device, such as an ultrasound imaging device or a fluoroscopic imaging device. In various embodiments, the video stream may be monocular or stereoscopic, and the system may be configured to recognize either switching back and forth between the types and generate the relevant output accordingly.

[0020] Another aspect of the present invention provides a computer-implemented method for assisting in arthroscopic surgery or other medical procedures. In some embodiments, the method includes receiving a video stream from an imaging device, receiving one or more sets of coordinates of one or more landmarks, overlaying the one or more landmarks on the video stream, and displaying the overlay on one or more displays intra-operatively for use by an operator during the arthroscopic surgery or other medical procedure. Application of embodiments of the method to assisting in other medical procedures, including endoscopic surgery, laparoscopic surgery, and minimally invasive procedures such as interventional cardiovascular surgery, is also contemplated.

[0021] In some embodiments, the method further includes identifying and labeling one or more elements in the video stream using at least one of the trained computer algorithms, the one or more elements including one or more of an anatomical structure, a surgical instrument, a procedure or action, or a pathology. In some embodiments, identifying and labeling the one or more elements in the video stream includes using one or more modules. In some embodiments, the one or more modules include one or more modules for video stream decomposition, instrument recognition, anatomical structure recognition, instrument tracking, gesture recognition, landmark registration, or anatomical structure and landmark tracking. In some embodiments, one or more landmarks are recommended based at least in part on the identified elements.

[0022] In some embodiments, the method further includes storing one or more sets of coordinates of one or more markers, altering the view of the display to exclude the overlaid markers from being displayed, restoring the view to the previous display, identifying the one or more sets of coordinates of the one or more markers, and re-overlaying the one or more markers. In some embodiments, an operator activates the altering and restoring steps. In some embodiments, the altering step is activated automatically based on a change in the identified anatomical structure or lesion.

[0023] In some embodiments, one or more sets of coordinates of one or more landmarks are provided intraoperatively by an operator. In some embodiments, one or more sets of coordinates of one or more landmarks are provided preoperatively by an operator. In some embodiments, the one or more sets of coordinates of one or more landmarks are generated from one or more medical images of the subject. In some embodiments, the one or more medical images are radiological images. In some embodiments, the radiological images are generated using fluoroscopy, MRI, or CT scans. In some embodiments, the video stream is provided by an arthroscope during arthroscopic surgery. In some embodiments, the arthroscopic surgery is used in rotator cuff replacement surgery. In some embodiments, the arthroscopic surgery is used in cruciate ligament tunnel placement in knee surgery. In some embodiments, the video stream is monocular. In some embodiments, the video stream is stereoscopic.

[0024] In some embodiments, the one or more video streams are recorded from a digital camera. The step of receiving the one or more video streams from the digital camera is performed using a wired media connection. In some embodiments, the latency between receiving the input from the digital camera and displaying the output and overlay of the video stream is at most 40 milliseconds (ms) to accommodate a digital camera with approximately 24 frames per second (fps). In some embodiments, the latency between receiving the input from the digital camera and displaying the output and overlay of the video stream does not exceed the time between two consecutive frames from the digital camera, and the step of receiving the one or more video streams from the digital camera is performed using a network connection.

[0025] In some embodiments, the method is performed using one or more computer processing units. In some embodiments, the one or more computer processing units include a central processing unit or an image processing unit. In some embodiments, the interventional imaging device is a digital camera. In some embodiments, the digital camera is mounted on a viewing instrument.

[0026] In some embodiments, the camera control unit is configured to control the light source and capture digital information generated by the digital camera, hi some embodiments, the camera control unit is configured to convert the digital information generated by the digital camera into a video stream.

[0027] In some embodiments, the camera control unit records digital information generated by the digital camera in a memory device, which may be a local memory device resident on or operably coupled to a computer system that performs one or more operations / steps of the above method, or a remote memory device, such as a cloud-based memory device. In some embodiments, the digital camera is connected to the camera control unit. In some embodiments, the video stream is received from the camera control unit by one or more computer processing units for processing. In some embodiments, the camera control unit is configured to overlay output from processing by the one or more computer processing units onto the video stream. In some embodiments, the above method further comprises a display monitor.

[0028] In some embodiments, the method further includes utilizing a mechanism for receiving input from at least one operator during surgery to activate or deactivate the marking. In one or more embodiments, the mechanism may be configured to receive input via a push button, a touch screen device, a foot pedal, a gesture recognition method, or a voice recognition method.

[0029] In some embodiments, one or more markers are tracked during arthroscopic surgery or other medical procedure (endoscopic surgery, laparoscopic surgery, cardioscopic surgery). In some embodiments, the tracking of the one or more markers is associated with a set of coordinates of the one or more markers related to at least one of an anatomical structure, an injury or lesion to the structure, an implant in the structure, or a repair to the structure. In some embodiments, the display of the one or more markers is blended with the display of the video stream. In some embodiments, the one or more markers are displayed when the relevant anatomical structure is identified in the video streaming. In some embodiments, the operator can choose to make the one or more markers invisible temporarily during arthroscopic surgery or throughout the entire arthroscopic surgery.

[0030] Another aspect of the present invention provides a computer-implemented method for assisting arthroscopic surgery or other medical procedures with arbitrary landmark placement using radiological imaging. In some embodiments, the method includes receiving at least one radiographic image of a subject, identifying one or more anatomical features in the at least one radiographic image using a trained machine learning algorithm, generating a 3D representation of the identified one or more anatomical features, receiving the locations of one or more markers from an operator, overlaying the one or more markers on the 3D representation of the one or more anatomical features, and displaying the overlay on a display device used by the operator. Application of method embodiments to assistance in other medical procedures, including minimally invasive procedures such as endoscopic surgery, laparoscopic surgery, and interventional cardiovascular surgery, are also contemplated.

[0031] In some embodiments, the anatomical feature comprises a bony structure or a tendon. In some embodiments, the at least one radiological image comprises one or more of an MRI scan, a CT scan, or a combination thereof. In some embodiments, the at least one radiological image comprises an image of a landmark. In some embodiments, the method further comprises identifying a location of the landmark.

[0032] In some embodiments, the method further includes recommending a location for a marker based at least in part on the identified location of the marker in the at least one radiographic image. In some embodiments, the at least one radiographic image or the one or more markers are overlaid on a video stream from an imaging device. In some embodiments, the blended image is displayed on a display device. In some embodiments, the displaying of the blended image is during arthroscopic surgery. In some embodiments, the imaging device is an interventional imaging device. In some embodiments, the video stream is monocular. In some embodiments, the video stream is stereoscopic.

[0033]

[0033] Another aspect of the present invention provides a non-transitory computer-readable medium containing machine-executable code that, when executed by one or more computer processors, performs any of the methods described above or elsewhere in this specification.

[0034] Another aspect of the present invention provides a system comprising one or more computer processors and a computer memory coupled thereto, the computer memory containing machine-executable code that, when executed by the one or more computer processors, performs any of the methods described above or elsewhere herein.

[0035]

[0035] Further aspects and advantages of the present invention will become readily apparent to those skilled in the art from the following detailed description, in which merely exemplary embodiments of the invention are shown and described. As will be realized, the invention is capable of other and different embodiments, and its several details are capable of modification in various obvious respects, all without departing from the invention. Accordingly, the drawings and description are to be regarded as illustrative in nature and not as restrictive. Incorporation by Reference

[0036] All publications, patents, and patent applications mentioned herein are incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent that the publications and patents or patent applications incorporated by reference conflict with the disclosure contained herein, the present specification is intended to supersede and / or take precedence over such conflicting matter.

[0036]

[0037] The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be realized by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings (also referred to herein as "drawings" and "figures"), of which: [Brief explanation of the drawings]

[0037] [Figure 1]

[0038] FIG. 1 illustrates a schematic example of a hardware configuration for a system for assisting arthroscopic surgery by enabling arbitrary computer-implemented marker placement, according to some embodiments. [Figure 2A]

[0039] 10A-10C illustrate examples of marker placement on model femoral condyles, according to some embodiments. [Figure 2B] 10A-10C illustrate examples of marker placement on model femoral condyles, according to some embodiments. [Figure 3]

[0040] FIG. 1 illustrates an example flowchart of a sign placement system, according to some embodiments. [Figure 4]

[0041] FIG. 1 illustrates an overview of an exemplary workflow for landmark placement using pre-operative images, according to some embodiments. [Figure 5]

[0042] FIG. 1 illustrates an overview of an exemplary workflow of a system for recommending sign placements, according to some embodiments. [Figure 6]

[0043] 1 is a schematic flow chart of an exemplary system for processing stereoscopic video streams, according to some embodiments. [Figure 7]

[0044] FIG. 1 illustrates a computer system that is programmed or otherwise configured to perform the methods presented herein, according to some embodiments. [Figure 8]

[0045] 1A-1C illustrate examples of marking placement and occlusion compensation, according to some embodiments. [Figure 9]

[0046] 1A-1C illustrate examples of markers being removed from objects, according to some embodiments. [Figure 10]

[0047] 10A and 10B are diagrams illustrating examples of stabilizing landmarks with respect to camera movement relative to the anatomy, according to some embodiments. [Figure 11A]

[0048] FIG. 1 illustrates an example of feature detection, according to some embodiments. [Figure 11B] FIG. 1 illustrates an example of feature detection, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0038]

[0049] While various embodiments of the present invention have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions may occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed.

[0039]

[0050] Various embodiments of the present invention provide computer-implemented medical systems, devices, and methods for using AI to assist surgeons in intraoperative situations. The systems, devices, and methods disclosed herein may improve upon existing surgical landmark placement methods by providing fast and reliable classification (e.g., real-time) of various elements involved in a surgical procedure (e.g., surgical instruments, anatomical features, operative procedures) and placement of virtual landmarks with high precision and accuracy based on the classification of the various elements. For example, the systems, devices, and methods provided herein may use AI methods (e.g., machine learning, deep learning) to build classifiers that improve real-time classification of elements involved in a surgical procedure, and may utilize intraoperative frames from the operator (e.g., through the use of a surgical probe). The location of the markers is identified by a virtual machine command or by processing preoperative medical images (e.g., MRI, CT scan, or fluoroscopy) if the preoperative images contain the markers. AI approaches may leverage large datasets to gain new insights from the dataset. The classifier model can improve real-time characterization of various surgical elements, leading to higher surgical success rates. The classifier model can provide the operator (e.g., surgeon, operating room nurse, surgical technician) with information for more accurate placement of virtual markers to overcome imperfections in physical markers. The virtual markers can be tracked, removed, or changed using buttons. The virtual markers can not interfere with the physical movement of surgical instruments during the surgical procedure. The systems and methods herein can overlay markers on the video stream of the surgical procedure as desired (e.g., showing or hiding based on a command from the operator).

[0040]

[0051] Reference will now be made in detail to various embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention and the described embodiments. However, embodiments of the invention may optionally be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments. In the drawings, like reference numbers indicate like or similar steps or components.

[0041]

[0052] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the scope of the claims. When used in describing the embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Also, the term "and / or," as used herein, should be understood to refer to and encompass any and all possible combinations of one or more of the associated listed items.

[0042]

[0053] As used herein, the word "if" is optionally interpreted to mean "when" or "when," or "in response to determining," or "in accordance with determining," or "in response to detecting," depending on the context, that the preceding stated condition is true. Similarly, the phrases "when it is determined that [the preceding stated condition is true]" or "when [the preceding stated condition is true]" or "when [the preceding stated condition is true]" can be optionally interpreted to mean "upon determining," or "in response to determining," or "in accordance with determining," or "upon detecting," or "in response to detecting," that the preceding stated condition is true, depending on the context.

[0043]

[0054] As used herein, unless otherwise specified, the term "about" or "approximately" refers to an acceptable range of error for a particular value as determined by one of ordinary skill in the art, which depends in part on how the value is measured or determined. In certain embodiments, the term "about" or "approximately" means within 1, 2, 3, or 4 standard deviations. In certain embodiments, the term "about" or "approximately" means within 30%, 25%, 20%, 15%, 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, 0.5%, 0.1%, or 0.05% of a given value or range.

[0044]

[0055] As used herein, the words "comprises," "comprising," or any other variation thereof, are intended to cover non-exclusive inclusions, such that a process, method, article, or apparatus that comprises the recited elements not only includes those elements but also includes other elements not expressly recited or that are inherent to such process, method, article, or apparatus.

[0045]

[0056] As used herein, the terms "subject" and "patient" are used interchangeably. As used herein, "subject" refers to a human being. In certain embodiments, the subject has undergone surgery. In certain embodiments, the subject is aged 0 to 6 months, 6 to 12 months, 1 to 5 years, 5 to 10 years, 10 to 15 years, 15 to 20 years, 20 to 25 years, 25 to 30 years, 30 to 35 years, 35 to 40 years, 40 to 45 years, 45 to 50 years, 50 to 55 years, 55 to 60 years, 60 to 65 years, 65 to 70 years, 70 to 75 years, 75 to 80 years, 80 to 85 years, 85 to 90 years, 90 to 95 years, or 95 to 100 years.

[0046]

[0057] Whenever the phrase "at least," "greater than," or "greater than or equal to" is placed before the first number in a series of two or more numbers, the phrase "at least," "greater than," or "greater than or equal to" applies to each and every number in the series. For example, 1, 2, or 3 or more is equivalent to 1 or more, 2 or more, or 3 or more.

[0047]

[0058] Whenever the phrase "not greater than," "less than," or "less than or equal to" precedes the first number in a series of two or more numbers, the phrase "not greater than," "less than," or "less than or equal to" applies to each and every number in the series. For example, 3, 2, or 1 or less is equivalent to 3 or less, 2 or less, or 1 or less.

[0048]

[0059] The terms "surgical AI" or "surgical AI module," as used herein, generally refer to a system, device, or method that uses artificial intelligence algorithms to assist before, during, and / or after a surgical procedure. A surgical AI module may be defined as a combination of input data, machine learning or deep learning algorithms, training data sets, or other data sets.

[0049]

[0060] The term "machine learning," as used herein, may generally refer to computer algorithms that can automatically improve over time. Any description herein of machine learning may also apply to artificial intelligence, or vice versa, or any combination thereof.

[0050]

[0061] As used herein, the terms "continuous," "continuously," or any other variation thereof generally refer to a process that is nearly uninterrupted, or a process that has an acceptable time delay in the context of the process.

[0051]

[0062] The terms "video stream" or "video feed," as used herein, refer to data generated by a digital camera. A video feed can be a series of still images or a moving image.

[0052]

[0063] The terms "site," "organ," "tissue," and "structure," as used herein, may generally refer to anatomical features of the human body. A site may be larger than and may include an organ. An organ may include one or more tissue types and structures. A tissue may refer to a group of cells structurally joined to complete a common function. A structure can refer to a portion of a tissue. In some embodiments, a structure may refer to one or more portions of one or more tissues joined together to create an anatomical feature.

[0053]

[0064] The term "surgical field of view" or "field of view," as used herein, refers to the range of vision captured by an interventional imaging device. Field of view may refer to the range of visual data observable by the human eye and captured by a digital camera.

[0054]

[0065] The term "determining," as used herein, refers to a machine learning or AI algorithm. Decisions may refer to outputs from a system. Decisions may include labeling, classification, prediction, etc.

[0055]

[0066] The term "interventional imaging device," as used herein, generally refers to an imaging device used for medical purposes. An interventional imaging device may refer to an imaging device used in a surgical procedure, such as one or more of an arthroscope, cardioscope, endoscope, or laparoscope, or other similar device. A surgical procedure may, in some embodiments, be a simulation of surgery or other medical procedure.

[0056]

[0067] As used herein, the term "operator" refers to a medical professional involved in a surgical procedure. An operator can be a surgeon, an operating room nurse, or a surgical technician.

[0057]

[0068] The terms "landmark," "any landmark," "virtual landmark," and "fiducial marker," when used interchangeably herein, refer to marks used to guide surgical or other medical procedures.

[0058]

[0069] One aspect of the present invention provides a system for assisting arthroscopic surgery by enabling computer-implemented arbitrary marker placement. The system may include one or more computer processors and one or more non-transitory computer-readable storage media storing instructions operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations, which may include receiving a video stream from an arthroscopic imaging device, receiving one or more sets of coordinates of one or more markers, overlaying the one or more markers on the video stream, and displaying the overlay on one or more displays intraoperatively for use by an operator during arthroscopic surgery. In some embodiments, an operator (e.g., a surgeon) provides one or more sets of coordinates of the one or more markers pre-operatively.

[0059]

[0070] The video stream may be provided by an arthroscopic imaging device during arthroscopic surgery. In some embodiments, the arthroscopic imaging device comprises a digital camera. The video stream may be obtained from a digital camera specialized for arthroscopic surgery. The digital camera may be mounted on a rigid scope suitable for operation in an arthroscopic joint. The rigid scope may include optical fibers to illuminate the surgical field. The digital camera may be mounted on a camera control unit. In some embodiments, the camera control unit is configured to capture digital information generated by the digital camera. In some embodiments, the camera control unit converts the digital information generated by the digital camera into a video stream. In some embodiments, the camera control unit is configured to control a light source. In some embodiments, the camera control unit is configured to record the digital information generated by the digital camera in a memory device. In some embodiments, the memory device used to record the digital information is a local memory device. In some embodiments, the camera control unit is configured to overlay output from one or more computer processors onto the video stream. In some embodiments, the memory device is a remote or cloud-based memory device. The camera control unit may transmit the video stream to one or more computer processors. In some embodiments, there is more than one camera control unit. In some embodiments, there are two, three, four, five, or more camera control units. One or more camera control units may transmit video streams to a computer processor via a network connection or a wired media connection. The video streams may be stereoscopic or monocular. In some embodiments, the system further comprises a display monitor. In some embodiments, the system allows for control from at least one operator during surgery (e.g., to activate or deactivate marking). In some embodiments, the mechanism receives input via a push button, a touch screen device, a foot pedal, a gesture recognition system, or a voice recognition system.

[0060]

[0071] In some embodiments, the one or more sets of coordinates described above are provided intraoperatively using a digital pointer (e.g., a computer mouse or related device) that can mark on an image in a video stream to select a point or site for performing a surgical procedure and / or surgical action (e.g., tissue resection, ablation, etc.). In some embodiments, an operator (e.g., a surgeon) provides the one or more sets of coordinates intraoperatively by indicating a desired location using a standard surgical probe. In some embodiments, after the coordinates of the desired location are selected or indicated, the operator can issue a command to enable the system to register the location. In some embodiments, the system receives a registration command from the operator via a push button, a touchscreen device, a foot pedal, a gesture recognition system, or a voice recognition system.

[0061]

[0072] FIG. 1 is a diagram illustrating a schematic example of a hardware configuration of a system described herein. The exemplary system 100 may include multiple inputs. The multiple inputs may include a video stream input 101, an operator (e.g., surgeon) input 102, and one or more pre-operative imaging inputs. The pre-operative imaging input may include a fluoroscopic imaging input 103 and a medical data system (e.g., radiology imaging, such as an MRI or CT scan) input 104. In some embodiments, each of the multiple inputs is connected to a corresponding interface. For example, the video stream input 101 may be connected to a camera control unit (CCU) 111, the operator input 102 may be connected to a control interface 112, the fluoroscopic imaging input 103 may be connected to a fluoroscopy interface 113, or the medical data system input 104 may be connected to a medical data system (e.g., radiology imaging) interface 114. Each of the interfaces may be configured to receive input from its corresponding input. System 100 may support other external interfaces to receive inputs in various modalities from surgeons, clinical data systems, surgical equipment, etc. Inputs received by the interfaces may then be sent to a processing unit and processed using an artificial intelligence (AI) pipeline. In some embodiments, the processing unit may include a central processing unit (CPU) 106, a graphics processing unit (GPU) 107, or both. In some embodiments, a CPU or GPU includes multiple CPUs or GPUs. A CPU or GPU may be connected to multiple interfaces via media connectors (e.g., HDMI cables, DVI connectors). A CPU or GPU may be connected to multiple interfaces (e.g., surgical video cameras) via network connections (e.g., TCP / IP), thereby providing greater flexibility with fewer wired connections. In some embodiments, latency in video processing and playback may be higher when the connection is over a network compared to a media connector.In some embodiments, the network connection may be a local network connection. A local network including a predefined set of devices (e.g., devices used in a surgical procedure) may be isolated. For real-time (e.g., during a surgical procedure) feedback, lower latency may be more desirable. In some embodiments, a system configuration with higher latency may be usable for training purposes (e.g., mock surgery). The AI ​​pipeline may comprise one or more machine learning or AI modules, including one or more computer vision (CV) modules. In some embodiments, the AI ​​and CV modules are supported by a video and AI inference pipeline (VAIP) 105. In some embodiments, the VAIP 105 supports the AI ​​and CV modules and manages the flow of control and information between the modules. VAI. The P105 may include a configuration file containing instructions for connecting and managing its flows. The VAIP 105 may support execution of AI algorithms on the GPU 107. The VAIP 105 may also support a direct media interface (e.g., HDMI or DVI). One or more outputs may be generated from the multiple inputs, including the sign location 120 and one or more features identified from the multiple inputs 109, that are processed by the AI ​​pipeline. The one or more outputs may be overlaid on the video stream input 101 to generate the output 130. In some embodiments, the system 100 includes a display monitor. In some embodiments, the output 130 is displayed on a display monitor (e.g., a monitor, a television (TV)). In some embodiments, the system includes a display device. In some embodiments, the sign location 120 and the input 109 are sent back to the CCU and overlaid on the video stream to generate the output 130.

[0062]

[0073] In some embodiments, the arthroscope may generate continuous images (e.g., a video feed) at a rate of at least about 10 frames per second (fps). In some embodiments, there is latency in the system, which is the time required to receive an image (e.g., a video feed) and provide an overlay (e.g., a processed image). In some other cases, two consecutive frames from a video stream (e.g., video stream input 301) may be generated at a rate of 1 / fps (frames per second). In some embodiments, the latency in the system is at most 1 / fps. The latency of the system may be less than the reciprocal of the rate of continuous image generation by the surgical camera. For example, when the input signal is streamed at 20 frames per second, the latency may be 1 / 20 (1 / fps) or 50 ms or less. In some embodiments, the latency may include periods of downtime in the system.

[0063]

[0074] In some embodiments, the operations may further include identifying and labeling one or more elements in the video stream using at least one of the trained computer algorithms, the one or more elements including one or more of an anatomical structure, a surgical instrument, a procedure or action, or a pathology. In some embodiments, identifying and labeling the one or more elements in the video stream includes using one or more AI modules. In some embodiments, the one or more AI modules may include one or more modules for video stream decomposition, instrument recognition, anatomical structure recognition, instrument tracking, gesture recognition, landmark registration, or anatomical structure and landmark tracking. In some embodiments, the system recommends one or more markers based at least in part on the identified elements. In some embodiments, the system recommends one or more markers based at least in part on the identified elements.

[0064]

[0075] 2A and 2B are diagrams illustrating examples of marker placement on a model femoral condyle. In some embodiments, as shown in FIG. 2A, an operator (e.g., a surgeon) indicates the desired location of the marker using a standard surgical probe 201. The operator may then activate a registration command to the system, which registers the desired location. The location of the marker may be visualized on a screen displaying a video stream of the surgical procedure with a dot 202 (e.g., a blue dot). In some embodiments, the system saves the location of the marker and tracks the marker throughout the surgical procedure. The visualized marker dot can be displayed on the screen or removed from the screen at any time by the operator. In some embodiments, the marker is a single isolated dot. In some embodiments, the marker is multiple isolated dots 202, as shown in FIG. 2B. In some embodiments, the marker is a virtual arbitrary pattern or predefined shape. In some embodiments, the location of the marker can be indicated and / or selected preoperatively. The marker can be used to indicate implant location or anchor location during arthroscopic surgery. In some embodiments, the arthroscopic surgery is used in rotator cuff tear repair. In some embodiments, the arthroscopic surgery is used in cruciate ligament repair in knee surgery. In some embodiments, the arthroscopic surgery is used in graft placement. In some embodiments, the arthroscopic surgery is used in decompression surgery. In some embodiments, the decompression surgery involves removal or reshaping of bony structures to relieve pain. In some embodiments, the arthroscopic shoulder surgery involves placement of a graft.

[0065]

[0076] In some embodiments, arthroscopic surgery includes the removal or resection of inflamed tissue and / or fissured tendons. In some embodiments, arthroscopic surgery is used to remove or resect one or more inflamed tissues. In some embodiments, arthroscopic surgery is used to remove or resect one or more fissured tendons, and the video stream is monocular. For example, other imaging methods, such as radiographic imaging or fluoroscopic imaging, can be used to identify the location of marker placement. Images obtained from these imaging methods can be provided to the system during surgery and overlaid on the video stream. In some embodiments, the system indicates potential anatomical structures or lesions, for example, the operator (e.g., surgeon) may protect the ureter, which may not be exposed during the surgical procedure, because the system visualizes its location. For example, the system may incorporate information from an external system, such as a fluoroscopic imaging system. The marker may have a vascular form that is then visualized with a fluorescent dye that is imaged using the fluoroscopic imaging system. The system may continue to retain and track the vascularity during the procedure (e.g., arthroscopic surgery).

[0066]

[0077] In some embodiments, system 100 operates on stored video content. Video recordings of arthroscopic procedures can be played back and transmitted to an interface. The system may then overlay any markers onto the video stream as described herein. In some embodiments, marker placement on surgical recordings is used for training purposes. In some embodiments, the system operates on stereoscopic video streams. In some embodiments, the system may be used during robotic arthroscopic procedures (e.g., surgical procedures). In some embodiments, the view of the display may be changed. In some embodiments, by changing the view, markers overlaid on the video stream can be excluded from being displayed. In some embodiments, the operator can choose to make markers invisible temporarily during arthroscopic procedures or throughout the entire arthroscopic procedure. The operator may return the view to a previous view. The operator may identify a new set of coordinates for the markers. The new markers may be overlaid on the video stream and displayed. Multiple markers may be selected to be displayed simultaneously or one at a time. In some embodiments, the change in view is automatic. In some embodiments, the change of view is triggered by the AI ​​pipeline identifying an anatomical structure or a lesion.

[0067]

[0078] In some embodiments, the set of landmark coordinates is provided by the operator during surgery. Figure 3 is a diagram illustrating an exemplary flowchart of a landmark placement system 300. The system may include multiple modules that operate on a video stream input 301 generated by an arthroscopic camera and input 302 received from an operator (e.g., a surgeon). In some embodiments, the video stream input 301 is processed by a video stream decomposition module 303 that includes a CV algorithm that decomposes the video stream into a series of images. The series of images may be stored in a memory device. One or more images from the series of images may be provided to one or more downstream components, including an instrument recognition module 304 or an anatomy recognition module 305. In some embodiments, the video stream decomposition module 303 outputs an image of the surgical field of view.

[0068]

[0079] In some embodiments, the instrument recognition module 304 uses an AI network to recognize surgical instruments within the field of view. Non-limiting examples of AI networks used in the instrument recognition module 304 may include Mask R-CNN, UNET, ResNET, YOLO, YOLO-2, or any combination thereof. In some embodiments, the AI ​​network is trained to recognize surgical instruments of interest using machine learning training, including training techniques specific to its architecture. In some embodiments, the trained AI network detects the presence of a surgical instrument in an image and outputs a mask. This mask may be a set of pixels extracted from the input image that provides a high-precision outline of the surgical instrument. In some embodiments, the AI ​​network outputs a box (e.g., a rectangular area) within which the instrument is detected or displayed.

[0069]

[0080] In some embodiments, the anatomy recognition module 305 uses an AI network to recognize anatomical structures within the field of view. Non-limiting examples of AI networks used in the anatomy recognition module 305 may include Mask R-CNN, UNET, ResNET, YOLO, YOLO-2, or any combination thereof. In some embodiments, the AI ​​network is trained to recognize the anatomical structure of interest using a training technique specific to its architecture. In some embodiments, the trained AI network recognizes the anatomical structure when seen within the field of view. In some embodiments, the trained network outputs a pixel mask that may show a highly accurate outline of the recognized anatomical structure. In some embodiments, the trained network outputs a box (e.g., a rectangular area) within which the instrument is detected or displayed.

[0070]

[0081] In some embodiments, output from the instrument recognition module 304 is provided to the instrument tracking module 306. In some embodiments, the instrument tracking module 306 tracks the movement of one or more instruments identified by the instrument recognition module 304. In some embodiments, the instrument position (e.g., the instantaneous position of the instrument) may be stored in memory (e.g., a buffer). In some embodiments, the instrument tracking module 306 uses a CV algorithm to calculate the instrument's velocity and acceleration and stores those values ​​in memory. This data may be stored as a fixed-length array. In some embodiments, the array is stored in chronological order of acquisition. In some embodiments, the array is stored in reverse chronological order. The array may have a fixed length, and as new data is added, older entries may be removed from the array and memory buffer. In some embodiments, the addition of a new entry causes the oldest entry to be removed from the array. The output of the instrument tracking module 306 may include a mask of the recognized instrument along with an array of instrument velocity and / or acceleration. The instrument tracking module 306 may provide one or more instrument positions or an array of positions to a gesture recognition module 307 and a landmark registration module 308 (eg, a BluDot point registration module).

[0071]

[0082] In some embodiments, the gesture recognition module 307 uses an AI network with memory (e.g., a recurrent neural network (RNN)) to interpret the instrument movements. In some embodiments, the AI ​​network is trained to recognize specific instruments and / or identify specific movement patterns. For example, a tap may involve the instrument moving in a particular manner relative to the background anatomical structure. In some embodiments, the surgeon can indicate the location of any marker by using a predefined gesture with the surgical instrument. Non-limiting examples of gestures may include a tap, double tap, triple tap, and a shake (e.g., moving the instrument from left to right). In some embodiments, the gesture recognition module 307 outputs a label of the gesture made by the operator using the instrument. In some embodiments, the gesture recognition module 307 recognizes the gesture made by the operator and outputs a label of the recognized gesture. A label for the name of the gesture is generated and provided to a downstream component, which may be the sign registration module 308 .

[0072]

[0083] In some embodiments, the landmark registration module 308 receives one or more inputs from the instrument tracking module 306 and / or the gesture recognition module 307, as described herein. In some embodiments, input from the gesture recognition module 307 instructs the landmark registration module 308 that a gesture from the operator be recognized. This gesture may then be mapped to an action. In some embodiments, this mapping is configured pre-operatively and loaded from a database when the system is initialized. Non-limiting examples of actions mapped by the landmark registration module 308 may include initiate, replace, or clear all. In some embodiments, the landmark registration module 308 may initiate to assign unique identifiers to the landmarks (e.g., BluDots). Actions including a command to clear one or more landmarks or a command to clear all may activate the landmark registration module 308 to update the list of one or more landmarks. The initiate action may trigger the landmark registration module 308 to provide the position of the instrument to the anatomy and landmark tracking component 309. A replace action may trigger the landmark registration module 308 to replace data associated with one or more landmark locations with new landmark locations. A clear all action may trigger the landmark registration module 308 to erase any landmarks that are displayed or stored in memory. In some embodiments, the landmark registration module 308 receives direct input from the operator to place, replace, or erase a landmark. This direct input may be provided using digital or mechanical buttons, for example, by pressing a foot pedal or a dedicated button on the arthroscopic device. In some embodiments, the CCU communicates the direct input to the VAIP described herein through a specialized interface. For example, the specialized interface may include gestures mapped to actions customized for the operator.

[0073]

[0084] In some embodiments, the landmark registration module 308 distinguishes between techniques in which landmarks are specified (e.g., pre-operatively, intra-operatively via gestures, or intra-operatively via direct commands from the operator) for purposes of rendering and / or recall. For example, landmarks obtained from pre-operative planning may be exempt from deletion. In some embodiments, the landmark registration module 308 provides a set of coordinates of instruments identified by the instrument recognition module 304 in an image of the surgical field. The updated list in the landmark registration module 308 may be passed to the downstream anatomical structure and landmark tracking component 309, which may stop tracking and displaying landmarks that may be set for deletion. For example, to assist the operator (e.g., a surgeon) in identifying an artery or vein, a fluorescent dye may be injected into the blood vessel. In some embodiments, the surgical procedure is performed near a highly vascularized area where injuring the artery or vein could have serious consequences. Images of the identified artery or vein (e.g., using the dye) may be received and overlaid as landmarks on the surgical video by the VAIP. A confidence level may be calculated on a near-continuous basis, representing the VAIP's confidence in the accuracy of tracking and recall of landmarks (e.g., identified arteries or veins). In some embodiments, the confidence level may be below a threshold, which may be approximately 80%, 70%, 60%, 50%, 40%, 30%, 20%, 10%, or less. This threshold may be set by the operator. In some embodiments, a change in confidence level is due to a change in the surgical field (e.g., when surgery is performed, the anatomical structures may change). The system may then indicate that the confidence level of the landmark tracking has decreased. The operator may enhance the landmark, for example, by injecting a dye into the vessel. The system may then replace the previously identified landmark with the newly identified landmark and overlay it on the surgical image (e.g., video stream).

[0074]

[0085] In some embodiments, the anatomy and landmark tracking component 309 receives masks of anatomical structures from the anatomy recognition module 305 and / or masks of identified instruments from the instrument recognition module 304 on a near-continuous basis. In some embodiments, the anatomy and landmark tracking component 309 receives input from the landmark registration module 308 indicating an action for the anatomy and landmark tracking component 309 to perform a series of operations. The series of operations may include determining an overlay of the instrument and the anatomical structures from the mask to determine on which anatomical structure the instrument is being placed or held. The coordinates of the instrument and the anatomical structures may be used to identify the overlay of the instrument with the anatomical structures. The series of operations may further include extracting features to identify the location of landmarks (e.g., BluDots) relative to the location of one or more anatomical structures. In some embodiments, the features include details on the image that may include vascularity patterns, tissue edges, or locally unique patches of pixels. Using this feature, the system can stabilize the landmarks (e.g., BluDots) against camera movement relative to the anatomy (as also shown in FIGS. 10A-10B). In some embodiments, if the instrument is moving independently of the anatomy, the feature can include a point on the instrument in the surgical field of view. The feature on the instrument can then be filtered out in the anatomy and landmark tracking component 309 to stabilize the landmark against instrument movement (as also shown in FIGS. 8 and 9).

[0075]

[0086] In some embodiments, the markers are initialized and assigned to the initial position of the instrument. In some embodiments, the anatomy and landmark tracking component 309 identifies changes in the position of the anatomical features and reassigns the markers' positions. In some embodiments, the anatomical features are modeled as slightly deformable solids, and appropriate computational techniques are used to track the anatomical features. In some embodiments, the anatomy and landmark tracking component 309 continuously acquires the features and tracks their movement on an augmented canvas. The augmented canvas may include a surgical field in various images acquired from a video stream, which may be connected together to create a larger field of view. In some embodiments, using the features described herein, the system tracks the markers with a high degree of certainty, even if the markers or underlying anatomical structures move out of the camera's field of view. In some embodiments, during a surgical procedure, the operator may move the camera away from the location of the markers and surrounding tissue, causing the operator to lose sight of the markers. In some embodiments, the markers' positions need to be reacquired when the operator reenters the normal field of view. In some cases, the anatomical features are initially identified as described above, excluding any instruments in the field of view. One or more features may be identified to identify the location of the marker according to a previously recognized anatomical structure.

[0076]

[0087] For example, one or more feature points may be identified that may be separated by the anatomical structure in which they appear. When the surgeon re-enters the surgical field analyzed in the previous image, the anatomy recognition module may recognize the previously processed image. Upon matching the new coordinates of the feature points in the current image with the coordinates of the feature points in the previously processed image, a marker may be placed at that location. The marker's position may be reset based on the feature points in the current image along with the previously identified feature points. This feature point matching process may be repeated to improve the accuracy of the marker placement. This process may be performed using parallel computing (e.g., GPU). The system may discard feature points identified in the previously processed image and replace them with feature points identified in the current image. The process described herein may be performed using the anatomy and landmark tracking module 309, the out-of-site recognition module 310, and the anatomy reacquisition module 311.

[0077]

[0088] 11A-11B illustrate examples of feature detection. For example, multiple features (or feature points) 1101 may be detected on an instrument 1100 (shown as green dots 1101 in FIGS. 11A-11B ) and can be distinguished from a set of features 1102 (shown as red dots 1102 in FIGS. 11A-11B ) recognized on an anatomical structure 1103. In some embodiments, during a surgical procedure, the surgical field of view may be changed. For example, the procedure may include soft tissue debridement, which may change the field of view. Once an instrument is recognized and tracked (e.g., in real time), feature points detected on the instrument may be erased. In some embodiments, feature points detected on the anatomical structure may be used to track landmarks. This may improve stabilization of the landmarks against instrument movement that may occlude them. FIG. 8 illustrates an example of instrument occlusion being ignored when overlaying landmarks on a video or image. In some embodiments, bleeding or bodily fluids may change the field of view. In some embodiments, operations in the anatomy and landmark tracking component 309 include continuously acquiring features from the field of view and discarding features that are missing in successive images to stabilize the landmarks against changes in the field of view due to actions performed in the procedure. The features may be acquired relative to the anatomy as a reference. In some embodiments, the anatomy and landmark tracking component 309 comprises an out-of-situ recognition module 310 and an anatomy reacquisition module 311. In some embodiments, the anatomy and landmark tracking component 309 updates the location of the landmarks based on features in the observable portion of the anatomy. In some embodiments, the field of view excluding the anatomy or landmarks may be shifted as described herein. If the camera pans back to the location of the landmarks, the anatomy and landmark tracking component 309 may increase the confidence in the location of the landmarks by using the out-of-situ recognition module 310 and the anatomy reacquisition module 311 as described herein. The output of the anatomy and landmark tracking component 309 includes the location of the landmarks (e.g., BluDots) within the field of view and / or the frame or range of the image being processed.The location of this marker is transmitted to module 320. This marker may then be overlaid on output video stream 330. An example of output video stream 330 is shown in FIG. 8. In some embodiments, the surgical field of view is about 3 centimeters (cm) to about 6 cm. In some embodiments, the range of movement of the surgical camera (e.g., arthroscope) is about 3 cm to about 6 cm. In some embodiments, the range over which stabilization can be achieved is similar to the range of movement of the surgical camera, which is about 3 cm to about 6 cm. In some embodiments, the accuracy of stabilization of the marker relative to changes in field of view is about 1 millimeter (mm) to about 3 mm.

[0078]

[0089] In some embodiments, the output from the landmark location module 320 is overlaid onto the input video stream from module 301 in a video blending module 312. The output from the video blending module 312 may be displayed on an output video stream 330 (e.g., using a screen, monitor, TV, laptop screen, etc.). The output from 320 may be sent to a camera control unit, scaled, and overlaid onto the video stream of the procedure.

[0079]

[0090] Another aspect of the present invention provides a system for assisting arthroscopic surgery by enabling computer-implemented arbitrary marker placement using radiological imaging, the system comprising one or more computer processors and one or more non-transitory computer-readable storage media storing instructions operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations, which may include receiving a radiological image of a subject, generating a 3D representation of the radiological image, identifying anatomical structures in the radiological image using a trained machine learning algorithm, receiving marker locations from an operator, overlaying the markers onto the 3D representation of the radiological image, and displaying the overlay on a display device used by the operator.

[0080]

[0091] In some embodiments, an operator identifies or sets the location of the marker during the pre-operative surgical planning stage. In some embodiments, the marker may be set by an operator (e.g., a surgeon) on a radiographic image acquired from the subject. The marker may then be provided to a marker registration module similar to marker registration module 308 of FIG. 3. This allows the operator to hide or show the marker during the pre-operative surgical planning stage.

[0081]

[0092] FIG. 4 is a diagram illustrating an example workflow for marker placement using radiological imaging. As shown in FIG. 4, several modules may be added to the system 300 shown in FIG. 3 that enable pre-operative medical imaging data (e.g., radiological imaging data such as an MRI or CT scan) of a subject 401 to be used to position markers on a video stream (e.g., from the video stream input 301) of an arthroscopic surgery. In some embodiments, a pre-operative medical imaging import module 402 interfaces with an external repository to import the pre-operative medical imaging data 401. The pre-operative medical imaging data may include radiological images of the subject. In some embodiments, the radiological images are from and associated with a joint or other bony structure of the subject, such as the shoulder, knee, hip, ankle, or spine. In various embodiments, the radiological images may be generated using one or more of fluoroscopy, magnetic resonance imaging (MRI), x-ray, computed tomography (CT) scan, positron emission tomography (PET) scan, or ultrasound. In some embodiments, the pre-operative medical images include: In some embodiments, MRI or CT scan images are acquired from a subject for arthroscopic surgery (e.g., knee surgery, shoulder surgery, or hip surgery). The MRI or CT scan images may include images of the subject's knee or shoulder. In some embodiments, the MRI, CT scan, or other images are acquired from a repository in a standard format (e.g., DICOM). In some embodiments, the pre-operative medical imaging acquisition module 402 includes an application programming interface (API) layer that abstracts external systems associated with the imaging system (e.g., MRI, CT scan, or PET imaging) from the system 400. In some embodiments, the repository of images includes images of landmarks. In some embodiments, images of landmarks from the repository are placed by an operator (e.g., a surgeon) on the MRI or CT scan images of the subject. Output from the pre-operative medical imaging acquisition module 402 may be provided to a three-dimensional (3D) image reconstruction module 403.In some embodiments, the 3D image reconstruction module 403 transforms volumetric data in images from the pre-operative medical imaging capture module 402, including one or more slices of a two-dimensional (2D) image, and converts the data into a 3D image in computer memory. In some embodiments, the coordinates of landmarks set by the operator are mapped onto the 3D image. In some embodiments, the 3D image reconstruction module 403 may generate a multidimensional array including a 3D representation of the radiological image and the landmarks mapped to the image. In some embodiments, the output from the 3D image reconstruction module 403 may be integrated with a mask generated by the anatomical structure recognition module 305 using the mapping module 404. In some embodiments, 404 comprises a trained AI network that recognizes anatomical structures in images acquired before surgery (e.g., MRI or CT scan images). In some embodiments, the anatomical structures may include bone structures. In some embodiments, the anatomical structures may include tendons. In some embodiments, anatomical structures recognized in an image (e.g., an MRI or CT scan image) may be masked (e.g., labeled) in the mapping module 404 using the same labeling system used in the anatomy recognition module 305. The anatomical structures recognized in the mapping module 404 may then be aligned with the anatomical structures recognized in the anatomy recognition module 305. In some embodiments, the labels identified in the 3D image reconstruction module 403 are mapped onto the anatomical structures recognized in the anatomy recognition module 305. The mapping may be provided to landmark registration module 308. As previously described herein, landmark registration module 308 may process and transmit landmark and anatomical structure information to be overlaid on the surgical video stream. In some embodiments, 320 is adapted for surgical camera motion. In some embodiments, if similar structures are identified from pre-operative medical images (e.g., MRI or CT scan images) and from images from the surgical video stream, the two anatomical structures are aligned (e.g., in mapping module 404) and the frames are corrected for any image discrepancies associated with surgical camera motion. In some embodiments, each frame from the video stream is corrected for surgical camera motion.

[0082]

[0093] In some embodiments, the system includes a recommender module that can recommend marker placement based at least in part on the surgical context. FIG. 5 illustrates an overview of an exemplary workflow of a system for recommending marker placement. The system 500 illustrated in FIG. 5 may include the system 400 and a recommender module 501 that makes recommendations for marker placement. In some embodiments, the system 500 is a surgical decision support system. In some embodiments, 501 receives an anatomical feature mask or anatomy mask from the mapping module 404. In some embodiments, based on the received anatomical feature mask or anatomy mask, 501 identifies the surgical context (e.g., an anatomical site or a portal).

[0083]

[0094] In some embodiments, based at least on the identified context, 501 recommends marker placement. Non-limiting examples of recommendations from 501 may include femoral and tibial tunnel placement in anterior cruciate ligament (ACL) surgery or anchor placement in rotator cuff tear repair. In some embodiments, 501 recommends marker locations based at least in part on the locations of markers in pre-operative medical images of the subject identified by the 3D image reconstruction module 403 and / or mapping module 404. The recommended markers and / or marker locations are sent to the marker registration module 308, processed as described herein, overlaid on a video stream of the surgery, and displayed on a display device (e.g., a monitor). In some embodiments, images acquired pre-operatively (e.g., MRI, CT scan, etc.) may be combined with the instrument tracking module 306 and processed as described herein to provide the information necessary to estimate the size or location of the markers. For example, an imaging modality (e.g., CT scan, MRI) may generate images including anatomical features that can be recognized by a system as described herein. The images may further include the location of the markers. In some embodiments, these images are three-dimensional images including voxels, where a voxel (volumetric pixel) may represent a volume in physical space. Thus, the location of the markers may be identified on the surgical view image by matching identified anatomical structures (e.g., by recognizing anatomical features on the pre-operative and surgical images). The location of the markers may further be identified based in part on measurements by measuring the size of the anatomical structures based in part on the size of the voxels in the pre-operative images. The measurements may be used to place the markers at the locations of the anatomical structures on the surgical image that correspond to the locations of the markers identified on the pre-operative images (e.g., CT scan, MRI).

[0084]

[0095] In some embodiments, the system is configured to process video streams from a stereoscopic surgical camera (e.g., a stereoscopic arthroscope). FIG. 6 shows a schematic flow chart of an exemplary system for processing a stereoscopic video stream (e.g., 3D video). In some embodiments, system 600 comprises the components in system 500 and multiple modules for processing a stereoscopic video input or stream 601. In some embodiments, the stereoscopic video input or stream 601 is decomposed by a stereoscopic video decomposition module 602 to generate images from the stereoscopic video input or stream 601. The images are first processed. In some embodiments, the stereoscopic video decomposition module 602 provides images from the stereoscopic video input or stream 601 to the instrument recognition module 603 and / or the anatomy recognition module 605. The modules of the instrument recognition module 603 and the anatomy recognition module 605 are similar to the instrument recognition module 304 and the anatomy recognition module 305, respectively. In some embodiments, the instrument recognition module 603 and the anatomy recognition module 605 can process surgical views in images with view shifts due to parallax in the stereoscopic video stream. The stereoscopic video stream or images from the stereoscopic video stream may include two channels (e.g., right side, left side). Parallax may include a misalignment or difference in the visual position of an object viewed along two different lines of sight. In some embodiments, the instrument recognition module 603 provides one or more masks for surgical instruments to the instrument localization module 604. In some embodiments, the instrument recognition module 603 provides one or more masks for anatomical structures to the anatomy localization module 606. In some embodiments, the instrument localization module 604 uses the difference in scale of a given instrument to locate the instrument in 3D space. In some embodiments, the instrument localization module 604 comprises an instrument recognition algorithm applied to two channels of images from a stereoscopic video stream (e.g., a binocular video stream). Landmarks may be registered using surgical instruments as described herein. In some embodiments, the landmarks appear in 3D space when viewed using a 3D viewing device (e.g., a binocular viewer). In some embodiments, the anatomy recognition module 605 provides one or more masks for anatomical structures to the anatomy localization module 606. In some embodiments, the anatomy localization module 606 processes the anatomy masks in the two channels of images from the stereoscopic video stream and generates a mask that can be visualized in a 3D viewer based on the spatial information of the anatomy provided by the anatomy recognition module 605.In some embodiments, the markers (e.g., BluDots) are rendered into the field of view such that the markers are independently positioned in the left and right channels of the stereoscopic display channels. In some embodiments, the markers are offset (e.g., laterally) to create the perception of depth. The output video stream 330 may include markers that are visualized or displayed in 3D overlaid (e.g., added) on anatomical structures in the video stream of the surgical procedure.

[0085]

[0096] In some embodiments, the object (e.g., a probe or surgical instrument) 801 may be placed in the same position as the landmark 802. The system identifies the instrument 801 as described herein and compensates for any occlusion (FIG. 8). The landmark 802 may also move to correspond to its position as the anatomical structure 803 moves. FIG. 9 illustrates another example of a landmark 802 being removed from an object (e.g., an instrument) 801 that may occlude the landmark 802. FIGS. 10A and 10B illustrate the operation of the system in stabilizing a landmark 1001 (e.g., a BluDot) with respect to movement of the camera relative to the anatomical structure. The camera may move relative to the anatomical structure 1002, but the landmark 1001 may remain at the marked location on the anatomical structure. That is, the landmark 1001 may move with the anatomical structure 1002 as the camera changes position. Computer Systems

[0097] Various embodiments of the present invention also provide computer systems programmed to implement the methods of the present invention. Accordingly, details of one or more embodiments of such computer systems are described below. Figure 7 illustrates a computer system 701 that is programmed or otherwise configured to perform one or more functions or operations of the methods of the present invention. The computer system 701 may, for example, receive images from an interventional imaging device, identify features in the images using image recognition algorithms, overlay the features on a video feed on a display device, and provide recommendations to an operator based on the identified features in the images. The computer system 701 may control various aspects of the present invention, such as providing a user with an electronic device, making an action or suggestion, etc. The computer system 701 may be a user's electronic device or a computer system located remotely relative to the electronic device. The electronic device may be a mobile electronic device.

[0086]

[0098] The computer system 701 includes a central processing unit (CPU, herein referred to as a "processor" and "computer processor") 705, which may be a single-core or multi-core processor or multiple processors for parallel processing. The computer system 701 also includes memory or storage 710 (e.g., random access memory, read-only memory, flash memory), an electronic storage unit 715 (e.g., a hard disk), a communication interface 720 (e.g., a network adapter) for communicating with one or more other systems, and peripheral devices 725, such as cache, other memory, data storage, and / or electronic display adapters. The memory 710, storage unit 715, interface 720, and peripheral devices 725 are in communication with the CPU 705 via a communication bus (solid lines), such as a motherboard. The storage unit 715 may be a data storage unit (or data repository) for storing data. The computer system 701 may be operatively coupled to a computer network ("network") 730 using the communication interface 720. Network 730 may be the Internet, an Internet and / or extranet, or an intranet and / or extranet in communication with the Internet. In some embodiments, network 730 is a telecommunications network and / or a data network. Network 730 may include one or more computer servers that may enable distributed computing, such as cloud computing. In some embodiments, network 730 may implement a peer-to-peer network that may enable devices coupled to computer system 701 to act as clients or servers.

[0087]

[0099] The CPU 705 may execute a sequence of machine-readable instructions, which may be embodied in a program or software. The instructions may be stored in a storage location, such as the memory 710. The instructions may be issued to the CPU 705, which may then be programmed or otherwise configured to implement the methods of the present invention. Examples of operations performed by the CPU 705 may include fetch, decode, execute, and write-back.

[0088]

[0100] The CPU 705 may be part of a circuit such as an integrated circuit. One or more other components may be included in the circuit, hi some embodiments, the circuit is an application specific integrated circuit (ASIC).

[0089]

[0101] The storage unit 715 stores drivers, libraries, and stored programs. Storage unit 715 may store files such as, for example, user preferences and user programs. In some embodiments, computer system 701 may include one or more additional data storage units external to computer system 701, such as located on a remote server in communication with computer system 701 via an intranet or the Internet.

[0090]

[0102] The computer system 701 communicates with one or more computers via a network 730. For example, computer system 701 may communicate with a user's remote computer system (e.g., a portable computer, a tablet, a smart display device, a smart TV, etc.). Examples of remote computer systems include personal computers (e.g., portable PCs), slate PCs, or tablet PCs (e.g., Apple® iPads, Samsung® iPads, etc.). A user may access the computer system 701 via a network 730, including a mobile device such as a Galaxy Tab, a phone, a smartphone (e.g., an Apple® iPhone, an Android-enabled device, a Blackberry®), or a personal digital assistant.

[0091]

[0103] The methods described herein involve the use of electronic storage locations in a computer system 701, The instructions may be implemented via machine (e.g., computer processor) executable code stored, for example, on memory 710 or electronic storage unit 715. The machine-executable or machine-readable code may be provided in the form of software. In use, the code may be executed by processor 705. In some embodiments, the code may be retrieved from storage unit 715 and stored in memory 710 for access by processor 705. In some cases, electronic storage unit 715 may be excluded, and the machine-executable instructions are stored in memory 710.

[0092]

[0104] The code can be used in conjunction with a machine having a processor adapted to execute the code. The code can be pre-compiled and configured for use in, or can be compiled on the fly. The code can be supplied in a programming language that can be selected to allow the code to be executed in a pre-compiled or as-compiled manner.

[0093]

[0105] The systems and methods provided herein, such as computer system 701 Aspects of the technology can be embodied in programming. Various aspects of the technology can be thought of as a typical "product" or "article of manufacture" in the form of machine (or processor) executable code and / or associated data carried on or embodied in some type of machine-readable medium. In various embodiments, the machine-executable code can be stored in an electronic storage unit, such as memory (e.g., read-only memory, random-access memory, flash memory) or a hard disk. "Storage" type media can include any or all of the tangible memory of a computer, processor, etc., various semiconductor memories, tape drives, disk drives, etc., that may provide non-transitory storage for software programming at any given time, or its associated modules. All or portions of the software can, in some cases, be communicated via the Internet or various other telecommunications networks (including wireless and wired networks). Such communication may enable, for example, loading of the software from one computer or processor to another, e.g., loading from an administrative server or host computer into an application server computer platform. Thus, other types of media that may bear software elements include light waves, radio waves, and electromagnetic waves, such as those used across physical interfaces between local devices, over wired and optical cable networks, and over various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., may also be considered media that bear software. As used herein, unless limited to non-transitory tangible "storage" media, terms such as computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

[0094]

[0106] Thus, machine-readable media such as computer-executable code may be used in conjunction with tangible storage media, The storage medium may take many forms, including but not limited to a carrier wave medium, or a physical transmission medium. Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices in any computer, such as a computer that may be used to implement a database, etc., as shown in the drawings. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media are storage media that are stored in a computer Examples of computer-readable media include coaxial cables, i.e., copper wire, and fiber optics, including the wiring that comprises a bus within a computer system. Carrier-wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic medium, CD-ROMs, DVDs, or DVD-ROMs, any other optical medium, punched cards, paper tape, any other physical storage medium with perforated patterns, RAMs, ROMs, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or memory cartridges, carrier waves carrying data or instructions, cables or links carrying such carrier waves, or any other medium from which a computer may read programming code and / or data. Many of these forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0095]

[0107] The computer system 701 may, for example, monitor the video feed from the arthroscope. An electronic display 735 may be included or in communication with the electronic display 735, which includes a user interface (UI) 740, to provide an overlay of identified features or to provide recommendations to the operator during the course of a surgical procedure. Examples of UIs include, but are not limited to, graphical user interfaces (GUIs) and web-based user interfaces.

[0096]

[0108] In various embodiments, the methods and systems of the present invention utilize one or more algorithms. The algorithms may be implemented via software that, when executed by the central processing unit 705, may include, for example, receiving images from an interventional imaging device, identifying features in the images using image recognition algorithms, overlaying the features onto a video feed on a display device, and making recommendations or suggestions to the operator based on the identified features in the images.

[0097]

[0109] While preferred embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that It should be apparent that such embodiments are provided by way of example only. The present invention is not intended to be limited by the specific examples provided herein. While the present invention has been described with reference to the foregoing specification, the description and illustration of the embodiments herein are not intended to be construed in a limiting sense. Numerous variations, changes, and substitutions may occur to those skilled in the art without departing from the present invention. Furthermore, it should be understood that all aspects of the present invention are not limited to the specific depictions, configurations, or relative proportions set forth herein, which depend upon a variety of conditions and variables. It should be understood, however, that various alternatives to the embodiments of the present invention described herein may be employed in practicing the present invention. It should therefore be understood that the present invention is intended to encompass various alternatives, modifications, variations, or equivalents of the embodiments of the present invention described herein.

[0098]

[0110] Also, certain implementations may be used to form many further embodiments within the scope of the present invention. Elements, features, or operations from the embodiments may be readily recombined or substituted for one or more elements, features, or operations from other embodiments. Furthermore, elements shown or described as being combined with other elements may exist as standalone elements in various embodiments. Furthermore, embodiments of the present invention specifically contemplate the exclusion of elements, operations, or features, etc., if the element, operation, or feature is explicitly recited. Accordingly, the scope of the present invention is not limited to the details of the described embodiments, but rather is limited only by the scope of the appended claims.

Claims

1. 1. A system for assisting minimally invasive procedures by enabling computer-implemented arbitrary marker placement, comprising: one or more computer processors; and one or more non-transitory computer-readable storage media storing instructions operable, when executed by the one or more computer processors, to cause the one or more computer processors to perform operations; The operation is receiving a video stream from an arthroscopic imaging device; receiving one or more sets of coordinates of one or more landmarks; overlaying the one or more signs over the video stream; displaying said overlay on one or more displays intra-operatively for use by at least one operator during arthroscopic surgery; In a system comprising: The operation may further include: storing the one or more sets of coordinates of one or more landmarks; modifying a view of the display to exclude the overlaid indicia from being displayed; restoring the view to a previous display; identifying the one or more sets of coordinates of the one or more landmarks; re-overlaying said one or more indicators; Including, and modifying the view is automatically activated based on changes in identified anatomy or lesions. system.

2. The system described in claim 1, wherein the operator activates the changing step and the returning step.

3. The system of claim 1 , wherein the minimally invasive procedure is arthroscopic surgery.

4. 10. The system of claim 1, wherein the operations further comprise using at least one trained computer algorithm to identify and label one or more elements in the video stream, the one or more elements comprising one or more of an anatomical structure, a surgical instrument, a procedure or action, or a pathology.

5. The system of claim 4 , wherein identifying and labeling one or more elements in the video stream comprises using one or more modules.

6. The system of claim 5 , wherein the one or more modules include video stream decomposition, instrument recognition, anatomy recognition, instrument tracking, gesture recognition, landmark registration, or anatomy and landmark tracking.

7. The system of claim 5 , wherein the system recommends one or more indicators based at least in part on the identified elements.

8. The system of claim 1 , wherein the one or more sets of coordinates of the one or more landmarks are provided intraoperatively by an operator.

9. The system of claim 1 , wherein the one or more sets of coordinates of the one or more landmarks are provided pre-operatively by an operator.

10. The system of claim 1 , wherein the one or more sets of coordinates of the one or more landmarks are generated from one or more medical images of the object.

11. The system of claim 10 , wherein the one or more medical images are radiographic images of the subject.

12. The system of claim 11 , wherein the radiological image is from the target joint.

13. The system of claim 11 , wherein the radiological image is associated with a shoulder, a knee, or a hip of the subject.

14. 12. The system of claim 11, wherein the radiological image is generated using fluoroscopy, magnetic resonance imaging (MRI), computed tomography (CT) scan, positron emission tomography (PET) scan, or ultrasound imaging.

15. The system of claim 3 , wherein the video stream is provided by an arthroscope during arthroscopic surgery.

16. The system of claim 3 , wherein the arthroscopic surgery is used in a rotator cuff tear repair procedure.

17. The system of claim 3 , wherein the arthroscopic procedure is used for cruciate ligament repair in knee surgery.

18. The system of claim 3 , wherein the arthroscopic procedure is used in a graft placement procedure.

19. The system of claim 3 , wherein the arthroscopic procedure is used in a decompression procedure.

20. The system of claim 3 , wherein the arthroscopic procedure is used to remove or resect one or more inflamed tissues.

21. The system of claim 3 , wherein the arthroscopic surgery is used in the removal or resection of one or more torn tendons, and the video stream is monocular.

22. The system of claim 1 , wherein the video stream is stereoscopic.

23. The system of claim 1 , wherein the one or more computer processors receive the video streams from one or more camera control units using a wired media connection.

24. The system described in claim 23, wherein the latency between receiving input from a digital camera and outputting and overlaying the video stream is at most 40 milliseconds (ms) to accommodate digital cameras of approximately 24 frames per second (fps).

25. The system described in claim 23, wherein the latency between receiving input from a digital camera and outputting and overlaying the video stream does not exceed the time between two consecutive frames from the digital camera.

26. The system of claim 1 , wherein the one or more computer processors receive the video streams from one or more camera control units using a network connection.

27. The system of claim 3 , wherein the arthroscopic imaging device is a digital camera specialized for use in arthroscopes.

28. 28. The system of claim 27, wherein the digital camera is mounted on a rigid scope suitable for operation in an arthroscopic joint.

29. The system described in claim 27, wherein the one or more computer processors receive the video stream from one or more camera control units using a wired media connection, and the camera control units are configured to control a light source to capture digital information generated by the digital camera.

30. 30. The system of claim 29, wherein the camera control unit converts the digital information produced by the digital camera into the video stream.

31. 30. The system of claim 29, wherein the camera control unit records the digital information generated by the digital camera in a memory device.

32. 32. The system of claim 31, wherein the memory device is a local memory device.

33. 32. The system of claim 31, wherein the memory device is a cloud-based memory device.

34. 28. The system of claim 27, wherein the digital camera is connected to a camera control unit.

35. 35. The system of claim 34, wherein the camera control unit is configured to overlay output from the one or more computer processors onto the video stream.

36. The system of claim 1 further comprising a display monitor.

37. The system of claim 1 , wherein the one or more computer processors include a central processing unit or a graphics processing unit.

38. The system of claim 1 , further comprising a mechanism for receiving input (to activate or deactivate said marking) from said at least one operator during surgery.

39. 40. The system of claim 38, wherein the mechanism receives the input via a push button, a touch screen device, a foot pedal, a gesture recognition system, or a voice recognition system.

40. The system of claim 1 , wherein the one or more landmarks are tracked during the minimally invasive procedure.

41. 41. The system of claim 40, wherein the tracking of one or more landmarks related to at least an anatomical structure is associated with the set of coordinates of the one or more landmarks.

42. 41. The system of claim 40, wherein the display of the one or more signs is overlaid on the display of the video stream.

43. 43. The system of claim 42, wherein the operator can select to make the one or more markers invisible temporarily during the minimally invasive procedure or throughout the minimally invasive procedure.

Images