Methods and apparatus for tracking anatomical structure in video assisted surgeries

Abstract

This disclosure describes an intra-operative anatomical structure tracking and alerting system for improving surgeon's context awareness during laparoscopic and video-assisted surgeries. During laparoscopic and video-assisted surgeries, a surgeon can mark anatomical structures for the system to track (for example: vessel, nerve, bile duct, triangle of pain in hernia repair, etc.), then continue with the surgery. When the scope moves, such a marked structure may become invisible. The system uses SLAM (Simultaneous Localization and Mapping) to track and recover the precise location of the scope. When the camera moves to an angle where the previously marked structure becomes visible again, the system will redraw the marks to indicate successful tracking of such a structure. Also, if a surgical tool (for example, energy device) comes near the anatomical structure, an alert (for example, sound or visual alert on screen) can be sent to the surgeon for caution.

Description

METHODS AND APPARATUS FOR TRACKING ANATOMICAL STRUCTURE IN VIDEO ASSISTED SURGERIESCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 387,984, filed on December 19, 2022, the entire content of which is incorporated as reference.TECHNICAL FIELD OF THE INVENTION

[0002] The present application relates to laparoscopic and video-assisted surgeries, in particular, an intra-operative anatomical structure tracking and alerting system for improving surgeon’s context awareness during laparoscopic and video-assisted surgeries.BACKGROUND OF THE INVENTION

[0003] During surgeries, it is crucial for surgeons to accurately identify critical anatomical structures, such as safe and dangerous dissection zones, major blood vessels, organs, nerves, etc., in all camera angles. One major challenge in laparoscopic or videoassisted surgery is the limited viewing condition for surgeons, due to the small viewing angle of laparoscopic or other types of cameras. While surgeons can successfully identify such structures in most camera angles, keeping track of them at all angles can cause significant mental fatigue or is sometimes infeasible. Also, surgical devices, such as scalpels and energy devices might inadvertently contact such anatomical structures, causing patient injury. “Energy devices” as used herein may include hand-held instruments that emit energy, such as high frequencies, ultrasonic waves, cryo- or thermal, light beams of different spectrums and microwaves used to seal blood vessels, cut tissue, ablate abnormal tissues, or tumors and stop bleeding.

[0004] There have been previous attempts to utilize machine leaming / deep learning / computer vision to recognize critical structures in surgical scenes, such as safe / dangerous zones during laparoscopic procedures, identification of blood vessels during surgery, etc. For example, see Madani, A. et al, Artificial Intelligence for Intraoperative Guidance: Using Semantic Segmentation to Identify Surgical Anatomy during Laparoscopic Cholecystectomy. Ann Surg. 2022 Aug 1; 276(2):363-369. doi:10.1097 / SLA.0000000000004594. These approaches recognize and highlight target anatomical structures when they are visible. However, when the camera moves to an angle where those structures are no longer visible, the recognition is lost. Further, when there are multiple cameras in use during surgery, the safe / dangerous zones visible on one screen are not easily translated to other screens / video feeds.

[0005] Thus, there is a need for methods and apparatus that can render video streams that clearly indicate the safe / dangerous zones in all the video feeds in real time, even when the camera angles are changed during the procedure.SUMMARY OF THE INVENTION

[0006] In one aspect, the invention provides a method for tracking and locating a region of interest, such as anatomical structure, from all camera angles. This will enable monitoring of potential or actual contact between a surgical instrument and the anatomical structure. The method involves taking a snapshot of a video stream of the surgical procedure. Then, the region of interest is marked using a suitable marking tool on the snapshot. Subsequently, as the camera is moved, or the angle of viewing is changed, or the focal length of the camera is changed, the mark is adaptively redrawn at a current position of the region of interest so that the surgeon is always aware of the position of the region of interest. At any time during the surgery, if a surgical tool (such as a scalpel, an energy device, etc.) comes close to the mark, an alert is provided. The alert may be in the form of a visual alert, such as a bright flashing light, a text appearing on the screen, and so on, and combinations thereof. Alternatively, the alert may be in the form of a sound, such as an alarm sound. Further, the alert may be varied based on the gap between the surgical tool and the region of interest. Thus, as the gap distance reduces, a visual alert may change color from shades of orange to shades of red, or the volume of the sound alert may be increased. Thus, the surgeon can focus on performing the surgery and not be mentally burdened about potentially damaging any anatomical structure during the procedure.

[0007] In another aspect, the invention provides an apparatus for use during a surgical procedure. The apparatus comprises an imaging device, such as a laparoscopic camera or an endoscopy camera, etc., configured to stream video. The apparatus of the invention then includes a display unit to display the streamed video. The display unit is also capable of displaying snapshot images taken of the streamed video. To enable this, the display unit isconfigured to display multiple windows, such as a separate window for each individual camera feed, a distinct window for zoomed images, a separate window for snapshots of the streamed video, and so on.

[0008] The apparatus of the invention then includes a marking tool that allows marking a region of interest. The marking tool may be a keyboard wherein the marking is achieved through the use of arrow and enter keys or other predefined keys, a mouse, a touch input if the screen is so enabled, a stylus if the screen is capable of taking on stylus inputs, a handheld controller, such as a wireless mouse, and the like, and combinations thereof.

[0009] Then, a memory is included as part of the apparatus, which includes Read Only Memory (ROM), Random Access Memory (RAM), Solid-State Drive Hard Drive (SSD-HD), and so on, and combinations thereof. The memory includes instructions for a code related to the method of the invention as described herein. The memory is also used to store a model for identifying parts of an image. For example, the model may be used to detect regions of interest, anatomical structures, surgical tools, and the like. In some specific instances, the model is used to detect tips of surgical tools. The model is preferably based on a machine learning (ML) model, an artificial learning (Al) model, a computer vision (CV) model, and the like, and combinations thereof. In preferred embodiments, the model is generated and trained through a specific database that is annotated by surgeons.

[0010] The apparatus further comprises a processor configured to execute the code. The code includes instructions for performing at least the following steps: to receive the streamed video; take a snapshot of a particular streamed video; communicate with the marking tool; receive instructions from a user to mark a region of interest on the snapshot using the marking tool; identify the region of interest on subsequent video streams using the model; render subsequent video streams by adaptively redrawing the mark on the region of interest wherever it is present; identify a position of a surgical tool; and estimate a gap between a tip of the surgical tool and the mark.

[0011] The apparatus then comprises an alert tool to raise an alert if the gap between the tip of the surgical tool and the mark falls below a threshold as estimated by the processor. “Threshold” as used herein means a distance between the tip of the surgical tool and the mark. The distance may be measured at an appropriate location on the mark. For example, when the mark is a line, the distance between the closest point on the mark to the tip will beconsidered. Alternatively, the threshold may be the distance between a predetermined point on the mark to the tip. Depending on the nature of the surgical tool in use, the threshold may vary. For example, when a scalpel is used, the threshold may be 1 micrometer, whereas when an energy device is used, the threshold may be 10 micrometers. In other instances, the threshold may mean that the tip of the surgical tool tip actually overlaps the mark. In some instances, the alert tool is included in the display unit, wherein the alert will appear as a text or an image in a suitable color, such as a flashing red text box. In other instances, the alert tool is in the form of a sound alert, towards which a speaker can be included, or a sound card can be incorporated into, for example, the display unit. Combinations of visual and sound alert is also envisioned within the scope of the invention.

[0012] The method and apparatus of the invention provide a significant advantage over the prior art in that the identified region of interest is marked in one image, and subsequently, all the video feeds / streams will include the mark wherever the region of interest is present. This is achieved using Simultaneous Location and Mapping (SLAM), which maintains a memory of areas seen before. Thus, even in feeds / streams / frames where the marked structure is not visible, the location of the region of interest is generally known. Subsequently, when the camera moves back to an angle or focal length where the marked structure is visible again, it will be marked again and highlighted for the surgeon’s attention.

[0013] In preferred embodiments, the initial identification of the region of interest and the mark is made by a trained human, such as a surgeon, a healthcare professional, a surgeon’s assistant, or the like because some anatomical features are not recognized by Al / ML / CV models. Thus, any potential errors that may occur due to false identifications by the models are negated.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] These and other features, aspects, and advantages of the present invention will become better understood when the following detailed description is read with reference to the accompanying drawings in which like characters represent like parts throughout the drawings, wherein:

[0015] Figure 1 is a schematic illustration of the method steps based on an embodiment of the invention.

[0016] Figure 2 A is an exemplary display of a hernia repair surgery.

[0017] Figure 2B is yet another exemplary display of a laparoscopic cholecystectomy.

[0018] Figure 3 shows a tracking surgical tool tip based on an embodiment of the present invention (the green cross represents the neural network output of the detected monopolar curved scissor tip).

[0019] Figure 4 is a block diagrammatic representation of an apparatus based on an embodiment of the invention.DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS OF THE INVENTION

[0020] The definitions provided herein are to facilitate understanding of certain terms used frequently herein and are not meant to limit the scope of the present disclosure.

[0021] As used in this specification and the appended claims, the singular forms "a", "an", and "the" encompass embodiments having plural referents, unless the content clearly dictates otherwise.

[0022] Unless otherwise indicated, all numbers expressing feature sizes, amounts, and physical properties used in the specification and claims are to be understood as being modified in all instances by the term "about." Accordingly, unless indicated to the contrary, the numerical parameters set forth in the foregoing specification and attached claims are approximations that can vary depending upon the desired properties sought to be obtained by those skilled in the art utilizing the teachings disclosed herein.

[0023] As used in this specification and the appended claims, the term "or" is generally employed in its sense including "and / or" unless the content clearly dictates otherwise.

[0024] Reference is made to Figure 1, which is a schematic illustration of the method steps based on an embodiment of the present invention, generally represented by numeral 100. The method comprises obtaining at least one video stream 102 from a suitable imaging device, e.g., an endoscope or laparoscope. The method then involves taking a snapshot of the video stream in step 104. The manner in which a snapshot of a video stream is taken may be well known in the art and may include, for example, but not limited to, pushing a keystrokeon a keyboard, clicking via a handheld controller, tapping a certain portion of a touchpad or a screen, gesture, voice command, and the like, and combinations thereof.

[0025] Then, the method comprises marking a region of interest in step 106. As noted herein, the marking of the region of interest may be done manually and / or with assistance from Al-based or ML-based models. The marking is preferably performed by a human, such as a surgeon, a surgeon’s assistant, a nurse, another healthcare practitioner, and the like. This is because not all anatomical structures are well recognized by models. The marking may be achieved using a suitable method, such as a mouse click at a certain region on the screen representing the region of interest, a stylus, a touch, a gesture, and the like, and combinations thereof. As an illustration of this step, when the interested anatomical structure is visible on the laparoscope monitor, the surgeon holds the laparoscope still and takes a snapshot to obtain a still image. Then a touchpad or mouse is used to mark the interested anatomical structure on such an image. Either a surgeon will pause surgery and mark the anatomical structure, or an assistant or nurse can draw such marking. A line or a filled shape can be used to mark a vessel or nerve, and a polygon can be used to mark a region, such as a triangle of pain in hernia repair surgery. The region of interest is used to identify a critical space around an anatomical structure, which is visible to the surgeon during surgery but is not meant to come in contact with the surgical tool, or the space is tight that the surgical tool entering the space may risk introducing unexpected accidental contact to the anatomical structure. Thus, it is advantageous for the surgeon to mark the region of interest ahead of time so that, during the surgical procedure, the regions of interest can be avoided, and the surgery can be performed only on the surgical site. It must be noted that the region of interest may include more than one location surrounding the surgical site.

[0026] The method of the invention then involves identifying a region of interest in all video streams in step 108. The method of the invention advantageously uses simultaneous localization and mapping, sometimes abbreviated in the art as SLAM (e.g., OrbSLAM-2), to track the camera trajectory and build out a map of the surgical scene as the laparoscope moves around the surgical site. Specifically, SLAM is a method used for simultaneously building a map and localizing the camera in that map, allowing the camera to map out unknown environments. The 3D structure of the surgical scene is gradually built as the camera moves around. With such an intra-operationally built 3D model of the surgical scene, each point in the current laparoscope frame can be mapped to a 3D point on the 3D model; onthe other hand, every 3D point on the 3D model can be projected to the 2D space of the current frame, and if the projected location of the 3D point is outside the frame boundary, it means that the 3D point is currently not visible. To re-identify the anatomical structure marked by the surgeon (the region of interest), the marked structure’s 3D points are projected back to the current frame to visualize those points inside the current frame’s boundary. Based on the camera angle, 3 scenarios are possible:1. The entire region of interest is visible and highlighted in the current frame;2. Only part of the region of interest is visible and therefore only the visible part is highlighted in the current frame; it is possible to build an additional virtue reality frame that renders invisible parts overlay on the current frame;3. The entire region of interest is outside the current frame boundary, and it is not highlighted in the current frame; it is possible to build an additional virtue reality frame that renders invisible parts overlay on the current frame.

[0027] In this manner, in step 110, the marking is reproduced in all the video streams where the region of interest is visible in real-time. Thus, the surgeon is always aware of the position of the anatomical structure while performing surgery.

[0028] Figure 2A is an exemplary display of a hernia repair surgery resulting from the method of the invention, wherein line 202 demarcates safe region 204 and unsafe region 206. Further, visual aids, such as graphic indicators or emojis - for example, smiley and crying faces - can be used to indicate the safe and unsafe zones to enable easier identification by surgeons during surgical procedures.

[0029] Figure 2B is yet another exemplary display of a laparoscopic cholecystectomy resulting from the method of an embodiment of the invention showing more nuanced variations within the safe and unsafe zones, which are demarcated by line 208. The safe zone is further marked by a triangle (an exemplary polygon) to indicate very safe zones D 210, and relatively safe zones I 212. Appropriate choice of colors can also be applied for quick identification by the surgeon. Similarly, an unsafe zone is further marked with a first triangle labeled ‘Pain’ 214 that may be colored using a certain color to represent a mildly dangerouszone, a second triangle labeled ‘Doom’ 216 and appropriately (red to indicate severe danger), and a third triangle labeled F 218.

[0030] Then, the position of the tip of the surgical tool is identified in step 112 of the method based on an embodiment of the invention. This may be the tip of a scalpel or an energy device, as examples of surgical tools. This is advantageously achieved using well- trained Al / ML / CV models to ensure speed and accuracy. In some instances, a neural network is trained to detect various surgical tools, such as a scalpel or an energy device.

[0031] Figure 3 shows an exemplary surgical tool tip 302.

[0032] Then a gap between the tip of the surgical tool and the mark is estimated in step 114. If the gap is below a threshold, or there is an overlap between the position and the tip of the surgical tool and the mark, then this suggests that there is potential for contact between the tip of the surgical tool and the anatomical structure. This may be a cause for concern, due to the potential for further complications, unwanted injuries, etc. An alert is given to the surgeon, which may be in the form of a visual alert on a screen in the form of a colored flashing light or pop-up text, a sound alarm, or combinations thereof.

[0033] Reference is now made to Figure 4, which is a block diagrammatic representation of an apparatus based on an embodiment of the invention, generally represented by numeral 400. Apparatus 400 comprises at least one imaging device 402 that is configured to capture video, which includes a plurality of frames. The video may be of any portion of the interior of the body, including, for example, the gastrointestinal tract or urinary tract, etc. Apparatus 400 further comprises a medical image processing system 404, which comprises a memory 406, a display unit 408, a marking tool 410, a processor 412, along with communication interfaces and software for carrying out the functions prescribed by the present invention.

[0034] Processor 412 as used herein may comprise one or more processors, controllers, control modules, or other processing devices. Processor 412 might be implemented using a general-purpose or special-purpose processing engine such as, for example, a microprocessor, controller, Graphic Processing Unit (GPU), Field Programmable Gate Arrays (FPGAs), or other control logic. Processor 412 is connected to a bus or any other communication medium to facilitate interaction with other components shown herein or to communicate externally.

[0035] Memory 406 comprises a non-volatile memory, such as one or more hard disk drives, and / or a volatile memory, such as random-access memory (RAM). Other memory modules may also be included in memory 406. For example, preferably, random access memory (RAM) or other dynamic memory might be used for storing information and instructions to be executed by processor 412. Hard disk drives, solid state drives, or other main memory might also be used for storing temporary variables or other intermediate information during the execution of instructions to be executed by processor 412. Further, a read-only memory (“ROM”) or other static storage device may be used for storing static information and instructions for processor 412.

[0036] Also, memory 406 may also include one or more various forms of information storage mechanism, which might include, for example, a media drive and a storage unit interface. The media drive might include a drive or other mechanism to support fixed or removable storage media. For example, a hard disk drive, a floppy disk drive, a magnetic tape drive, an optical disk drive, a CD or DVD drive (R or RW), or other removable or fixed media drive might be provided. Accordingly, storage media might include, for example, a hard disk, a floppy disk, magnetic tape, cartridge, optical disk, a CD or DVD, or other fixed or removable medium that is read by, written to, or accessed by media drive. As these examples illustrate, the storage media can include a computer usable storage medium having stored therein computer software or data.

[0037] In alternative embodiments, memory 406 might include other similar instrumentalities for allowing computer programs or other instructions or data to be loaded into a suitable computing module. Such instruments might include, for example, a fixed or removable storage unit and an interface. Examples of such storage units and interfaces can include a program cartridge and cartridge interface, a removable memory (for example, a flash memory or other removable memory module) and memory slot, a PCMCIA slot and card, and other fixed or removable storage units and interfaces that allow software and data to be transferred from the storage unit to the computing module.

[0038] The apparatus is deployed in a console physically located in an operating room and connected to a surgical system. The apparatus is configured to perform real-time inference of the Al I ML I CV models. The apparatus will also record tracking, SLAM and detection results generated during the procedure and save such information to disk as system logs. The apparatus may be a stand-alone device with a tracking and alerting system madeavailable within the physical apparatus envisioned based on an embodiment of the present invention. Alternatively, the apparatus may be provided as software that is integrated into existing infrastructure already available in a suitable environment, such as a hospital.

[0039] Where components or modules of the technology are implemented in whole or in part using software, in one embodiment, these software elements can be implemented to operate with a computing or processing module capable of carrying out the functionality described with respect thereto. After reading this description, it will become apparent to a person skilled in the relevant art how to implement the technology using other computing modules or architectures. Computing module may represent, for example, computing or processing capabilities found within desktop, laptop, and notebook computers; hand-held computing devices (PDA's, smartphones, cell phones, palmtops, etc.); mainframes, supercomputers, workstations, or servers; or any other type of special-purpose or general- purpose computing devices as may be desirable or appropriate for a given application or environment. Computing module might also represent computing capabilities embedded within or otherwise available to a given device. For example, a computing module might be found in other electronic devices, such as digital cameras, navigation systems, cellular telephones, portable computing devices, modems, routers, WAPs, terminals, and other electronic devices that might include some form of processing capability.

[0040] The computing module might also include a communications interface that is used to allow software and data to be transferred between the computing module and external devices. Examples of communication interfaces include a modem, a network interface (such as an Ethernet, network interface card, WiMedia, IEEE 802.XX, and other interface), a communications port (such as a USB port, IR port, RS232 port Bluetooth® interface, or other port), or other such communications interface. Software and data transferred via communications interface might typically be carried on signals, which can be electronic, electromagnetic (which includes optical) or other signals capable of being exchanged by a given communications interface. These signals might be provided to communications interface via a wired or wireless communication medium. Some examples of a channel might include a phone line, a cellular link, an RF link, an optical link, a network interface, a local or wide area network, and other wired or wireless communications channels.

[0041] The computing module also includes input / output (I / O) devices (including but not limited to keyboards, displays, pointing devices, etc.) that can be coupled to the apparatus either directly or through intervening I / O controllers.

[0042] The terms “computer program medium” and “computer usable medium” are used to generally refer to media such as memory, storage unit, and media. These and other various forms of computer program media or computer-usable media may be involved in carrying one or more sequences of one or more instructions to a processing device for execution. Such instructions embodied on the medium, are generally referred to as “computer program code” or a “computer program product” (which may be grouped in the form of computer programs or other groupings). When executed, such instructions might enable the computing module to perform features or functions of the disclosed technology as discussed herein. Computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object-oriented programming language, such as Java, Small talk, C++, or the like, and conventional procedural programming languages, such as the C programming language or similar programming languages.

[0043] While various embodiments of the disclosed technology have been described above, it should be understood that they have been presented by way of example only, and not of limitation. Likewise, the various diagrams may depict an example architectural or other configuration for the disclosed technology, which is done to aid in understanding the features and functionality that can be included in the disclosed technology. The disclosed technology is not restricted to the illustrated example architectures or configurations, but the desired features can be implemented using a variety of alternative architectures and configurations. Indeed, it will be apparent to one of skill in the art how alternative functional, logical or physical partitioning and configurations can be implemented to implement the desired features of the technology disclosed herein. Also, a multitude of different constituent module names other than those depicted herein can be applied to the various partitions. Additionally, with regard to flow diagrams, operational descriptions and method claims, the order in which the steps are presented herein shall not mandate that various embodiments be implemented to perform the recited functionality in the same order unless the context dictates otherwise.

[0044] Although the disclosed technology is described above in terms of various exemplary embodiments and implementations, it should be understood that the various features, aspects and functionality described in one or more of the individual embodiments are not limited in their applicability to the particular embodiment with which they are described, but instead can be applied, alone or in various combinations, to one or more of the other embodiments of the disclosed technology, whether or not such embodiments are described and whether or not such features are presented as being a part of a described embodiment. Thus, the breadth and scope of the technology disclosed herein should not be limited by any of the above-described exemplary embodiments.

[0045] Terms and phrases used in this document, and variations thereof, unless otherwise expressly stated, should be construed as open-ended as opposed to limiting. As examples of the foregoing: the term “including” should be read as meaning “including, without limitation” or the like; the term “example” is used to provide exemplary instances of the item in discussion, not an exhaustive or limiting list thereof; the terms “a” or “an” should be read as meaning “at least one,” “one or more” or the like; and adjectives such as “conventional,” “traditional,” “normal,” “standard,” “known” and terms of similar meaning should not be construed as limiting the item described to a given time period or to an item available as of a given time, but instead should be read to encompass conventional, traditional, normal, or standard technologies that may be available or known now or at any time in the future. Likewise, where this document refers to technologies that would be apparent or known to one of ordinary skill in the art, such technologies encompass those apparent or known to the skilled artisan now or at any time in the future.

[0046] The presence of broadening words and phrases such as “one or more,” “at least,” “but not limited to” or other like phrases in some instances shall not be read to mean that the narrower case is intended or required in instances where such broadening phrases may be absent. The use of the term “module” does not imply that the components or functionality described or claimed as part of the module are all configured in a common package. Indeed, any or all of the various components of a module, whether control logic or other components, can be combined in a single package or separately maintained and can further be distributed in multiple groupings or packages or across multiple locations.

[0047] Additionally, the various embodiments set forth herein are described in terms of exemplary block diagrams, flow charts, and other illustrations. As will become apparentto one of ordinary skill in the art after reading this document, the illustrated embodiments and their various alternatives can be implemented without confinement to the illustrated examples. For example, block diagrams and their accompanying description should not be construed as mandating a particular architecture or configuration.

Claims

CLAIMSWhat is claimed is:

1. A method comprising: obtaining at least one video stream of a surgical procedure; taking a snapshot of the video stream; marking a region of interest on the snapshot; identifying a presence or absence of the region of interest in a subsequent video stream; and drawing a mark at the region of interest wherever present in the subsequent video stream to inform a surgeon the region of interest’s position.

2. The method of claim 1 wherein the marking is performed by a human.

3. The method of claim 1 further comprising identifying a position of a tip of a surgical tool in the subsequent stream.

4. The method of claim 3 further comprising providing an alert when a gap between the tip of the surgical tool and the mark is below a threshold.

5. The method of claim 3 wherein the position of the tip of the surgical tool is identified by an artificial intelligence method.

6. The method of claim 1 wherein the mark is a line or a filled shape.

7. The method of claim 4 wherein the alert is at least one of a visual alert, a sound alert, and a combination thereof.

8. The method of claim 1 wherein the video stream is used to build a three- dimensional model using a simultaneous localization and mapping method.

9. The method of claim 1 wherein the region of interest comprises at least one anatomical structure or a space around the anatomical structure.

10. An apparatus for use during a surgical procedure, the apparatus comprising: an imaging device configured to stream video; a display unit to display the streamed video; a marking tool; a memory in which a model is stored; and a processor configured to: receive the streamed video; take a snapshot of a particular streamed video; communicate with the marking tool; receive instructions from a user to draw a mark on a region of interest on the snapshot; identify the region of interest on a subsequent video stream using the model by drawing the mark; and re-mark the region of interest on the subsequent video stream.

11. The apparatus of claim 10 wherein the subsequent video stream is rendered using a three-dimensional model that uses a simultaneous localization and mapping method.

12. The apparatus of claim 10 wherein the model is configured to identify a tip of a surgical tool.

13. The apparatus of claim 12 wherein the processor is further configured to identify a gap between the tip of the surgical tool and the mark at the region of interest.

14. The apparatus of claim 13 wherein the processor is configured to provide an alert based on the gap being less than a predefined threshold.

15. The apparatus of claim 14 wherein the alert is at least one of a visual, a sound, and a combination thereof.

Images