Method and apparatus for tracking anatomical structures in visually assisted surgery

Abstract

The present invention relates to an intraoperative anatomical structure tracking and alert system for improving surgeons' situational awareness during laparoscopic and visual-assisted surgery. During laparoscopic and visual-assisted surgery, surgeons can mark anatomical structures (e.g., blood vessels, nerves, bile ducts, and pain triangles in hernia repair) for systemic tracking and continue surgery. As the endoscope moves, the marked structures may become invisible. The system uses synchronous localization and mapping (SLAM) to track and restore the endoscope's exact position. When the camera moves to an angle where the previously marked structure is once again visible, the system redraws the marking to indicate successful tracking of the structure. Additionally, if a surgical instrument (e.g., an energy device) approaches an anatomical structure, an alert (e.g., an audio or visual alert on a screen) can be sent to the surgeon to prompt his or her attention.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 387,984, filed December 19, 2022, the entire contents of which are incorporated herein by reference.

[0002] The present invention relates to laparoscopic and visually assisted surgery, and more particularly to an intraoperative anatomy tracking and alert system for improving a surgeon's situational awareness during laparoscopic and visually assisted surgery. [Background technology]

[0003] During surgery, it is crucial for surgeons to accurately identify important anatomical structures (e.g., safe zones and anatomical risk zones, major blood vessels, organs, nerves, etc.) at all camera angles. One of the major challenges of laparoscopic or visual-assisted surgery is the limited viewing conditions for surgeons due to the small viewing angle of laparoscopes and other types of cameras. While surgeons can identify these structures at most camera angles, continuously tracking them at all angles can cause significant mental fatigue and is sometimes even impossible. Furthermore, surgical instruments (e.g., surgical knives and energy devices) may unintentionally come into contact with these anatomical structures and cause injury to the patient. As used herein, the term "energy device" may include handheld instruments that emit energy such as radiofrequency, ultrasound, cryo- or thermal energy, different spectrums of light, microwaves, etc., to occlude blood vessels, cut tissue, ablate abnormal tissue or tumors, or achieve hemostasis.

[0004] Previous attempts have been made to identify important structures in surgical scenes (e.g., safe / dangerous areas during laparoscopic surgery, identifying blood vessels during surgery) using machine learning, deep learning, and computer vision. For example, methods like those described in "Artificial Intelligence for Intraoperative Guidance: Using Semantic Segmentation to Identify Surgical Anatomy during Laparoscopic Cholecystectomy" by Madani (Ann Surg. 2022 Aug. 1; 276(2):363-369. doi: 0.1097 / SLA.0000000000004594) identify and highlight target anatomical structures when they are visible. However, when the camera moves to an angle where these structures are not visible, the identification is lost. Also, when multiple cameras are used during surgery, the safety / danger zones visible on one screen cannot be easily translated to other screens / video streams. Summary of the Invention [Problem to be solved by the invention]

[0005] Therefore, what is needed is a method and apparatus that can clearly indicate safe / dangerous zones in real time in all video streams, even as the camera angle changes during surgery. [Means for solving the problem]

[0006] In one aspect, the present invention provides a method for tracking and locating a region of interest (e.g., an anatomical structure) from all camera angles, thereby enabling monitoring of potential or actual contact between a surgical instrument and an anatomical structure. The method involves taking snapshots of a video stream during surgery. The region of interest is then marked on the snapshot using an appropriate marking tool. Subsequently, as the camera moves, the viewing angle changes, or the camera's focal length is altered, the marking is adaptively redrawn to the current location of the region of interest, ensuring that the surgeon is always aware of the location of the region of interest. If, at any point during surgery, a surgical instrument (e.g., a surgical knife, energy device, etc.) approaches the marking, an alert is generated. The alert may be provided in the form of a visual alert, such as a bright flashing light or text displayed on a screen, or a combination thereof. Alternatively, the alert may be audio, such as an audible alarm. Furthermore, the alert may change depending on the distance between the surgical instrument and the region of interest. For example, the color of the visual alert may change from orange to red, or the volume of the audio alert may increase as the distance decreases. This allows the surgeon to focus on performing the surgery and not worry about which anatomical structures they may damage during the procedure.

[0007] In another aspect, the present invention provides an apparatus for use during surgery, the apparatus including an imaging device (e.g., a laparoscopic camera, an endoscopic camera, etc.) configured to transmit a video stream. The apparatus also includes a display unit for displaying the video stream. The display unit is also capable of displaying snapshot images of the captured video stream. To this end, the display unit is configured to display multiple windows, such as a separate window for each individual camera source, a separate window for scaled images, and a separate window for the snapshot of the video stream.

[0008] The device of the present invention also includes a marking tool that can be used to mark the region of interest, which can be a keyboard (using the arrow keys and Enter key or other predefined keys to make the marking), a mouse, a screen that supports touch input (if the screen supports it), a touch pen for a screen that supports touch pen input, a handheld controller (e.g., a wireless mouse), etc., and combinations thereof.

[0009] The device, in turn, includes, as part thereof, a memory including a read-only memory (ROM), a random access memory (RAM), a solid-state drive hard drive (SSD-HD), etc., and combinations thereof. The memory stores code instructions related to the methods of the present invention described herein. The memory is also used to store a model for identifying portions of an image. For example, the model can be used to detect regions of interest, anatomical structures, surgical instruments, etc. In a specific example, the model is used to detect the tip of a surgical instrument. The model is preferably based on a machine learning (ML) model, an artificial learning (AI) model, a computer vision (CV) model, etc., and combinations thereof. In a preferred embodiment, the model is generated and trained by a specific database annotated by a surgeon.

[0010] The apparatus also includes a processor configured to execute the code, which includes instructions for at least receiving a video stream, taking a snapshot of a particular video stream, communicating with a marking tool, receiving instructions from a user and marking a region of interest on the snapshot with the marking tool, identifying a region of interest on a subsequent video stream using the model, rendering the subsequent video stream by adaptively redrawing the marking where the region of interest is located, identifying a position of a surgical instrument, and estimating a spacing between the tip of the surgical instrument and the marking.

[0011] The device then includes an alert tool that issues an alert when the distance between the tip of the surgical instrument and the marking falls below a threshold estimated by the processor. As used herein, "threshold" refers to the distance between the tip of the surgical instrument and the marking. The distance can be measured at an appropriate position on the marking. For example, if the marking is a line, the distance of the point on the marking closest to the tip is considered. Alternatively, the threshold can be the distance from a predetermined point on the marking to the tip. The threshold can vary depending on the nature of the surgical instrument used. For example, if a surgical knife is used, the threshold can be 1 micrometer, while if an energy device is used, the threshold can be 10 micrometers. In other cases, the threshold can mean that the tip of the surgical instrument actually overlaps the marking. In some cases, the alert tool is included in the display unit, and the alert is displayed in the form of text or an image using an appropriate color (e.g., a flashing red text box). In other cases, the alert tool takes the form of an audio alert and can include a speaker or, for example, a sound card integrated into the display unit. A combination of visual and audio alerts is also within the scope of the present invention. [Effects of the Invention]

[0012] The method and apparatus of the present invention offer significant advantages over the prior art because an identified region of interest is marked in one image, and all subsequent video sources / streams will include that marking at the location where the region of interest is located. This is achieved using Simultaneous Location and Mapping (SLAM), which maintains a memory of previously viewed regions. Thus, even in sources / streams / frames where the marked structure is not visible, the location of the region of interest is generally known. When the camera subsequently returns to an angle or focal length where the marked structure is once again visible, it is again marked and highlighted, drawing the surgeon's attention.

[0013] In a preferred embodiment, the initial identification and marking of the region of interest is performed by trained personnel (e.g., surgeons, medical personnel, surgeon's assistants, etc.) because some anatomical features cannot be identified by AI / ML / CV models, thus eliminating potential errors due to model misidentification. [Brief explanation of the drawings]

[0014] These and other features, aspects and advantages of the present invention will be better understood from the following detailed description when read in conjunction with the accompanying drawings, in which like reference numerals represent like parts in all drawings: [Figure 1] FIG. 1 is a schematic diagram illustrating the steps of a method according to one embodiment of the present invention. [Figure 2A] FIG. 2A is an exemplary representation of a hernia repair procedure. [Figure 2B] FIG. 2B is another exemplary representation of a laparoscopic cholecystectomy. [Figure 3] FIG. 3 shows tracking of a surgical tool tip according to one embodiment of the present invention (green cross represents the output of the neural network for the detected tip of a monopolar curved scissors). [Figure 4] FIG. 4 is a block diagram of an apparatus according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] The definitions provided herein are intended to facilitate understanding of certain terms used frequently herein and are not intended to limit the scope of the present disclosure.

[0016] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include embodiments including plural referents unless the context clearly dictates otherwise.

[0017] Unless otherwise noted, all numerical values ​​expressing dimensions, quantities and physical properties of features used in the specification and claims are understood to be modified in all instances by the term "about." Accordingly, unless otherwise indicated to the contrary, the numerical parameters set forth in the foregoing specification and appended claims are approximations that may vary depending upon the desired properties sought by those of ordinary skill in the art using the teachings disclosed herein.

[0018] As used in this specification and the appended claims, the term "or" is generally used in its sense including "and / or" unless the context clearly dictates otherwise.

[0019] 1, which is a schematic diagram illustrating steps of a method according to one embodiment of the present invention, generally designated 100. The method includes acquiring at least one video stream from a suitable imaging device (e.g., an endoscope or laparoscope) at step 102. The method then includes taking snapshots of the video stream at step 104. Methods for taking snapshots of a video stream are known in the art and may include, for example, pressing a key on a keyboard, clicking on a handheld controller, clicking on a specific portion of a touchpad or screen, gestures, voice commands, etc., and combinations thereof.

[0020] Next, the method includes marking the region of interest in step 106. As described herein, marking the region of interest can be performed manually and / or with the assistance of an AI or ML-based model. Marking is preferably performed by a human, such as a surgeon, surgical assistant, nurse, or other medical professional, because not all anatomical structures are adequately identified by the model. Marking can be achieved by any suitable method, such as clicking a mouse on an area on the screen representing the region of interest, using a stylus, touching, gestures, or a combination thereof. As an example of this step, if the anatomical structure of interest is visible on the laparoscopic monitor, the surgeon holds the laparoscope still and takes a snapshot to obtain a still image. The surgeon then marks the anatomical structure of interest on the image using a touchpad or mouse. The surgeon may pause the surgery to mark the anatomical structure, or an assistant or nurse may draw the marking. Lines and filled shapes can be used to mark blood vessels and nerves, and polygons can be used to mark areas such as the pain triangle in hernia repair surgery. The region of interest is used to identify critical spaces around anatomical structures that are visible to the surgeon during surgery but should not come into contact with surgical instruments, or tight spaces into which the entry of surgical instruments creates a risk of inadvertent contact with anatomical structures. Therefore, it is advantageous for the surgeon to mark the region of interest in advance, so that during surgery, the region of interest can be avoided and surgery can be performed only at the surgical site. It should be noted that the region of interest may include multiple locations around the surgical site.

[0021] Next, the method of the present invention includes identifying regions of interest in all video streams in step 108. The method of the present invention advantageously uses Synchronous Localization and Mapping (sometimes abbreviated as SLAM in the industry, e.g., OrbSLAM-2) to track the camera's trajectory and build a map of the surgical scene as the laparoscope moves around the surgical site. Specifically, SLAM is a method of positioning the camera within a map as it is being built, allowing the camera to orient itself in an unknown environment. As the camera moves, a 3D structure of the surgical scene is gradually built. Using this intraoperatively built 3D model of the surgical scene, each point in the current laparoscope frame can be mapped to a 3D point on the 3D model. Meanwhile, each 3D point on the 3D model can be projected into the 2D space of the current frame if the projected position of the 3D point is outside the frame boundary, meaning that the 3D point is currently not visible. To re-identify the anatomical structures (regions of interest) marked by the surgeon, the 3D points of the marked structures are projected onto the current frame and visualized within the boundaries of the current frame. Depending on the camera angle, three cases are possible:

[0022] 1. The entire region of interest is visible and highlighted in the current frame. 2. Since only a portion of the region of interest is visible, only that visible portion is highlighted in the current frame. An additional virtual reality frame can be constructed that renders the invisible portion superimposed on the current frame. 3. The entire region of interest is outside the boundaries of the current frame and is not highlighted in the current frame. An additional virtual reality frame can be constructed that renders the invisible portion superimposed on the current frame.

[0023] In this way, in step 110, the markings are reproduced in all video streams where the region of interest is visible in real time, so the surgeon is always aware of the location of the anatomical structures while performing the surgery.

[0024] 2A is an exemplary representation of a hernia repair procedure employing the methods of the present invention, with line 202 demarcating safe and unsafe areas 204 and 206. Additionally, visual aids such as graphical indicators or emojis (e.g., smiling and crying faces) can be used to indicate the safe and unsafe areas, making them easier for the surgeon to identify during the procedure.

[0025] 2B is another exemplary display of a laparoscopic cholecystectomy using a method of one embodiment of the present invention, illustrating more subtle variations within the safe and unsafe regions demarcated by line 208. The safe regions are further marked with triangles (exemplary polygons) to indicate very safe regions D 210 and relatively safe regions I 212. Appropriate color selection may also be applied to allow quick identification by the surgeon. Similarly, the unsafe regions are further marked with a first triangle marked "pain" 214 (which may use a color representing a mild risk region), a second triangle marked "fracture" 216 (which may be colored an appropriate color, such as red, representing a severe risk), and a third triangle marked F 218.

[0026] Next, in step 112 of a method according to an embodiment of the present invention, the location of the surgical tool tip is identified. This can be the tip of a surgical knife or an energy device, for example. This is preferably achieved using a trained AI / ML / CV model to ensure speed and accuracy. In some cases, a neural network is trained to detect various surgical tools, such as a surgical knife or an energy device.

[0027] FIG. 3 shows an exemplary surgical instrument tip 302 .

[0028] Next, in step 114, the spacing between the surgical tool tip and the markings is estimated. If the spacing is below a threshold, or if the position of the surgical tool tip overlaps with the markings, this indicates potential contact between the surgical tool tip and the anatomical structure. This can be problematic as it can lead to further complications, unexpected damage, etc. The surgeon is alerted in the form of a visual alert on the screen (e.g., a colored flashing light or pop-up text), an audio alert, or a combination thereof.

[0029] 4, which is a block diagram of an apparatus, generally designated 400, in accordance with one embodiment of the present invention. The apparatus 400 includes at least one imaging device 402 configured to capture video including a plurality of frames. The video may be of any portion of the body, such as the gastrointestinal tract or the urinary tract. The apparatus 400 also includes a medical image processing system 404, which includes a memory 406, a display unit 408, a marking tool 410, a processor 412, and a communication interface and software for performing the prescribed functions of the present invention.

[0030] As used herein, processor 412 may include one or more processors, controllers, control modules, or other processing devices. Processor 412 may be implemented using a general-purpose or special-purpose processing engine, such as a microprocessor, controller, GPU (graphics processor unit), FPGA (field programmable gate array), or other control logic. Processor 412 is connected to a bus or other communication medium to facilitate interaction with other components shown herein or external communication.

[0031] Memory 406 includes non-volatile memory (e.g., one or more hard disk drives) and volatile memory (e.g., random access memory (RAM)). Memory 406 may also include other memory modules. For example, random access memory (RAM) or other dynamic memory may preferably be used to store information and instructions for execution by processor 412. A hard disk drive, solid state drive, or other primary storage device may also be used to store temporary variables or other intermediate information while processor 412 executes instructions. Additionally, read-only memory ("ROM") or other static memory devices may be used to store static information and instructions for processor 412.

[0032] Memory 406 may also include one or more of various types of information storage mechanisms, which may include, for example, a media drive and a storage unit interface. A media drive may include a drive or other mechanism for supporting fixed or portable storage media. For example, a hard disk drive, a floppy disk drive, a tape drive, an optical disk drive, a CD or DVD drive (R or RW), or other portable or fixed media drive may be provided. Thus, a storage medium may include, for example, a hard disk, a floppy disk, a tape, a cassette, an optical disk, a CD or DVD, or other portable or fixed media that is read, written, or accessed by a media drive. As can be seen from these examples, a storage medium may include a computer-usable storage medium having computer software or data stored thereon.

[0033] In alternative embodiments, memory 406 may include other similar tools that allow computer programs or other instructions or data to be loaded into appropriate computing modules. Such tools may include, for example, fixed or portable storage units and interfaces. Examples of such storage units and interfaces include program boxes and box interfaces, portable memory (e.g., flash memory or other portable memory modules) and memory slots, PCMCIA slots and cards, and other fixed or portable storage units and interfaces that allow software and data to be transferred from the storage unit to the computing modules.

[0034] The device is physically located in the operating room and installed in a console connected to the console of the surgical system. The device is configured to perform real-time inference of AI / ML / CV models. The device also records tracking, SLAM, and detection results generated during surgery and stores the information as a system log on disk. The device may be a standalone device that includes a tracking and alert system provided within a physical device as envisioned in one embodiment of the present invention. Alternatively, the device may be provided as software and integrated into existing infrastructure in a suitable environment, such as a hospital.

[0035] When components or modules of the technology are implemented in whole or in part in software, in one embodiment, these software elements may be implemented to work in conjunction with computing or processing modules capable of performing their associated functions. After reading this specification, it will become apparent to one skilled in the relevant art how to use other computing modules or architectures to implement the technology. A computing module may represent, for example, the computing or processing power found in a desktop computer, a notebook computer, a handheld computing device (such as a PDA, smartphone, mobile phone, palmtop computer, etc.), a mainframe, a supercomputer, a workstation or server, or any other type of special-purpose or general-purpose computing device that is desirable or appropriate in a given application or environment. A computing module may also represent computing power embedded within a given device or computing power that is otherwise available in a given device. For example, a computing module may be found in other electronic devices such as digital cameras, navigation systems, mobile phones, portable computing devices, modems, routers, WAPs, terminals, and other electronic devices that may include some form of processing power.

[0036] The computing module may also include a communications interface for transmitting software and data between the computing module and external devices. Examples of communications interfaces include a modem, a network interface (e.g., Ethernet, network interface card, WiMedia, IEEE802.XX, or other interface), a communications port (e.g., a USB port, an IR port, an RS232 port, a Bluetooth® interface, or other port), or other such communications interface. The software and data transmitted over the communications interface are typically carried on signals, which may be electronic, electromagnetic (including optical), or other signals exchangeable by a given communications interface. These signals may be provided to the communications interface via wired or wireless communications media. Examples of channels include telephone lines, cellular lines, RF lines, optical lines, network interfaces, local or wide area networks, and other wired or wireless communications channels.

[0037] A computing module also includes input / output (I / O) devices, including a keyboard, display, pointing device, etc., which may be coupled to the device directly or through intermediate I / O controllers.

[0038] The terms "computer program medium" and "computer usable medium" typically refer to media such as memory, storage units, and media. These and various other forms of computer program medium or computer usable medium may be involved in carrying one or more sequences of one or more instructions to a processing device for execution. Such instructions embodied on the medium are typically referred to as "computer program code" or a "computer program product" (which may be grouped in the form of a computer program or other grouping). When executed, such instructions may cause a computing module to perform the features or functions of the disclosed technology discussed herein. Computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as the C programming language or similar programming languages.

[0039] As described above, while various embodiments of the disclosed technology have been described, it should be understood that the embodiments are illustrative only and not limiting. Similarly, various figures depict exemplary architectures or other configurations of the disclosed technology, which are helpful in understanding features and functionality that may be included in the disclosed technology. The disclosed technology is not limited to the exemplary architectures or configurations shown, and various alternative architectures and configurations can be used to achieve desired features. In fact, it will be apparent to those skilled in the art how to implement alternative functional, logical, or physical divisions and configurations to achieve the desired features of the technology disclosed herein. Furthermore, many different component module names other than those described herein may be applied to the various divisions. Furthermore, with respect to flowcharts, operational descriptions, and method claims, the order of steps presented herein does not require that various embodiments perform the described functions in the same order, unless the context dictates otherwise.

[0040] While the disclosed technology above has been described based on various exemplary embodiments and implementations, it should be understood that various features, aspects, and functions described in one or more individual embodiments do not have applicability limited to the particular embodiments described, but may be applicable alone or in various combinations to one or more other embodiments of the disclosed technology, even if such an embodiment is not described or even if such feature is not presented as part of a described embodiment. Thus, the scope of the technology disclosed herein should not be limited by any of the exemplary embodiments described above.

[0041] Terms and phrases used herein, and variations thereof, unless expressly stated otherwise, should be interpreted in an inclusive rather than limiting sense. For example, the term "including" should be interpreted as "including, but not limited to," or similar meanings; the term "exemplary" provides illustrative examples of the items being discussed, not an exhaustive or limiting list thereof; and the term "one" or "an" should be interpreted as "at least one," "one or more," or similar meanings. Adjectives of similar meaning, such as "conventional," "traditional," "usual," "standard," and "known," should not be interpreted as limiting the described items to items available during a particular period or at a particular time, but should be interpreted as including conventional, traditional, usual, or standard technology that is available or known at any time now or in the future. Similarly, when this specification refers to technology that would be obvious or known to one of ordinary skill in the art, that technology includes technology that would be obvious or known to that art at any time now or in the future.

[0042] Although overarching terms and phrases such as "one or more," "at least," "including but not limited to," or other similar phrases are used in some instances, they should not be construed as requiring or intended to imply a narrower context in the absence of such overarching phrases. The use of the term "module" does not imply that all components or functionality described or claimed as part of the module are located in one common package. Indeed, any or all of the various components of a module (whether control logic or other components) may be combined in a single package, maintained separately, or further divided into multiple groups or packages or distributed in multiple locations.

[0043] Furthermore, various embodiments described herein are illustrated with reference to exemplary block diagrams, flow charts, and other diagrams. After reading this specification, one of ordinary skill in the art will understand that the illustrated embodiments and their various alternatives can be implemented without being limited to the illustrated examples. For example, the block diagrams and their accompanying descriptions should not be construed as mandating a particular architecture or configuration.

Claims

1. acquiring at least one video stream during surgery; taking a snapshot of the video stream; marking an area of ​​interest on the snapshot; identifying the presence or absence of said region of interest in a subsequent video stream; and drawing a marking in the subsequent video stream at any location where the region of interest is located to notify a surgeon of the location of the region of interest.

2. The method of claim 1 , wherein the marking is performed by a human.

3. The method of claim 1 , further comprising identifying the location of the surgical instrument tip in the subsequent video stream.

4. The method of claim 3 , further comprising providing an alert if the spacing between the tip of the surgical instrument and the marking falls below a threshold.

5. The method of claim 3 , wherein the location of the surgical instrument tip is identified by artificial intelligence techniques.

6. The method of claim 1 , wherein the marking is a line or a solid shape.

7. The method of claim 4 , wherein the alert is at least one of a visual alert, an audio alert, and a combination thereof.

8. The method of claim 1 , wherein the video stream is used to construct a three-dimensional model using synchronous positioning and mapping techniques.

9. The method of claim 1 , wherein the region of interest includes at least one anatomical structure or a space surrounding the anatomical structure.

10. an imaging device configured to transmit a video stream; a display unit for displaying said video stream; Marking tools and a memory in which the model is stored; a processor, The processor: receiving the video stream; Take a snapshot of a specific video stream, communicating with the marking tool; receiving instructions from a user to draw markings on the snapshot in areas of interest; using the model to identify the region of interest on a subsequent video stream by drawing the markings; An intra-operative device configured to re-mark the region of interest on the subsequent video stream.

11. The apparatus of claim 10 , wherein the subsequent video stream is rendered using a three-dimensional model employing synchronized positioning and cartography techniques.

12. The apparatus of claim 10 , wherein the model is configured to identify a tip of a surgical instrument.

13. The apparatus of claim 12 , wherein the processor is further configured to identify a spacing between the tip of the surgical instrument and a marking in the region of interest.

14. The apparatus of claim 13 , wherein the processor is configured to provide an alert based on the interval being less than a predefined threshold.

15. 15. The device of claim 14, wherein the alert is at least one of a visual alert, an audio alert, and a combination thereof.

Images