Interactive Endoscopy for Intraoperative Virtual Annotation in VATS and Minimally Invasive Surgery
The interactive endoscopy system addresses the challenges of VATS by real-time image analysis and annotation, improving surgical precision and safety in lung cancer resection.
Patent Information
- Application Number
- JP2022516050
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-09-12
- Filing Date
- 2020-09-11
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-09-11
AI Technical Summary
Locating and resecting lung cancer tumors with safe margins while preserving healthy lung tissue remains challenging in minimally invasive surgeries like VATS, due to the 3D mismatch between preoperative CT scans and actual lung conditions, complex lung anatomy, and difficulty in identifying small tumors.
An interactive endoscopy system that uses a controller to analyze intraoperative images, detect anatomical features, and superimpose annotations on the images in real-time, aiding surgeons in identifying landmarks and tumors during VATS.
Enhances surgical precision by providing real-time anatomical guidance, reducing the time and uncertainty in surgical workflows, and ensuring complete tumor removal while minimizing healthy tissue loss.
Smart Images

Figure 0007577737000001 
Figure 0007577737000002 
Figure 0007577737000003
Abstract
Description
[Technical field]
[0001]
[0001] An interventional medical procedure is an invasive procedure that involves an intrusion into the patient's body. Surgery is an example of an interventional medical procedure and is suitable for treating early-stage lung cancer tumors. Endoscopy is also increasingly being used at various stages of lung cancer surgery. In lung cancer surgery, a fundamental pre-stage to tumor resection (removal) is the "surgical exploration" stage, where a medical professional such as a surgeon examines the lung tissue to mentally associate anatomical knowledge and preoperative imaging such as computed tomography (CT) with live endoscopic (e.g., thoracoscopic) video of the lung tissue. The surgical exploration stage helps clinicians to ensure that the entire lung cancer tumor can be removed during the resection stage, while avoiding the resection of healthy lung tissue, which would lead to reduced lung function. Surgical exploration using a thoracoscopic system provides familiarity with the lung tissue that is not usually obtained from preoperative imaging such as CT. This is because preoperative imaging mainly shows differences in attenuation of the lung tissue and highlights anatomical landmarks. During surgical exploration, clinicians view through a thoracoscope while manipulating the lung tissue with instruments to identify anatomical landmarks to facilitate resection. Specifically, during surgical exploration, clinicians attempt to identify known anatomical landmarks, such as blood vessels and airways, near the lung cancer tumor to properly orient the anatomical structures to locate the lung cancer tumor and avoid damaging the blood vessels and airways during resection. [Background technology]
[0002]
[0002] The most invasive form of lung cancer surgery is open surgery, in which the chest is opened to expose a large portion of the lung. In open surgery, surgical tools such as scalpels, electric scalpels, and sutures are inserted into large openings in the chest and used to remove lung cancer tumors. Until now, lung cancer tumors were large enough to be felt, allowing clinicians to find lung cancer tumors that were invisible to the naked eye. Open surgery techniques allow physical access for palpation, which is the ability to feel the tumor by touch. Summary of the Invention [Problem to be solved by the invention]
[0003]
[0003] In recent years, both the detection of embedded lung cancer tumors and the techniques for surgically removing lung cancer tumors have improved. For example, lung cancer screening programs now tend to identify early tumor nodules that are small and difficult to identify. Furthermore, a minimally invasive technique for removing lung cancer tumors, called video-assisted thoracic surgery (VATS), has emerged as an alternative to open surgery.
[0004]
[0004] Locating and resecting lung cancer tumors with safe margins while sparing healthy lung tissue remains a key aspect of successful VATS. In VATS, a small camera is inserted into the thoracic cavity through a small port (i.e., a small hole or incision) and surgical instruments are inserted through the same port or other small ports. The entire surgical resection is performed using the camera view. FIG. 1 illustrates a conventional VATS embodiment for lung resection. In FIG. 1, a thoracoscope is inserted through the thorax of patient P as one of the instruments. Through the thoracoscope, vision that would otherwise be blocked can be restored. In FIG. 1, instruments #1 and #2 are inserted separately into patient P through two small incisions to perform the resection. In recent years, robotic surgery has emerged as a minimally invasive approach similar to VATS, competing with VATS.
[0005]
[0005] However, there are still three major challenges in lung cancer surgery. First, the location of the lung cancer tumor is determined based on a preoperative CT scan with the lung fully inflated well before the operation. Therefore, during the subsequent operation, when the lung is deflated, the three-dimensional (3D) orientation of the lung and the location of the lung cancer tumor will not match the preoperative CT scan image used for planning. Second, the lung is complex with many blood vessels and airways that need to be carefully cut and addressed before the lung cancer tumor and the feeding airways or blood vessels are removed. Third, small and non-palpable lung cancer tumors are difficult to identify and find, especially using VATS or robotic surgery, so there is a possibility of removing more healthy lung tissue than necessary during the procedure to prevent the possibility of leaving lung cancer tumor tissue.
[0006]
[0006] To overcome these challenges, ways to improve the surgical workflow have been explored to better guide the resection of lung cancer tumors. Surgical exploration in VATS is time-consuming, uncertain and non-quantifiable, making many aspects of the procedure difficult to reproduce. Moreover, the interactive endoscopy for intraoperative virtual annotation in VATS and minimally invasive surgery described herein addresses these challenges. [Means for solving the problem]
[0007]
[0007] According to an aspect of the present disclosure, a controller for live annotation of an intervention image includes a memory storing software instructions and a processor executing the software instructions. The software instructions, when executed by the processor, cause the controller to perform a process including receiving an intervention image during an intraoperative intervention and automatically analyzing the intervention image for detectable features. The process performed when the processor executes the software instructions also includes detecting the detectable features and determining to add annotations to the intervention image for the detectable features. The process performed when the processor executes the software instructions further includes identifying a location for annotation as a identified location in the intervention image and adding annotations to the intervention image at the identified location to correspond to the detectable features. A video based on the intervention image and annotations, including the annotations superimposed on the intervention image at the identified location, is output as a video output during the intraoperative intervention.
[0008]
[0008] According to another aspect of the present disclosure, a system for live annotation of an intervention image includes an electronic display and a controller. The intervention image is displayed on the electronic display. The controller includes a memory for storing software instructions and a processor for executing the software instructions. The software instructions, when executed by the processor, cause the controller to perform a process including receiving the intervention image and automatically analyzing the intervention image for detectable features. The process performed when the processor executes the software instructions also includes detecting the detectable features and determining to add annotations to the intervention image for the detectable features. The process performed when the processor executes the software instructions further includes identifying a location for annotation as a identified location in the intervention image and adding annotations to the intervention image at the identified location to correspond to the detectable features. A video based on the intervention image and annotations, including the annotations superimposed on the intervention image at the identified location, is output as a video output to the electronic display during the intraoperative intervention.
[0009]
[0009] According to yet another aspect of the present disclosure, a method for live annotation of an intervention image includes receiving an intervention image from a thoracoscope that generates the intervention image as a view inside the chest during a video-assisted thoracoscopic procedure for lung tumor resection. The method further includes automatically analyzing the intervention image for detectable features by a processor executing software instructions from a memory. The method further includes detecting the detectable features and determining to add annotations to the intervention image for the detectable features. The method further includes identifying a location for annotation as a specified location in the intervention image and adding annotations to the intervention image at the specified location to correspond to the detectable features. A video based on the intervention image and annotations, including the annotations superimposed on the intervention image at the specified location, is output as a video output during the video-assisted thoracoscopic procedure. [Brief description of the drawings]
[0010]
[0010] Example embodiments are best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that the various features are not necessarily drawn to scale. In fact, dimensions have been arbitrarily increased or decreased for clarity of discussion. Where applicable and practical, like reference numerals refer to like elements.
[0011] [Figure 1] FIG. 1 illustrates a conventional video-assisted thoracic surgery (VATS) implementation for lung resection. [Figure 2A]
[0012] FIG. 2A illustrates an intraoperative virtual annotation method in VATS and minimally invasive surgery according to a representative embodiment. [Figure 2B]
[0013] FIG. 2B illustrates another intraoperative virtual annotation method in VATS and minimally invasive surgery according to a representative embodiment. [Figure 2C]
[0014] FIG. 2C illustrates another intraoperative virtual annotation method in VATS and minimally invasive surgery according to a representative embodiment. [Figure 2D]
[0015] FIG. 2D illustrates another intraoperative virtual annotation method in VATS and minimally invasive surgery according to a representative embodiment. [Figure 2E]
[0016] FIG. 2E illustrates another intraoperative virtual annotation method in VATS and minimally invasive surgery according to a representative embodiment. [Figure 2F]
[0017] FIG. 2F illustrates another intraoperative virtual annotation method in VATS and minimally invasive surgery according to a representative embodiment. [Diagram 3]
[0018] FIG. 3 illustrates another intraoperative virtual annotation method in VATS and minimally invasive surgery according to a representative embodiment. [Figure 4]
[0019] FIG. 4 illustrates three examples of virtual annotations for intraoperative virtual annotation in VATS and minimally invasive surgery according to a representative embodiment. [Figure 5A]
[0020] FIG. 5A illustrates a system for interactive endoscopy for intraoperative virtual annotation in VATS and minimally invasive surgery according to a representative embodiment. [Figure 5B]
[0021] FIG. 5B illustrates a controller for interactive endoscopy for intraoperative virtual annotation in VATS and minimally invasive surgery in accordance with a representative embodiment. [Figure 6]
[0022] FIG. 6 illustrates a general computer system in which a method for intraoperative virtual annotation in VATS and minimally invasive surgery according to another representative embodiment is implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012]
[0023] In the following detailed description, for purposes of explanation and not limitation, representative embodiments disclosing specific details are described to provide a thorough understanding of the embodiments according to the present teachings. Descriptions of known systems, devices, materials, methods of operation, and methods of manufacture may be omitted so as not to obscure the description of the representative embodiments. However, systems, devices, materials, and methods within the scope of those skilled in the art are within the scope of the present teachings and may be used in accordance with the representative embodiments. It is to be understood that the terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. Defined terms are to be given the technical and scientific meaning of the defined terms as commonly understood and accepted in the art of the present teachings.
[0013]
[0024] In this specification, terms such as "first," "second," and "third" are used to describe various elements or components, but it should be understood that these elements or components are not limited by these terms. These terms are used only to distinguish one element or component from another. Thus, a first element or component discussed below may be referred to as a second element or component without departing from the teachings of the inventive concept.
[0014]
[0025] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. As used in this specification and the appended claims, the singular form is intended to include both the singular and the plural, unless the context clearly dictates otherwise. Furthermore, as used herein, the terms "having," "including," and / or similar terms specify the presence of stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.
[0015]
[0026] Unless otherwise noted, when an element or component is said to be "connected," "coupled," or "adjacent" to another element or component, it is understood that the element or component may be directly connected or coupled to the other element or component, or there may be intervening elements or components present. That is, these and similar terms encompass the cases where one or more intermediate elements or components may be used to connect the two elements or components. However, when an element or component is said to be "directly connected" to another element or component, this only encompasses the cases where the two elements or components are connected to each other without any intermediate or intervening elements or components.
[0016]
[0027] The present disclosure is intended to provide one or more of the advantages specifically set forth below through one or more of its various aspects, embodiments, and / or specific features, subcomponents. For purposes of explanation and not limitation, example embodiments disclosing specific details are described to provide a thorough understanding of the embodiments according to the present teachings. However, other embodiments consistent with the present disclosure that depart from the specific details disclosed herein remain within the scope of the appended claims. Furthermore, descriptions of well-known devices and methods may be omitted so as not to obscure the description of the example embodiments. Such methods and devices are within the scope of the present disclosure.
[0017]
[0028] Interactive endoscopy for intraoperative virtual annotation in VATS and minimally invasive surgery, as described herein, provides an interactive annotation system that interactively annotates live video during surgery. The annotations can be viewed during the interventional medical procedure. Interactive endoscopy further provides informative annotations to assist the medical procedure personnel by annotating persistent features (anatomical or otherwise), tracking the annotated persistent features via annotations when found, and noting tasks that have already been performed.
[0018]
[0029] 2A-2F illustrate intraoperative virtual annotation methods in VATS and minimally invasive surgery according to various representative embodiments. Typically, these methods are performed during an interventional medical procedure using, for example, interactive endoscopy, where a thoracoscope and one or more surgical instruments are inserted into a patient. When inspecting tissue during an interventional medical procedure, the clinician reviews a display of an interventional image (e.g., an endoscopic image) that provides information such as anatomical features of interest and / or surgical events. The interventional image is, for example, a video provided by an endoscopic camera during VATS. According to various embodiments, all or part of this information can be annotated on the display using an interactive annotation system, for example, by overlaying virtual annotations on the interventional image to visualize the surgical field. The annotations can be placed at locations identified by the clinician or the interactive annotation system based on machine learning applied to previous instantiations. The location where the annotation is placed should be contextually meaningful, for example, placing a vascular annotation where a vessel appears in the interventional image.
[0019]
[0030] Additionally, the clinician issues one or more commands to trigger the entire annotation workflow or individual aspects of the annotation workflow. The clinician continues the interventional medical procedure, adding annotations as needed. Once the clinician identifies a feature or event to annotate, the clinician designates the feature or event to a software control system of the interactive annotation system, which assists in determining where and how to apply the annotation. In additional embodiments, the software control system may apply image processing algorithms to adjust the position and orientation of the annotation with respect to the corresponding image location in response to tissue and / or endoscope movement.
[0020]
[0031] 2A illustrates a method for intraoperative virtual annotation in VATS and minimally invasive surgery according to a representative embodiment. The method of FIG. 2A may be implemented, for example, using a controller 522 in a system 500 for interactive endoscopy of FIG. 5B, described below.
[0021]
[0032] Referring to FIG. 2A, in S210, an interventional image is received during an intraoperative intervention. The interventional image is captured using an imaging device, such as an endoscopic camera of an endoscope that provides an endoscopic image. For example, the endoscopic image is a video captured during an interventional medical procedure using an endoscope. Another example of an imaging device is a small camera positioned to focus on a particular part of an anatomical structure or tool during spinal surgery or a similar procedure. In this case, the interventional image is a camera image, such as a video of an anatomical structure or tool. The interventional image can be acquired from outside the patient or, in the case of an endoscopic image, from inside the patient.
[0022]
[0033] In response to the first command, in S220, the intervention image is automatically analyzed for detectable features. The detectable features include, for example, the patient's anatomical structure and / or the interventional tool used during the interventional medical procedure. The analysis of the detectable features in S220 can be performed using one or more video analysis algorithms, such as an algorithm for detecting and tracking low-level features in the image (e.g., video frame). For example, the first video analysis algorithm for detecting and tracking low-level features can include a scale-invariant feature transform (SIFT) algorithm, a speed-up robust feature (SURF) algorithm, an optical flow algorithm, or a learned feature algorithm. A second video analysis algorithm can be used to recognize and classify an anatomical structure of interest from the extracted low-level features. A third video analysis algorithm can be used to calculate tissue deformation, such as rotation around any of three axes or linear translation in any of three axes. The three axes are set relative to a fixed object or location.
[0023]
[0034] The automation of the functional features described herein can be based on machine learning, such as deep learning algorithms. The machine learning can be performed centrally for multiple different individual systems, such as system 500 of FIG. 5A, and can be performed in a cloud-based processing system, such as a data center. Alternatively, the machine learning can be performed centrally in a dedicated central computer system for multiple different individual systems that are, for example, geographically distributed, but associated with an entity that provides the dedicated central computer system.
[0024]
[0035] Instantiations of interventional images can be subject to machine learning to identify patterns and correlations. Results of the machine learning can be used to optimize aspects of the teachings herein, such as the analysis of detectable features in S220. Additionally, the interactive annotation system can be provided with configuration information prior to an interventional medical procedure to aid in automation, for example, by helping reduce processing demands. For example, the configuration information can be the type of medical intervention, such as a lung resection, so that the interactive annotation system analyzing the endoscopic images knows to look for the type of anatomical features typically found in the lung. Alternatively, the interactive annotation system can automatically recognize the context, such as the environment of the lung, thereby narrowing the analysis for detectable features to within or around the lung.
[0025]
[0036] In S230, detectable features in the intervention image are detected. To the extent that the detectable features are anatomical structures or tools, one or more image recognition algorithms are applied to detect the one or more detectable features. The tools detected as detectable features may be, for example, forceps, sutures, staples secured to the anatomical structure, or other implantable devices. The detection of the detectable features in S230 is based on recognition of one or more of the shape, color, relative position, and / or other characteristics of the detectable features.
[0026]
[0037] In response to the second command, it is determined to add an annotation of the detectable feature to the intervention image in S240. The second command received in S240 reflects a positive decision to add an annotation of the detectable feature detected in 230 and is received as a result of a prompt such as highlighting the detectable feature or an outline of the detectable feature on the screen after S230 or before S240. In response to the third command, a location for the annotation is automatically identified in the intervention image in S250. The identification of the location in S250 includes identifying a distance and direction placement for the corresponding detected feature, and also considers the context of the detected feature relative to the tumor, other anatomical features, and / or other tools in the intervention image. In response to the fourth command, an annotation is automatically added to the intervention image at the identified location to correspond to the detectable feature. The fourth instruction received in S260 reflects confirmation that the identified location is acceptable and the decision to add the annotation in S240. The first, second, third, and fourth instructions are, for example, software instructions provided by the controller 522 automatically and / or in response to input by a clinician.
[0027]
[0038] In S270, a video including the intervention image and the annotations overlaid on the intervention image at the identified locations is output. Overlaying the annotations on the endoscopic image is performed using, for example, an image overlay function such as that found in the open source software library OpenCV.
[0028]
[0039] In a non-illustrated embodiment, the features of S210-S270 are executed in a loop, so that even if a video output is output in S270, the interventional images are continuously received and analyzed in S210 and S220. Based on the features of S210-S270, the interaction between the clinician performing the interventional medical procedure and the interactive annotation system includes receiving any of various commands and performing a corresponding function based on the command. In this manner, the clinician performing the interventional medical procedure can interactively control the annotations on the interventional images in a readily accessible manner. Furthermore, the features of S210-S270 are performed by the interactive annotation system provided entirely in the space in which the interventional medical procedure is performed, and in a continuous, uninterrupted time frame from when the interventional medical procedure begins with the initial insertion of a medical device (e.g., a thoracoscope) into the patient to when the interventional medical procedure ends with the medical device being removed from the patient. Alternatively, the features of S210-S270 may be performed from the time the interactive annotation system is started, such as by command, until the interactive annotation system is stopped, such as by order from a clinician performing the interventional medical procedure.
[0029]
[0040] In S280, machine learning is applied to the interventional images overlaid with annotations, and optimal placement of the annotations is identified based on the machine learning. In S280, machine learning is used to optimize the placement of the annotations, but machine learning can also be used to understand the task or image content of the interventional medical procedure and / or adapt the user interface to improve the functionality of the interactive annotation system. The process of S220-S280 can be performed in a loop for multiple medical interventions.
[0030]
[0041] A feature unique to S280 reflects the actual placement of annotations. The actual details of the annotations and placements are pooled with the actual details of other instantiations, allowing for averages such as distance from relevant detectable features and identifying whether the annotations interfere with other features of the interventional image in potentially problematic ways. As an example, to learn optimized annotation placements, machine learning is applied to obtain quantitative metrics of existing pairs of anatomical features and virtual labels. For example, quantitative metrics include the distance between the anatomical features and the virtual labels. As another example, quantitative metrics include the area or volume of a particular type of virtual label or the area or volume between a particular type of anatomical feature and a virtual label.
[0031]
[0042] The machine learning applied in S280 is just one example of machine learning that may be applied in accordance with the teachings herein. Another example of machine learning in an interactive annotation system function is feature detection, such as in S220 and S230. Feature detection includes learning how and where to find detectable patterns or anatomical features. For example, multiple instantiations of previous endoscopic images may be examined to identify commonalities related to how and where users place annotations for a particular type of detectable feature. Feature detection may also include learning patterns in how users place annotations in the context of a particular type of medical intervention.
[0032]
[0043] Machine learning for understanding tasks and image content in interventional medical procedures can be used for many purposes. For example, feature detection can be used, such as in S220 and S230, to learn to find clinically relevant anatomical features and landmarks. Machine learning can also be used, such as in S250, to learn the annotations used in a given interventional medical procedure and when / where these annotations are placed. Machine learning can also be used to learn annotations that correspond to surgical events or actions, the timing and order in which annotations are placed, and patterns in surgical workflow that reflect the thought process of personnel involved in a medical intervention. Machine learning can also be used to learn when annotations become invisible, for example, when annotations for an anatomical structure disappear when that anatomical structure is part of an inverted or rotated organ. Machine learning can also be used to identify relationships between similar annotations used in different interventional medical procedures.
[0033]
[0044] Examples of machine learning for adapting user interfaces can be used to identify which annotations should be used for specific anatomical structures. Machine learning for adapting user interfaces can also be used to learn details such as the appropriate size and orientation of annotations, as well as the size and font of text annotations. Machine learning for user interfaces can also be applied to learn the number of annotations to use, such as the maximum or optimal number. Machine learning for interfaces can also identify patterns when certain annotations or annotation types are explicitly removed, such as when an interventional medical procedure practitioner determines that annotations are not necessary. Machine learning can also be used to identify user preferences to customize annotation options for a particular user.
[0034]
[0045] Machine learning can be used in the form of a feedback loop, where annotation features at one interventional medical procedure reflect machine learning optimization at a previous interventional medical procedure, i.e. annotation at one interventional medical procedure is based on machine learning applied to at least one previous instantiation of live annotation, and intervention images from that one interventional medical procedure are then considered and incorporated into the machine learning and applied to optimize a subsequent interventional medical procedure.
[0035]
[0046] As mentioned above, in the method of FIG. 2A, detectable features are detected based on an automated analysis of the intervention image. Virtual annotations are placed on the intervention image to correspond to the detectable features, and the annotated intervention image is output as an annotated video including the intervention image and the annotations. The annotations act as a kind of augmented reality. For example, the annotated intervention image is used as a roadmap, which eliminates the need for the clinician to repeatedly detect and identify the same detectable features. Of course, the annotations are not limited to detectable features from anatomical structures. The annotated intervention image can also be used, for example, to identify tools or to provide a checklist of tasks or other information completed during the procedure.
[0036]
[0047] As discussed above, an interactive annotation system can be used to control interactive endoscopy for intraoperative virtual annotation. The interactive annotation system provides functional features including at least analyzing an input live surgical video to detect detectable features, accepting dynamic command inputs to add virtual annotations (e.g., labels) to the live surgical video, determining locations within the video to add the annotations, and creating an output video that includes the original surgical video with the annotations overlaid on it.
[0037]
[0048] Figure 2B illustrates another intraoperative virtual annotation method for VATS and minimally invasive surgery according to a representative embodiment. The method of Figure 2B may supplement the method of Figure 2A as needed, but individual features of the method of Figure 2B may replace individual features of the method of Figure 2A where mentioned or appropriate.
[0038]
[0049] 2B, in S231, anatomical structures are detected as detectable features. In this case, S231 replaces S230 in FIG. 2A as an example of the type of detectable features that are detected. In S232, boundaries of the detected anatomical structures are delineated. The boundaries may be delineated by graphically identifying and outlining the boundaries, such as by highlighting the color or another characteristic of the appearance of the anatomical structures at the boundaries.
[0039]
[0050] In S261, the anatomical structures are labeled as annotations with virtual labels. The labeling of the anatomical structures in S261 of FIG. 2B is an example of the automatic addition of annotations in S260 of FIG. 2A. The virtual labels may include markers with text, or numbers or icons within a shape, e.g., a predefined outline corresponding to the anatomical structure. The labels and other annotations may vary in semantic complexity. For example, annotations may indicate time, identify blood vessels, outline tumors, etc. Also, instructions for using the annotations may be provided by touch screen, voice, or automatically. The labeling in S261 may be performed after the features of FIG. 2A, e.g., the decision to add annotations in S240 and the identification of locations for annotations in S250. That is, the features of FIG. 2A described herein with respect to the embodiment of FIG. 2B may be included in the embodiment of FIG. 2B, if desired.
[0040]
[0051] Examples of annotatable information that can be included in annotations, such as virtual labels, include tags for various anatomical features, such as lung fissures, tumors, blood vessels, airways, and other organs. A virtual label may include multiple checklist items. In this case, surgical events such as incision, stapling, resection, grasping, flipping, and retraction are examples of annotatable information. Other examples of annotatable information include perioperative information, such as elapsed time and applied force. Detailed information can also be provided as annotatable information, such as detailed information evaluated based on intraoperative information and sometimes combined with external reference data, including various forms of physiological data, models, and imaging. Postoperative information, such as the exact location of staple lines and tumors, which can serve as inputs to future learning algorithms, can also be provided as annotatable information. Most of the annotatable information described herein can be used as live feedback in the form of a roadmap for medical personnel of an interventional medical procedure.
[0041]
[0052] At S271, the movement of the anatomical structure is tracked. S271 occurs after the video output is output at S270 in the method of FIG. 2A. For example, the movement of the anatomical structure is tracked when the anatomical structure is rotated or translated horizontally and / or vertically. The anatomical structure may also move relative to the viewpoint of an imaging device, such as an endoscopic camera, when the imaging device moves. At S272, the movement of the anatomical structure is detected relative to an imaging device, such as an endoscopic camera, that generates the intervention image. The movement of the anatomical structure may be tracked to ensure that the virtual label moves with the anatomical structure, since the movement of the anatomical structure essentially reflects the usefulness of the corresponding virtual label.
[0042]
[0053] In S273, upon detecting a movement of the tracked anatomical structure in the endoscope-related tracking, the position and orientation of the virtual label are automatically adjusted based on a measurement of a deformation of the corresponding anatomical structure, such as a rotation about any of three axes or a linear translation in a direction along any of three axes.
[0043]
[0054] Figure 2C illustrates another intraoperative virtual annotation method for VATS and minimally invasive surgery according to a representative embodiment. The method of Figure 2C may supplement the method of Figure 2A as needed, although individual features of the method of Figure 2C may replace individual features of the method of Figure 2A where mentioned or appropriate.
[0044]
[0055] Referring to FIG. 2C, in S221, a first algorithm is used to detect and track low-level features of the intervention image. S221 can be performed as part of the analysis in S220 of FIG. 2A. Low-level features include features that form relatively unique patterns in the details of the intervention image so that they can be interpreted as anatomical structures or tools. Low-level features include patterns that are not recognizable by humans, observable by humans, or comprehensible by humans with the naked eye. As previously mentioned, examples of algorithms for detecting and tracking low-level features of images, such as video frames, include a scale-invariant feature transform (SIFT) algorithm, a speed-up robust feature (SURF) algorithm, an optical flow algorithm, or a learned feature algorithm.
[0045]
[0056] In S222, the method of Figure 2C includes extracting low-level features as extracted low-level features. S222 can also be performed as part of the analysis of S220 of Figure 2A.
[0046]
[0057] The method of FIG. 2C continues at S223 with detecting detectable features by using a second algorithm to recognize and classify anatomical structures as detectable features of the endoscopic image based on the extracted low-level features. S223 can be performed as part of the detection at S230 of FIG. 2A. The second algorithm includes features such as comparing the low-level features detected by the first algorithm with known characteristics of known anatomical features, for example, by color, size, shape, proximity to other known characteristics, etc. Both the low-level features and the characteristics of the known anatomical features may have patterns that are meaningless to humans, patterns that are not observable by humans, or patterns that are not detectable by humans, such as by the naked eye, but are recognizable and detectable by the algorithm used for image processing. The second algorithm can rank the possible matches by the extent to which they match the characteristics of the known anatomical features, and select the top-ranked known anatomical features. The top-ranked known anatomical features of the low-level features can be classified as relevant or irrelevant, such as based on the type of medical intervention being performed. Alternatively, the classification may be of known anatomical feature type, such as bone, tissue, organ, airway, etc. The algorithm then selectively prompts the interventional medical procedure practitioner to annotate the recognized anatomical features based on the known anatomical feature type and its relevance to the interventional medical procedure being performed.
[0047]
[0058] At S224, the method of Figure 2C includes delineating boundaries of the anatomical structures. Delineating at S224 may be performed by overlaying solid or dashed lines to indicate the boundaries of the anatomical structures.
[0048]
[0059] As mentioned above, in FIG. 2C, image analysis and feature detection are performed using one or more algorithms for image analysis and feature classification. The process of FIG. 2C is performed, for example, for each of a plurality of frames in an endoscopic image. Image analysis and feature detection are performed throughout the interventional medical procedure after the endoscopic image is initially generated, or after a specific command is received to initiate image analysis and feature detection, or after annotations are specifically added. That is, the person performing the interventional medical procedure can dynamically decide to annotate features during the interventional medical procedure. Also, the processing at S220 of FIG. 2A or S221 of FIG. 2C is initiated based on the person performing the interventional medical procedure initiating the process described herein.
[0049]
[0060] Figure 2D illustrates another intraoperative virtual annotation method for VATS and minimally invasive surgery according to a representative embodiment. The method of Figure 2D may supplement the method of Figure 2A as needed, although individual features of the method of Figure 2D may replace individual features of the method of Figure 2A where mentioned or appropriate.
[0050]
[0061] Referring to Fig. 2D, in S241, predetermined information of the detectable feature is identified from an external source. For example, when it is determined to add an annotation in S240 in the process of Fig. 2A, a label, an icon, an information item, or other predetermined information is identified in S241. In S262, the predetermined information from the external source is added to the intervention image as an annotation. For example, when a fourth command is received in S260, as a modification or supplement to the automatic addition of the annotation in S260 of Fig. 2A, the predetermined information from the external source is added to the intervention image in the form of an annotation in S262.
[0051]
[0062] Figure 2E illustrates another intraoperative virtual annotation method for VATS and minimally invasive surgery according to a representative embodiment. The method of Figure 2E may supplement the method of Figure 2A as needed, although individual features of the method of Figure 2E may replace individual features of the method of Figure 2A where mentioned or appropriate.
[0052]
[0063] 2E, in S211, intraoperative information from an intraoperative intervention is analyzed. The intraoperative information includes details about the settings of the interventional medical procedure in which the processes described herein are performed. For example, the intraoperative information includes speech, gestures, or specific inputs by medical personnel of the interventional medical procedure. The intraoperative information also includes information output from medical devices such as monitors for heart rate or pulse rate, and the time the information was output. The analysis in S211 may be performed on a particular predefined type of intraoperative information or on multiple predefined types of intraoperative information.
[0053]
[0064] In S242, the content of the annotation is derived based on the analyzed intraoperative information. For example, the content of the annotation is derived from intraoperative information based on conversations and sounds in the operating room, and from images taken in the operating room, such as a camera, endoscope, or another mechanism for medical imaging. The content of the annotation is derived from intraoperative information based on electronic signals from equipment, such as signals indicating tools are on / off, lights are on / off or up / down, and a blood monitor alarming because the patient's blood pressure exceeds an upper or lower threshold. These are all examples of how the content of the annotation can be derived in real time based on the intraoperative information in S242 derived from the analysis in S241. The annotation itself is then placed as a note on the intervention image so that it is within the field of view of the medical personnel of the interventional medical procedure.
[0054]
[0065] At S263, the process of Figure 2E includes adding content to the endoscopic image as annotations. For example, upon receiving a fourth command at S260, content derived from intraoperative information is added in the form of annotations at S263 as an alteration or supplement to the automatic addition at S260 of Figure 2A.
[0055]
[0066] Figure 2F illustrates another intraoperative virtual annotation method for VATS and minimally invasive surgery according to a representative embodiment. The method of Figure 2F may supplement the method of Figure 2A as needed, although individual features of the method of Figure 2F may replace individual features of the method of Figure 2A where mentioned or appropriate.
[0056]
[0067] 2F, in S205, a pre-operative segmented model of the anatomical structure is obtained. For example, the pre-operative segmented model is obtained from a pre-operative computed tomography (CT) scan of the anatomical structure. The pre-operative CT scan is used to generate a model that is later segmented.
[0057]
[0068] At S264, instructions to annotate anatomical structures are received as annotations along with the pre-operative segmented model, i.e., the anatomical structures represented in the pre-operative segmented model from S205 correspond to the instructions of S264.
[0058]
[0069] At S265, the anatomical structure in the endoscopic image is virtually replaced with the pre-operative segmented model of the anatomical structure, i.e., at S265, a command is received to replace the live intervention image of the anatomical structure with the pre-operative segmented model of the anatomical structure acquired at S205.
[0059]
[0070] In S266, the pre-operative segmented model of the anatomical structure is registered to the anatomical structure. That is, in S266, the live interventional image of the anatomical structure is registered with the pre-operative segmented model of the anatomical structure acquired in S205. As a result, the annotations automatically added in S260 include registering and replacing the live interventional image of the anatomical structure with the pre-operative segmented model of the anatomical structure acquired in S205. This form of annotation aids the interventional medical procedure practitioner in the on-screen visualization of the features of the patient's anatomy to improve concentration. That is, like most or all forms of annotation described herein, the annotations in FIG. 2F can be used as live feedback for the interventional medical procedure clinician, as an aid similar to a roadmap or navigation tool and / or as an aid similar to a sticky note.
[0060]
[0071] Any two or more of the methods of Figures 2A-2F may be integrated within the scope of the present disclosure. Additionally, the methods of Figures 2A-2F may include additional features, such as storing detected features identified and tracked from the interventional images. The stored features may be used later. For example, annotations such as virtual labels are automatically provided again when the endoscopic camera view moves to a significantly different region of the organ and then returns to the original field of view. Some or all aspects of the methods and processes of the representative embodiments described above in connection with Figures 2A-2F may be performed by the system 500 of Figure 5A or solely by the controller 522 of Figure 5B executing software instructions.
[0061]
[0072] The stored features and annotations can also be used in subsequent interventional procedures, even on different days. For example, if one or more anatomical features are identified in a first interventional medical procedure and similar anatomical features are identified in a second interventional medical procedure, the similar features can be used to automatically align the position and orientation of additional labels from the first interventional medical procedure in the second interventional medical procedure. This allows for less processing and faster annotation of known features. For example, the borders of a tumor can be defined in the first and second interventional medical procedures. Also, the area and / or volume of the tumor can be quantified and the area difference calculated in each of the first and second interventional medical procedures.
[0062]
[0073] The stored features and annotations can also be used for post-interventional medical procedure reporting. For example, the number of features and the type of features can be stored in the patient's electronic medical record. Or, if the feature is the type of implant / tool / device used, the count can be managed for restocking of these implants / tools / devices in a purchasing system.
[0063]
[0074] Figure 3 illustrates another interactive endoscopy method for intraoperative virtual annotation in VATS and minimally invasive surgery according to a representative embodiment. Some or all aspects of the representative embodiment method and process described above in connection with Figure 3 may be performed by the system 500 of Figure 5A or solely by the controller 522 of Figure 5B executing software instructions.
[0064]
[0075] 3, in S310, an instrument is placed in a medical intervention environment, such as would be found in an interventional medical procedure. For purposes of the description herein, the instrument is placed in or around a patient's anatomy within the field of view of an imaging device. In S320, an anatomical feature of interest or an event to record is identified. Since identifying an anatomical feature of interest and / or an event to record for annotation is part of the process of ultimately annotating an intervention image, S320 represents the trigger of the annotation process described herein.
[0065]
[0076] In S330, the intent to annotate is input by the clinician via a touch screen, keyboard, buttons, roller pad, mouse, or other physical interface. The intent to annotate may also be input via an audible command or visual gesture detected by a speech or video recognition mechanism implemented by a processor executing software instructions. In S340, the desired annotations appearing in the video feed are identified. The identified annotations result from the intent input in S330 and are overlaid or visually integrated into the video feed in S340. The annotations are the result of the various processes described with respect to the embodiment of Figures 2A-2F. In S350, the clinician proceeds with another type of intervention in the surgery or interventional medical procedure for which the virtual annotations were provided. In one embodiment, the processes of S320-S350 can be performed in a loop during the interventional medical procedure.
[0066]
[0077] FIG. 4 illustrates three examples of interactive endoscopy virtual annotation for intraoperative virtual annotation in VATS and minimally invasive surgery according to a representative embodiment.
[0067]
[0078] Referring to FIG. 4, in the first annotation example shown on the left, the tumor is labeled "TUMOR", the blood vessel is labeled "VESSEL", and the airway is labeled "AIRWAY". In addition, to clearly designate the airway, a virtual structure is superimposed on the airway as an annotation. In the illustrated example, the label is offset from the tissue image, but this is not required. The label may be superimposed on the tissue image or on the margin of the intervention image. In the second annotation example, the anatomical surface is labeled "LOBULAR FISSURE" and the tumor margin is labeled "TUMOR MARGIN". In addition, a virtual structure is superimposed on the border of the tumor margin to clearly designate the tumor and tumor margin. In the first and second examples, the text labels may be replaced with simple icons such as "A", "B", "C", "D", and "E", or numbers "1" to "5". In the third annotation example, shown on the right, the annotation includes a freehand curved contour and a label "X" overlaid on the tissue image. The curved contour also has the label "ORGAN BOUNDARY" offset from the tissue image. Additionally, a label with a checklist is overlaid on the tissue image at the bottom right.
[0068]
[0079] Thus, Figure 4 illustrates several example methods for annotating interventional images. The various annotations may vary in size, type, location, color, text, shading, orientation, etc. Additionally, some annotations may be provided for use in certain types of medical procedures but not others. Additionally, annotations may be provided to specific individuals based on input from the individual that is incorporated into machine learning.
[0069]
[0080] Examples of annotation types provided as virtual annotations consistent with this disclosure include markers, labels, differentiated colors, lines and curves, CAD (Computer Aided Design) models, and drawings. Markers include dots, stars, squares, dashes, arrows, and other symbolic icons that convey location, direction, anatomical context, time, or other relevant information. Labels include text or other semantic representations that convey embedded information as tags. Differentiated colors are used to delineate tissue regions or represent other forms of information. Lines and curves can be used to indicate locations or boundaries of clinical interests and can be entered using a touch screen or the like. CAD models include segmented preoperative computed tomography (CT) images that represent anatomical structures such as tumors, blood vessels, airways, or other tissues of interest. Drawings allow the interventional medical procedure personnel to sketch at will on the video screen and update the marking locations per the video content. Some examples of the aforementioned marker types are shown in FIG. 4. Markers and other forms of annotation can also be mutable to convey transitions in space or time, or to convey temporal information such as interactions between tools and tissues.
[0070]
[0081] In the embodiment of Figure 4 and other embodiments herein, the location and / or shape of the virtual annotations can be updated with each video frame depending on the image content, e.g., in the second example, shaded areas of a particular color, such as green, will not be visible if the lung is inverted and the corresponding tissue is not in view.
[0071]
[0082] FIG. 5A illustrates a system for interactive endoscopy for intraoperative virtual annotation in VATS and minimally invasive surgery according to a representative embodiment.
[0072]
[0083] As shown in FIG. 5A, system 500 includes a thoracoscope 510, a computer 520, a display 530, and an input mechanism 540. Computer 520 includes a controller 522. System 500 is a simplified block diagram of what may otherwise be a highly complex environment for implementing the teachings herein. System 500 may include some or all of the components of the interactive annotation system described herein. System 500 implements some or all aspects of the methods and processes of the representative embodiments described herein and associated with FIGS. 2A-3.
[0073]
[0084] The thoracoscope 510 is an example of an endoscope. The thoracoscope 510 is a long, thin camera that is typically used for examination, biopsy, and / or resection within the thoracic cavity. The thoracoscope 510 transmits endoscopic images (e.g., video) to a computer 520 via a wired connection and / or via a wireless connection such as Bluetooth®.
[0074]
[0085] Computer 520 includes at least a controller 522, but may include any or all of the elements of an electronic device, such as seen in computer system 600 of Figure 6 described below. For example, computer 520 includes a port or other type of communication interface for interfacing with thoracoscope 510 and display 530. Controller 522 includes at least a memory that stores software instructions and a processor that executes the software instructions to directly or indirectly implement some or all aspects of the various processes described herein.
[0075]
[0086] Display 530 is a video display that displays endoscopic or other interventional images obtained from thoracoscope 510 and / or any other imaging equipment in the environment in which the interventional medical procedure is performed. Display 530 is a monitor or television that displays video in color or black and white, and may also have audio capabilities to output audio signals.
[0076]
[0087] The input mechanism 540 may be or include a mouse, keyboard, touchpad, tablet, microphone, video camera (e.g., for capturing gestures), or any other device for a clinician to input commands. Commands entered into the input mechanism 540 are used in the process of annotating endoscopic or other interventional images, as described herein. As shown, the input mechanism 540 may communicate with the computer 520, but may also communicate with the display 530, such as when the input mechanism is a touch screen that is on, in, or connected to the display 530. The input mechanism 540 may communicate with a wired or wireless connection, as described above, and may be part of or integrated with the display 530 and / or the computer 520.
[0077]
[0088] The input mechanism 540 can be provided in a variety of ways, such that the various methods can be selectively adapted to the particular context of the various types of information and annotations. The input mechanism 540 can be or include a personal computer mouse and keyboard, whereby the personal computer mouse can be used to hover over the location of a desired annotation, such as a pulmonary fissure visible on the lung surface, and to draw along the contiguous portion of the desired location, and the keyboard can be used to input text labels as annotations. The input mechanism 540 can also be or include a touch screen to perform tasks such as location selection and data entry without the use of a keyboard and mouse. External buttons attached to the instrument and / or elsewhere can be used as the input mechanism 540 to replicate the functionality of a mouse at a location convenient for the clinician performing the interventional medical procedure.
[0078]
[0089] The camera can be aligned and an image / video recognition software program can be used as an input mechanism 540 to recognize gestures. For example, in an example image / video recognition software program, moving a surgical tool in a particular pattern within the field of view of the endoscope can activate annotation placement. Other examples of gestures that can be recognized by the image / video recognition software program include, for example, opening and closing a gripper twice in succession, rotating an item back and forth, and tapping tissue. The recognized gesture can be used to encode the type of annotation, or the type of annotation can be provided by other means described herein. The input mechanism 540 can also be or include voice recognition software. In an example voice recognition software, a voice command can activate the placement, deletion, or modification of a virtual annotation while providing the type of annotation to use. The input mechanism 540 can also recognize physical labeling of tissue, such as with an image / video recognition software program. For example, the image / video recognition software program can recognize physical labeling indicating where to place a suture or cauterization marks on the tissue surface to identify a particular feature. As another example of the input mechanism 540, the virtual annotations are automatically placed or suggested based on models, machine learning or deep learning, or other data-driven methods. Machine learning examples described herein can analyze video content to recognize features and actions. For example, using machine learning applied to previous instantiations, dynamically labeled features during a particular medical intervention can be analyzed and detected and automatically labeled based on similar features that appear similar to those previously labeled. As another example of the input mechanism 540, partial or semi-automatic input of virtual annotations triggers "auto-completion" of annotations. For example, when drawing a blood vessel on an image, auto-completion can be performed, causing a pre-operative blood vessel model to be overlaid on the video using the image as an alignment guide.
[0079]
[0090] The machine learning may be provided to system 500, for example, by a central system that receives instantiations of virtual annotations from system 500 over the Internet or other network, and provides machine learning results based on instantiations of virtual annotations from multiple systems, including system 500, to system 500 over the Internet or other network. For example, the machine learning may be performed in a cloud-based processing system, such as a data center. Alternatively, the machine learning may be performed centrally in a dedicated central computer system, such as by an entity that has a relationship with system 500.
[0080]
[0091] FIG. 5B illustrates a controller for interactive endoscopy for intraoperative virtual annotation in VATS and minimally invasive surgery according to a representative embodiment.
[0081]
[0092] The controller 522 of FIG. 5B includes a processor 52210, a bus 52208, and a memory 52220. The controller 522 includes components for implementing some or all aspects of the methods and processes of the exemplary embodiments described above in connection with FIGS. 2A-3. The processor 52210 will be described in more detail below in the description of the processor of the computer system 600 of FIG. 6. The processor 52210 executes software instructions to implement some or all aspects of the methods and processes of the exemplary embodiments described above in connection with FIGS. 2A-3. The memory 52220 will be described in more detail below in the description of the memory of the computer system 600 of FIG. 6. The memory 52220 stores software instructions that the processor 52210 executes to implement some or all aspects of the methods and processes described herein. The bus 52208 connects the processor 52210 and the memory 52220. In Figure 5B, the controller 522 is shown as a stand-alone element, which indicates that the controller 522 need not be provided as part of the computer 520 of Figure 5B. Rather, the controller 522 described herein may be provided as a stand-alone element or may be integrated into a variety of devices, such as a dedicated item of medical technology, a laptop or desktop computer, or a smart phone or tablet, to perform some or all aspects of the methods described herein.
[0082]
[0093] FIG. 6 illustrates a general computer system in which an interactive endoscopy method for intraoperative virtual annotation in VATS and minimally invasive surgery according to another representative embodiment may be implemented.
[0083]
[0094] The computer system 600 of Figure 6 illustrates a complete set of components of a communications or computing device. However, a "controller" as described herein may be implemented with fewer components than the set of components of Figure 6, such as by a combination of memory and a processor. The computer system 600 includes some or all of the elements of one or more component devices in the interactive annotation systems described herein. However, any such device may not include one or more of the elements described for the computer system 600, and may include other elements not described.
[0084]
[0095] 6, computer system 600 includes a set of software instructions executable to cause computer system 600 to perform any of the methods or computer-based functions disclosed herein. Computer system 600 can operate as a stand-alone device or can be connected to other computer systems or peripheral devices, for example, using network 601. In an embodiment, computer system 600 performs logical operations based on digital signals received via an analog-to-digital converter.
[0085]
[0096] In a networked arrangement, computer system 600 operates as a server or client user computer in a server-client-user network environment, or as a peer computer system in a peer-to-peer (or distributed) network environment. Computer system 600 may also be implemented as or incorporated into a variety of devices, such as a fixed computer, a mobile computer, a personal computer (PC), a laptop computer, a tablet computer, or any other machine capable of executing a series of software instructions (e.g., sequential) that specify actions to be taken by the machine. Computer system 600 may be implemented as or incorporated into an integrated system that includes additional devices. In an embodiment, computer system 600 may be implemented using electronic devices that provide voice, video, or data communications. Additionally, although computer system 600 is described in the singular, the term "system" is intended to include a collection of systems or subsystems that individually or jointly execute one or more sets of software instructions to perform one or more computer functions.
[0086]
[0097] As illustrated in FIG. 6, the computer system 600 includes a processor 610. The processor 610 may be considered as a representative example of the processor 52210 of the controller 522 of FIG. 5B, and executes instructions to implement some or all aspects of the methods and processes described herein. The processor 610 is tangible and non-transient. As used herein, the term "non-transient" is interpreted as a property of a state that persists for a period of time, rather than an eternal property of a state. The term "non-transient" specifically negates momentary properties, such as carrier waves or signals or other forms of properties that exist only temporarily at any time and place. The processor 610 is an article of manufacture and / or a machine part. The processor 610 executes software instructions to perform the functions described in various embodiments herein. The processor 610 may be a general-purpose processor or part of an application-specific integrated circuit (ASIC). The processor 610 may also be a microprocessor, a microcomputer, a processor chip, a controller, a microcontroller, a digital signal processor (DSP), a state machine, or a programmable logic device. The processor 610 may also be a logic circuit, such as a programmable gate array (PGA), such as a field programmable gate array (FPGA), or another type of circuit including discrete gate and / or transistor logic. The processor 610 may be a central processing unit (CPU), a graphics processing unit (GPU), or both. Additionally, any processor described herein may include multiple processors, parallel processors, or both. Multiple processors may be included or combined in a single device or multiple devices.
[0087]
[0098] The term "processor" as used herein encompasses an electronic component capable of executing a program or machine-executable instructions. References to a computing device that includes a "processor" should be interpreted to include multiple processors or processing cores, such as in a multi-core processor. A processor may also refer to a collection of processors within a single computer system or a collection of processors distributed across multiple computer systems. The term "computing device" should also be interpreted to include a collection or network of computing devices, each of which includes one or more processors. A program comprises software instructions that are executed by one or more processors, either within the same computing device or distributed among multiple computing devices.
[0088]
[0099] The computer system 600 further includes a main memory 620 and a static memory 630. The memories of the computer system 600 communicate with each other and with the processor 610 via a bus 608. Either or both of the main memory 620 and the static memory 630 may be considered representative examples of the memory 52220 of the controller 522 of FIG. 5B, storing instructions used to implement some or all aspects of the methods and processes described herein. The memory described herein is a tangible storage medium for storing data and executable software instructions, and is non-transient while the software instructions are stored therein. As used herein, the term "non-transient" is interpreted as a property of a state that persists for a period of time, rather than a permanent property of a state. The term "non-transient" specifically negates properties that are momentary, such as a carrier wave or signal or other form of property that exists only temporarily at any time and place. The main memory 620 and the static memory 630 are articles of manufacture and / or machine parts. The main memory 620 and the static memory 630 are computer-readable media from which a computer (such as the processor 610) can read data and executable software instructions. Each of the main memory 620 and the static memory 630 may be implemented as one or more of a random access memory (RAM), a read-only memory (ROM), a flash memory, an electrically programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a tape, a compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a floppy disk, a Blu-ray disk, or any other form of storage medium known in the art. The memory may be volatile or non-volatile, secure and / or encrypted, non-secure and / or non-encrypted.
[0089]
[0100] "Memory" is an example of a computer-readable storage medium. Computer memory is any memory that is directly accessible to a processor. Examples of computer memory include, but are not limited to, RAM memory, registers, and register files. References to "computer memory" or "memory" should be interpreted as referring to multiple memories. For example, a memory may be multiple memories within the same computer system. A memory may also be multiple memories distributed among multiple computer systems or devices.
[0090]
[0101] As shown, computer system 600 further includes a video display unit 650, such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid state display, or a cathode ray tube (CRT). Additionally, computer system 600 includes input devices 660, such as a keyboard / virtual keyboard, a touch input screen, a voice input with voice recognition, and a cursor control device 670, such as a mouse or a touch input screen or pad. Computer system 600 also optionally includes a disk drive unit 680, a signal generating device 690, such as a speaker or a remote control, and / or a network interface device 640.
[0091]
[0102] In an embodiment, as shown in FIG. 6, the disk drive unit 680 includes a computer readable medium 682 having one or more sets of software instructions 684 (software) embedded therein. The set of software instructions 684 is read from the computer readable medium 682 for execution by the processor 610. Furthermore, the software instructions 684, when executed by the processor 610, perform one or more steps of the methods and processes described herein. In an embodiment, the software instructions 684 reside in whole or in part in the main memory 620, the static memory 630, and / or the processor 610 during execution by the computer system 600. Furthermore, the computer readable medium 682 may include the software instructions 684 or may receive and execute the software instructions 684 in response to a propagated signal to enable devices connected to the network 601 to communicate voice, video, or data over the network 601. The software instructions 684 may be transmitted or received over the network 601 via the network interface device 640.
[0092]
[0103] In embodiments, dedicated hardware implementations such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic arrays, and other hardware components perform one or more of the methods described herein. One or more embodiments described herein may perform functions using two or more specific interconnected hardware modules or devices, with associated control and data signals that can be communicated between the modules. Thus, the present disclosure encompasses software, firmware, and hardware implementations. Nothing in this application should be construed as being implemented or capable of being implemented using only software rather than hardware such as tangible non-transitory processors and / or memories.
[0093]
[0104] In accordance with various embodiments of the present disclosure, the methods described herein may be implemented using a hardware computer system executing a software program. Furthermore, in non-limiting exemplary embodiments, implementations such as distributed processing, component / object distributed processing, and parallel processing are possible. A virtual computer system process may implement one or more of the methods or functions described herein. The processors described herein may also be used to support virtual processing environments.
[0094]
[0105] Thus, interactive endoscopy for intraoperative virtual annotation in VATS and minimally invasive surgery effectively transforms the interventional imaging modality from a passive modality to an interactive image-based modality that provides a user interface to the interventional medical procedure practitioner, allowing digital tagging at the surgical site during the interventional medical procedure. This allows the medical procedure practitioner to keep track of the anatomical structures once seen and the tasks once performed, eliminating excessive and / or duplicated exploration. Labeling of anatomical structures during the interventional medical procedure also helps to maintain data for use in retrospective review and feedback to optimize future interventional medical procedures. In the long term, the ability to accumulate annotated (e.g., labeled) surgical videos can be used for machine learning development to further improve the integration of information and analytics to augment endoscopic surgery. Automated labeling in particular will gain functionality as more surgical videos are labeled. This is enabled by a user-friendly interface for interactive endoscopy. The ability to label intraoperatively has the potential to generate a large amount of labeled endoscopic data. Using this amount of data allows for improved deformation-prone registration and accurate overlay of pre-operative imaging onto the endoscopic field of view.
[0095]
[0106] A fixed library of virtual labels can also be used in an automated process, complementing simple inputs to add or remove virtual annotations from raw endoscopic images without the use of machine learning. In an embodiment, for example, original features of an endoscopic image can be removed by covering them with virtual annotations of a particular color.
[0096]
[0107] In another example, virtual annotations can be provided under a supervised automated process rather than completing a partial input. For example, a user can indicate a particular structure of interest in an endoscopic image, and a supervised automated process can identify the structure by classification, delineate the boundary of the structure, identify locations for virtual labels near the structure, and track the structure as it moves. In this embodiment, even a single initial instruction can be used to perform the virtual annotation.
[0097]
[0108] Examples where intraoperative virtual annotation can be used include lung surgery to reach and possibly remove a target location. Intraoperative virtual annotation can also be used for tumor resection, lymph node removal and resection, foreign body removal, etc.
[0098]
[0109] Although interactive endoscopy for intraoperative virtual annotation in VATS and minimally invasive surgery has been described with reference to several exemplary embodiments, it is understood that the words used thus far are words of description and illustration, rather than words of limitation. Changes may be made in its aspects within the scope of the appended claims, as currently presented and as amended, without departing from the scope and spirit of interactive endoscopy for intraoperative virtual annotation in VATS and minimally invasive surgery. Although interactive endoscopy for intraoperative virtual annotation in VATS and minimally invasive surgery has been described with reference to specific means, materials, and embodiments, interactive endoscopy for intraoperative virtual annotation in VATS and minimally invasive surgery is not intended to be limited to the disclosed matters, but rather, interactive endoscopy for intraoperative virtual annotation in VATS and minimally invasive surgery extends to all functionally equivalent structures, methods, and uses as fall within the scope of the appended claims.
[0099]
[0110] The illustrations of the embodiments described herein are intended to provide a general understanding of the structure of various embodiments. The illustrations do not completely describe all elements and features of the disclosure described herein. Many other embodiments will be apparent to those skilled in the art upon review of the present disclosure. Other embodiments can be utilized and derived from the present disclosure, such as structural and logical substitutions and changes can be made without departing from the scope of the present disclosure. Moreover, the illustrations are merely representative and may not be to scale. Certain proportions in the illustrations may be exaggerated and other proportions may be minimized. Thus, the present disclosure and the figures should be considered illustrative and not restrictive.
[0100]
[0111] One or more embodiments of the present disclosure may be referred to herein, individually and / or collectively, by the term "invention" for convenience only and without any intention to spontaneously limit the scope of the present application to any particular invention or inventive concept. Also, although specific embodiments are illustrated and described herein, it should be understood that subsequent arrangements designed to achieve the same or similar purpose may be substituted for the specific embodiment shown. The present disclosure is intended to cover any and all subsequent adaptations or variations of the various embodiments. Combinations of the above embodiments with other embodiments not specifically described herein will be apparent to those of skill in the art upon review of the description.
[0101]
[0112] This Abstract of the Disclosure is provided for purposes of compliance with 37 C.FR §1.72(b) and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the above Detailed Description, various features may be grouped together or described in a single embodiment for purposes of streamlining the disclosure. This disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter may be directed to less than all features of any of the disclosed embodiments. Accordingly, the following claims are incorporated into the Detailed Description, with each claim standing on its own as defining separate claimed subject matter.
[0102]
[0113] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to practice the concepts described in the present disclosure. Therefore, the above disclosed subject matter is considered to be illustrative and not restrictive. Moreover, the appended claims are intended to cover all such modifications, enhancements, and other implementations that fall within the true spirit and scope of the present disclosure. Thus, to the maximum extent permitted by law, the scope of the present disclosure shall be determined by the broadest permissible interpretation of the following claims and their equivalents, and shall not be limited or restricted by the foregoing detailed description.
Claims
1. 1. A controller for live annotation of interventional images, comprising: a memory for storing software instructions; a processor for executing said software instructions; Including, When executed by the processor, the software instructions cause the controller to: receiving interventional images during an intraoperative intervention; automatically analyzing the interventional images for detectable features; Detecting a detectable feature; determining to add annotations to the interventional image for the detectable features; identifying a location for the annotation as an identified location within the interventional image; adding the annotation to the interventional image at the identified location corresponding to the detectable feature; outputting, during the intraoperative intervention, a video based on the intervention image and the annotation, including the annotation superimposed on the intervention image at the identified location, as a video output; applying machine learning to the video output; and identifying an optimized placement of the annotations based on the machine learning; A controller that causes the process to be performed.
2. The process performed when the processor executes the software instructions is: detecting an anatomical structure as the detectable feature; and Delineating boundaries of said anatomical structures; The controller of claim 1 further comprising:
3. The process performed when the processor executes the software instructions is: detecting an interventional tool as the detectable feature; and Delineating a boundary of the intervention tool; The controller of claim 1 further comprising:
4. The process performed when the processor executes the software instructions is: determining, from a source external to the controller, predetermined information about the detectable characteristic; and adding the predetermined information from the source to the interventional image as the annotation; Further comprising: the annotation includes a virtual label for virtually labeling the detectable feature with the predetermined information; The controller of claim 1 .
5. The process performed when the processor executes the software instructions is: detecting an anatomical structure as said detectable feature; labeling the anatomical structures with virtual labels as the annotations added to the interventional images; tracking movement of the anatomical structure relative to an endoscope generating the video output; detecting movement of the anatomical structure relative to the endoscope generating the video output; and automatically adjusting a position and orientation of the virtual label based on detection of movement of the anatomical structure relative to the endoscope; The controller of claim 1 further comprising:
6. The process performed when the processor executes the software instructions is: detecting and tracking low level features in the interventional image using a first algorithm; extracting the low-level features as extracted low-level features; and using a second algorithm to recognize and classify anatomical structures as detectable features in the interventional images based on the extracted low-level features; The controller of claim 1 further comprising:
7. The process performed when the processor executes the software instructions is: automatically analyzing the interventional image for detectable features in response to a first command; automatically identifying the location for the annotation in response to a second instruction; and in response to a third command, automatically adding the annotation to the interventional image at the identified location; The controller of claim 1 further comprising:
8. The process performed when the processor executes the software instructions is: in response to an instruction to add the annotation, automatically adding the annotation to the interventional image at the identified location based on the instruction; The controller of claim 1 further comprising:
9. The process performed when the processor executes the software instructions is: automatically identifying a location for the annotation in response to an instruction to identify the location; The controller of claim 1 further comprising:
10. The process performed when the processor executes the software instructions is: automatically detecting structures for virtual labeling in response to an instruction to detect the structures; The controller of claim 1 further comprising:
11. The process performed when the processor executes the software instructions is: Detecting movement of an anatomical structure relative to an endoscope generating the interventional image; and automatically adjusting a position and orientation of the annotation based on the detection of the movement of the anatomical structure relative to the endoscope; The controller of claim 1 further comprising:
12. The process performed when the processor executes the software instructions is: analyzing intraoperative information from said intraoperative intervention; Deriving content of the annotation based on the intraoperative information; and adding said content to said interventional image as said annotation; Further comprising: The controller of claim 1 , wherein the annotation comprises a virtual label.
13. The process performed when the processor executes the software instructions is: Obtaining a preoperative segmented model of an anatomical structure; receiving instructions to annotate the anatomical structure using the pre-operative segmented model as the annotation; and Virtually replacing the anatomical structure of the interventional image with the pre-operative segmented model of the anatomical structure; The controller of claim 1 further comprising:
14. The process performed when the processor executes the software instructions further includes registering the pre-operative segmented model of the anatomical structure to the anatomical structure; the instructions include selecting an icon corresponding to the pre-operative segmented model; The controller of claim 13.
15. The process performed when the processor executes the software instructions is: Detecting a trigger during the intraoperative intervention; and altering the annotation based on detection of the trigger; The controller of claim 1 further comprising:
16. 1. A system for live annotation of interventional images, comprising: A display for displaying the intervention image; a controller including a memory for storing software instructions; a processor for executing said software instructions; Including, When executed by the processor, the software instructions cause the controller to: receiving interventional images during an intraoperative intervention; automatically analyzing the interventional images for detectable features; Detecting a detectable feature; determining to add annotations to the interventional image for the detectable features; identifying a location for the annotation as an identified location within the interventional image; adding the annotation to the interventional image at the identified location corresponding to the detectable feature; outputting a video based on the interventional image and the annotations, including the annotations superimposed on the interventional image at the identified location, as a video output to the display during the intraoperative intervention; applying machine learning to the video output; and identifying an optimized placement of the annotations based on the machine learning; A system for performing a process including the steps of:
17. The detectable feature is detected based on applying the machine learning to results from at least one prior instantiation of a live annotation, and the identified location is identified based on applying the machine learning to results from the at least one prior instantiation of a live annotation.
17. The system of claim 16.
18. A tangible, non-transitory, computer-readable storage medium storing a computer program, the computer program, when executed by a processor, causes a system including the tangible, non-transitory, computer-readable storage medium to perform a process for live annotation of interventional images, the process being performed when the processor executes the computer program, comprising: receiving an intervention image from a thoracoscope that generates the intervention image as a view inside the chest during a video-assisted thoracoscopic surgery for lung tumor resection; automatically analyzing the interventional images for detectable features by a processor executing software instructions from a memory; Detecting a detectable feature; determining to add annotations to the interventional image for the detectable features; Identifying a location for the annotation as a identified location within the interventional image; adding the annotation to the interventional image at the identified location corresponding to the detectable feature; and outputting a video based on the intervention image and the annotations, including the annotations superimposed on the intervention image at the identified location, as a video output during the video-assisted thoracoscopic surgery; applying machine learning to the video output; and identifying an optimized placement of the annotations based on the machine learning; and 1. A tangible, non-transitory computer-readable storage medium, comprising:
19. The detectable feature is detected based on applying the machine learning to results from at least one prior instantiation of a live annotation, and the identified location is identified based on applying the machine learning to results from the at least one prior instantiation of a live annotation.
20. The tangible, non-transitory computer readable storage medium of claim 18.
20. The process performed when the processor executes the computer program is detecting an anatomical structure as the detectable feature; labeling the anatomical structures with virtual labels as the annotations added to the interventional images; tracking movement of the anatomical structure relative to an endoscope generating the video output; detecting movement of the anatomical structure relative to the endoscope generating the video output; automatically adjusting a position and orientation of the virtual label based on detection of movement of the anatomical structure relative to the endoscope; 20. The tangible, non-transitory computer readable storage medium of claim 18, further comprising:
Citation Information
Patent Citations
Branch structure determination device, method, and program
JP2016163609A
Optimized anatomical structure of interest labeling
JP2017534117A
Surgical recognition system
US20190069957A1
