Interactive endoscopy for intraoperative virtual annotation in VATS and minimally invasive surgery

Abstract

A controller (522) for live annotation of interventional imagery includes a memory (52220) that stores software instructions and a processor (52210) that executes the software instructions. When executed by the processor (52210), the software instructions cause the controller (522) to implement a process that includes receiving (S210) interventional imagery during an intraoperative intervention and automatically analyzing (S220) the interventional imagery for a detectable feature. The process further includes, when executed by the processor (52210) executing the software instructions, detecting (S230) the detectable feature and determining (S240) that an annotation is to be added to the interventional imagery for the detectable feature. The process further includes identifying (S250) a location in the interventional imagery for the annotation as an identified location and adding (S260) the annotation to the interventional imagery at the identified location to correspond to the detectable feature. During the intraoperative intervention, a video is output (S270) based on the interventional imagery and the annotation as a video output, the annotation including the annotation overlaid on the interventional imagery at the identified location.

Description

Background Technology

[0001] Interventional medicine procedures are invasive processes that involve entering a patient's body. Surgery is an example of an interventional medicine procedure and is the preferred treatment for early-stage lung cancer tumors. Furthermore, endoscopy is increasingly used at different stages of lung cancer surgery. For lung cancer surgery, the fundamental precursor to tumor resection (removal) is the "surgical exploration" stage, in which a medical clinician (such as a surgeon) examines the lung tissue to mentally correlate anatomical knowledge and preoperative imaging (such as computed tomography (CT)) with live endoscopic (e.g., thoracoscopy) video of the lung tissue. The surgical exploration stage helps ensure that the clinician can remove the entire lung cancer tumor during the resection stage, while avoiding the removal of healthy lung tissue that could otherwise lead to impaired lung function. Surgical exploration using thoracoscopy typically provides familiarity with lung tissue that cannot be obtained from preoperative imaging (such as CT), as preoperative imaging primarily shows differences in lung tissue attenuation to highlight anatomical landmarks. During surgical exploration, the clinician views the lung tissue through a thoracoscope while manipulating the lung tissue with instruments to identify anatomical landmarks to facilitate resection. Specifically, during surgical exploration, clinicians attempt to identify known anatomical landmarks (such as blood vessels and airways near the lung cancer tumor) in order to properly orient the anatomical structures to locate the lung cancer tumor and to avoid damaging blood vessels and airways during resection.

[0002] The most invasive form of lung cancer surgery is open surgery, in which an incision is made in the chest to expose most of the lung. In open surgery, surgical instruments such as scalpels, electrocautery, and sutures are inserted through a large opening in the chest cavity and used to remove the lung cancer tumor. In the past, lung cancer tumors were large enough to be felt by touch, allowing clinicians to find tumors invisible to the naked eye. Open surgery techniques allow for physical access for palpation to sense the tumor by touch.

[0003] In recent years, both the detection of embedded lung cancer tumors and the techniques used for surgical removal of lung cancer tumors have improved. For example, lung cancer screening programs now tend to identify small and difficult-to-discern early-stage tumor nodules. In addition, a minimally invasive technique called video-assisted thoracoscopic surgery (VATS) for removing lung cancer tumors has emerged as an alternative to open surgery.

[0004] Locating and removing lung cancer tumors with safe margins while preserving healthy lung tissue remains a crucial aspect of successful VATS. In VATS, a small camera is inserted into the thoracic cavity through a small port (i.e., a small incision or slit), and surgical instruments are inserted through the same port or other small ports. The entire surgical resection is performed using the camera's view. Figure 1 The illustration depicts a standard VATS implementation method for lung resection. Figure 1In this procedure, a thoracoscope can be inserted through the patient's chest cavity. Otherwise, an obstructed view can be restored via the thoracoscope. Figure 1 In this procedure, instruments #1 and #2 are inserted into the patient's P through two separate small incisions to perform the resection. In recent years, robotic surgery has emerged as a competitive minimally invasive method similar to VATS.

[0005] However, three main challenges remain in lung cancer surgery. First, the location of the lung cancer tumor can be determined based on preoperative CT scans with the lung fully inflated long before surgery. Therefore, when the lung collapses during subsequent surgery, the three-dimensional (3D) orientation of the lung and the location of the lung cancer tumor will not match the images from the preoperative CT scans used for planning. Second, the lung is complex, with numerous blood vessels and airways that must be carefully dissected and addressed before removing the lung cancer tumor and any feeding airways or blood vessels. Third, because small, intangible lung cancer tumors are particularly difficult to identify and locate, especially using VATS or robotic surgery, there is still a possibility that additional healthy lung tissue can be removed during the procedure to prevent the presence of lung cancer tumor tissue.

[0006] To overcome these challenges, methods to improve surgical workflows for better guiding the resection of lung cancer tumors have been investigated. Surgical exploration involved in VATS is time-consuming, indeterminate, and unquantifiable, making many aspects of the procedure difficult to reproduce. Furthermore, the interactive endoscopy for intraoperative virtual annotation in VATS and minimally invasive surgery described in this paper addresses these challenges. Summary of the Invention

[0007] According to one aspect of this disclosure, a controller for live annotation of interventional images includes a memory and a processor, the memory storing software instructions that the processor executes. When executed by the processor, the software instructions cause the controller to perform a process including receiving interventional images during an intraoperative intervention and automatically analyzing the interventional images for detectable features. The process performed when the processor executes the software instructions further includes detecting detectable features and determining to add annotations to the interventional images for the detectable features. The process performed when the processor executes the software instructions further includes identifying a location in the interventional images corresponding to the annotations as an identified location, and adding the annotations to the interventional images at the identified locations to correspond to the detectable features. During the intraoperative intervention, video is output as video output based on the interventional images and the annotations, the annotations including annotations superimposed on the interventional images at the identified locations.

[0008] According to another aspect of this disclosure, a system for live annotation of interventional images includes an electronic display and a controller. The electronic display displays the interventional images. The controller includes a memory and a processor, the memory storing software instructions, and the processor executing the software instructions. When executed by the processor, the software instructions cause the controller to perform a process including receiving the interventional images and automatically analyzing the interventional images for detectable features. The process performed when the processor executes the software instructions further includes detecting the detectable features and determining to add annotations to the interventional images for the detectable features. The process performed when the processor executes the software instructions further includes identifying a location in the interventional images corresponding to the annotations as an identified location, and adding the annotations to the interventional images at the identified locations to correspond to the detectable features. During intraoperative intervention, a video is output as a video output on the electronic display based on the interventional images and the annotations, the annotations including annotations superimposed on the interventional images at the identified locations.

[0009] According to another aspect of this disclosure, a method for live annotation of interventional images includes receiving interventional images from a thoracoscope during video-assisted thoracoscopic surgery for lung tumor resection, the thoracoscope generating the interventional images as a view of the interior of the chest. The method further includes automatically analyzing the interventional images for detectable features by a processor executing software instructions from a memory. The method also includes detecting the detectable features and determining to add annotations to the interventional images for the detectable features. The method further includes identifying a location in the interventional images corresponding to the annotations as an identified location, and adding the annotations to the interventional images at the identified locations to correspond to the detectable features. During video-assisted thoracoscopic surgery, video is output as video output based on the interventional images and the annotations, the annotations including annotations superimposed on the interventional images at the identified locations. Attached Figure Description

[0010] When with attachment Figure 1 When reading this, the exemplary embodiments are best understood in light of the following detailed description. It should be emphasized that the various features are not necessarily drawn to scale. In fact, dimensions may be increased or decreased arbitrarily for clarity of discussion. Wherever applicable and useful, similar reference numerals refer to similar elements.

[0011] Figure 1 The illustration shows a routine video-assisted thoracoscopic surgery (VATS) procedure for lung resection.

[0012] Figure 2A The illustration depicts a method for intraoperative virtual annotation in VATS and minimally invasive surgery according to a representative embodiment.

[0013] Figure 2B The illustration shows another method for intraoperative virtual annotation in VATS and minimally invasive surgery, according to a representative embodiment.

[0014] Figure 2C The illustration shows another method for intraoperative virtual annotation in VATS and minimally invasive surgery, according to a representative embodiment.

[0015] Figure 2D The illustration shows another method for intraoperative virtual annotation in VATS and minimally invasive surgery, according to a representative embodiment.

[0016] Figure 2E The illustration shows another method for intraoperative virtual annotation in VATS and minimally invasive surgery, according to a representative embodiment.

[0017] Figure 2F The illustration shows another method for intraoperative virtual annotation in VATS and minimally invasive surgery, according to a representative embodiment.

[0018] Figure 3 The illustration shows another method for intraoperative virtual annotation in VATS and minimally invasive surgery, according to a representative embodiment.

[0019] Figure 4 Three examples of virtual annotations for intraoperative virtual annotation in VATS and minimally invasive surgery, according to representative embodiments, are illustrated.

[0020] Figure 5A The illustration depicts a system for interactive endoscopic examination of intraoperative virtual annotations in VATS and minimally invasive surgery, according to a representative embodiment.

[0021] Figure 5B The illustration shows a controller for interactive endoscopic examination of intraoperative virtual annotation in VATS and minimally invasive surgery, according to a representative embodiment.

[0022] Figure 6 The illustration shows a general-purpose computer system according to another representative embodiment, on which a method for intraoperative virtual annotation in VATS and minimally invasive surgery is implemented. Detailed Implementation

[0023] In the following detailed description, representative embodiments with specific details disclosed are set forth for purposes of explanation and not limitation, in order to provide a thorough understanding of embodiments according to this teaching. Descriptions of known systems, apparatuses, materials, methods of operation, and methods of manufacture may be omitted to avoid obscuring the description of representative embodiments. Nevertheless, systems, apparatuses, materials, and methods within the scope of knowledge of those skilled in the art can be used within the scope of this teaching and can be used according to representative embodiments. It should be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. The defined terminology is supplementary to its technical and scientific meaning as commonly understood and accepted in the art as described in this teaching.

[0024] It will be understood that although the terms first, second, third, etc., may be used in this document to describe various elements or components, these elements or components should not be limited by these terms. These terms are only used to distinguish one element or component from another. Therefore, without departing from the teachings of the inventive conception, the first element or component discussed below may be referred to as the second element or component.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the specification and claims, the singular terms “a,” “an,” and “the” are intended to include both the singular and plural forms unless the context clearly specifies otherwise. Furthermore, when used herein, the terms “comprising” and / or “including” and / or similar terms specify the presence of stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0026] Unless otherwise specified, when an element or component is referred to as "connected to," "coupled to," or "proximity to" another element or component, it will be understood that the element or component may be directly connected to or coupled to the other element or component, or that there may be intermediary elements or components present. That is, these and similar terms cover situations where one or more intermediate elements or components may be used to connect two elements or components. However, when an element or component is referred to as "directly connected" to another element or component, this only covers situations where two elements or components are connected to each other without any intermediate or intermediary elements or components.

[0027] Therefore, this disclosure is intended to present one or more of the advantages specifically pointed out below through its various aspects, embodiments, and / or specific features or sub-components. Example embodiments with specific details disclosed are set forth for purposes of explanation and not limitation in order to provide a thorough understanding of embodiments according to this teaching. However, other embodiments consistent with this disclosure that depart from the specific details disclosed herein remain within the scope of the claims. Furthermore, descriptions of well-known apparatuses and methods may be omitted so as not to obscure the description of the example embodiments. Such methods and apparatuses are within the scope of this disclosure.

[0028] The interactive endoscopy described herein for intraoperative virtual annotation in VATS and minimally invasive surgery provides an interactive annotation system for interactively annotating live video during surgery, which can be viewed during interventional medical procedures. Interactive endoscopy also provides informative annotations to help personnel involved in the medical intervention annotate persistent features, anatomical or other features, track annotated persistent features via annotation as they are found, and indicate tasks already performed.

[0029] Figures 2A to 2F The illustrations depict methods for intraoperative virtual annotation in VATS and minimally invasive surgery, according to various representative embodiments. Typically, the methods are performed during interventional medicine procedures using interactive endoscopy, such as when a thoracoscope and one or more surgical instruments are inserted into the patient. While examining tissues during the interventional medicine procedure, clinicians view a display of interventional images (e.g., endoscopic images) that provide information such as anatomical features of interest and / or surgical events. Interventional images can be, for example, video provided by an endoscopic camera during VATS. According to various embodiments, all or part of this information can be annotated on the display using an interactive annotation system, for example, by overlaying virtual annotations onto the interventional images to visualize the surgical field. Annotations can be placed at locations identified by the clinician or by the interactive annotation system based on machine learning applied to previously instantiated data. The placement of annotations should be contextually meaningful, such as placing annotations for vessels where they appear in the interventional images.

[0030] Furthermore, clinicians can provide one or more instructions to trigger the entire annotation workflow or an individual aspect of the annotation workflow. The clinician continues the interventional medical procedure, adding annotations as needed. When the clinician identifies a feature or event to be annotated, they assign that feature or event to the software control system of the interactive annotation system, which helps determine where and how the annotation is applied. In an additional embodiment, as tissue and / or the endoscope moves, the software control system can apply image processing algorithms to adjust the position and orientation of the annotation relative to the corresponding image location.

[0031] Figure 2A The illustration depicts a method for intraoperative virtual annotation in VATS and minimally invasive surgery according to a representative embodiment. Figure 2A The method can be, for example, as described below. Figure 5B The controller 522 in the system 500 for interactive endoscopic examination is used to implement this.

[0032] refer to Figure 2A At S210, interventional images are received during the intraoperative intervention. Interventional images are captured using imaging devices, such as an endoscopic camera on an endoscope to provide endoscopic images. For example, endoscopic images can be video captured using an endoscope during an interventional medical procedure. Another example of an imaging device is a small camera positioned to focus on a specific part of an anatomical structure or instrument during spinal surgery or a similar procedure; in this case, the interventional image is a video of the camera image, such as the anatomical structure or instrument. Interventional images can be obtained from outside the patient, or, in the case of endoscopic images, from inside the patient.

[0033] At S220, in response to the first instruction, the interventional image is automatically analyzed for detectable features. Detectable features may include, for example, anatomical structures of the patient's anatomy and / or interventional tools used during the interventional medical procedure. The analysis for detectable features at S220 may be performed using one or more video analysis algorithms, such as algorithms for detecting and tracking low-level features in images (e.g., video frames). For example, the first video analysis algorithm for detecting and tracking low-level features may include a Scale Invariant Feature Transform (SIFT) algorithm, a Speed-Up Robust Feature Transform (SURF) algorithm, an optical flow algorithm, or a learned feature algorithm. A second video analysis algorithm may be used to identify and classify anatomical structures of interest based on the extracted low-level features. A third video analysis algorithm may be used to calculate tissue deformations, such as rotation about any of the three axes or linear translation in any of the three axes, wherein the three axes are set relative to a fixed object or position.

[0034] The automation of the functional features described in this paper can be implemented based on machine learning algorithms such as deep learning. Machine learning can be applied to multiple different individual systems (such as...) Figure 5A Machine learning can be implemented centrally in a system (500), and can be performed in a cloud-based processing system (such as at a data center). Alternatively, machine learning can be implemented centrally in a dedicated central computer system (such as for multiple geographically dispersed individual systems that are related to the entity providing the dedicated central computer system).

[0035] Instantiation of interventional images can undergo machine learning to identify patterns and correlations, and the results of machine learning can be used to optimize aspects taught in this paper, such as the analysis of detectable features at S220. Furthermore, interactive annotation systems can be provided with setup information prior to the interventional medical procedure to facilitate automation, such as by helping to reduce processing requirements. For example, setup information could be the type of medical intervention, such as lung resection, so that the interactive annotation system analyzing endoscopic images knows the types of anatomical features typically found in the lungs. Alternatively, the interactive annotation system can automatically identify the background environment, such as the lungs, to narrow down the analysis of detectable features to those typically found in or around the lungs.

[0036] At S230, detectable features are detected in the interventional images. Where the detectable feature can be an anatomical structure or an instrument, one or more image recognition algorithms can be applied to detect the detectable feature or multiple detectable features. For example, an instrument detected as a detectable feature can be forceps, sutures, staples, or other implantable devices attached to an anatomical structure. Detection of the detectable feature at S230 can be based on identifying one or more of the detectable feature's shape, color, relative placement, and / or other characteristics.

[0037] At S240, in response to a second instruction, it is determined that an annotation should be added to the interventional image for the detectable feature. The second instruction received at S240 may reflect a positive confirmation of adding an annotation to the detectable feature detected at S230, and may be received as a result of prompts such as highlighting the detectable feature or the outline of the detectable feature on the screen after S230 and before S240. At S250, in response to a third instruction, the location for the annotation is automatically identified in the interventional image. The identification of the location at S250 may include identifying the distance and orientation relative to the corresponding detected feature, and may take into account the background of where the detected feature is located relative to the tumor, relative to other anatomical features, and / or relative to other tools in the interventional image. At S260, in response to a fourth instruction, an annotation is automatically added to the interventional image at the identified location to correspond to the detectable feature. The fourth instruction received at S260 may reflect confirmation that the identified location is acceptable and a determination to add the annotation at S240. The first, second, third, and fourth instructions may be software instructions provided, for example, automatically by the controller 522 and / or in response to input from the clinician.

[0038] At S270, the output includes the interventional image and a video with annotations superimposed on the interventional image at the identified locations. For example, image overlay functions (such as those included in the open-source software library OpenCV) can be used to perform the overlay of annotations onto the endoscopic image.

[0039] In an embodiment, although not shown, the features from S210 to S270 can be executed cyclically, such that interventional images continue to be received and analyzed at S210 and S220 even when video output is output at S270. Based on the features from S210 to S270, the interaction between the clinician involved in the interventional medical procedure and the interactive annotation system can include receiving any instruction from a variety of commands and performing the corresponding function based on that instruction. In this way, the clinician involved in the interventional medical procedure can interactively control the annotation on the interventional images in an immediately useful manner. Furthermore, the features from S210 to S270 can be implemented by an interactive annotation system that is provided entirely within the space where the interventional medical procedure occurs and in continuous, uninterrupted time frames between the start of the interventional medical procedure by initially inserting medical equipment (e.g., a thoracoscope) into the patient and the end of the interventional medical procedure by removing the medical equipment from the patient. Alternatively, the features from S210 to S270 may begin when the interactive annotation system is affirmatively started, such as by a command, and continue until the interactive annotation system is affirmatively shut down, such as by an instruction from a clinician involved in the interventional medical process.

[0040] At S280, machine learning is applied to interventional images with overlaid annotations, and the optimal placement of the annotations is identified based on machine learning. The machine learning used in S280 can be used to optimize annotation placement, but it can also be used to improve the functionality of interactive annotation systems, to understand tasks and image content in interventional medical procedures, and / or to refine the user interface. The process from S220 to S280 can be executed in a loop for multiple medical interventions.

[0041] The S280-specific functionality reflects the actual placement of the annotations. The actual details of the annotations and placement can be pooled using other instantiated actual details to identify, for example, the average distance to relevant detectable features and whether the annotations interfere with other features of the interventional image in potentially problematic ways. As an example, optimized annotation placement can be learned by applying machine learning to obtain quantitative measures of existing pairs of anatomical features and virtual labels. For instance, the quantitative measure could include the distance between anatomical features and virtual labels. As another example, the quantitative measure could include the area or volume of certain types of virtual labels, or the area or volume between certain types of anatomical features and virtual labels.

[0042] The machine learning applied at S280 is merely one example of machine learning that can be applied according to the teachings in this paper. Examples of machine learning for the functionality of interactive annotation systems also include feature detection, such as at S220 and S230. Feature detection can include learning how and where to find detectable patterns and anatomical features, such as by studying multiple instantiations of previous endoscopic images and identifying commonalities related to how and where the user annotates specific types of detectable features. Feature detection can also include learning patterns of when the user places annotations in the context of specific types of medical interventions.

[0043] Machine learning for understanding tasks and image content in interventional medical procedures can be used for a variety of purposes. For example, feature detection can be used to learn to find clinically relevant anatomical features and landmarks, such as at S220 and S230. Machine learning can also be used to learn what annotations are used in a given interventional medical procedure, and when / where those annotations are placed, such as at S250. Machine learning can also be used to learn what annotations correspond to surgical events or actions, the timing and sequence of annotation placement, and patterns in the surgical workflow that reflect the thought processes of those involved in the medical intervention. Machine learning can also be used to learn when annotations disappear from the view, such as when annotations for anatomical structures disappear when the anatomical structure is a part of an organ that is flipped or otherwise rotated. Machine learning can also be used to identify relationships between similar annotations used in different interventional medical procedures.

[0044] Examples of machine learning used for user interface tuning can include identifying which annotations should be used for specific anatomical structures. Machine learning for user interface tuning can also be used to learn the appropriate size and orientation for annotations, and even details such as the size and font of text annotations. Machine learning for user interfaces can also be applied to learn the number of annotations to use, such as the maximum or optimal number. Machine learning for user interfaces can also identify patterns of when to explicitly remove specific annotations or types of annotations, such as when personnel involved in interventional medical procedures determine that an annotation is no longer useful. Machine learning can also be used to identify user preferences, enabling the customization of annotation options for any particular user.

[0045] Machine learning can be used in the form of a feedback loop, where aspects of the annotations in an interventional medical procedure reflect optimizations from machine learning in previous interventional medical procedures. That is, the annotations in an interventional medical procedure can be based on at least one previously instantiated machine learning instance applied to the live-action annotation. Interventional images from an interventional medical procedure can then be studied and incorporated into machine learning for use in optimizing subsequent interventional medical procedures.

[0046] As mentioned above, in Figure 2A In this method, detectable features can be detected based on automated analysis of interventional images. Virtual annotations are placed in the interventional images to correspond to the detectable features, and the annotated interventional images are output as an annotated video including the interventional images and annotations. The annotations can be used as a form of augmented reality. For example, the annotated interventional images can be used as roadmaps, so that clinicians do not have to repeatedly detect and identify the same detectable features. Of course, the annotations are not limited to detectable features derived from anatomical structures, as the annotated interventional images are also used to identify tools and provide checklists for tasks completed during surgery and other information.

[0047] As described above, an interactive annotation system can be used to control interactive endoscopic examinations for intraoperative virtual annotation. The interactive annotation system provides functional features including at least analyzing input from a live surgical video to detect detectable features, accepting dynamic command input to add virtual annotations (e.g., labels) to the live surgical video, determining the locations(s) in the video where annotations(s) should be added, and generating an output video containing the original surgical video with overlaid annotations.

[0048] Figure 2B The illustration depicts another method for intraoperative virtual annotation in VATS and minimally invasive surgery, according to a representative embodiment. Where appropriate, Figure 2B The method can supplement Figure 2A The method, but Figure 2B The individual characteristics of the method may be replaced where indicated or where otherwise appropriate. Figure 2A Individual characteristics of the method.

[0049] refer to Figure 2B Anatomical structures are detected as detectable features at S231. In this case, S231 replaces the features from... Figure 2A S230 serves as an example of the type of detectable feature to be detected. At S232, the boundary of the detected anatomical structure is depicted. This can be done, for example, by graphically identifying and outlining the boundary (such as by highlighting the appearance of the anatomical structure at the boundary with color, or on the other hand).

[0050] At S261, anatomical structures are labeled using virtual tags as annotations. Figure 2B The anatomical structure marked at S261 is Figure 2A An example of automatically added annotations at S260. For example, virtual labels may include markers with text or numbers and / or icons within shapes (such as predetermined outlines corresponding to, for example, anatomical structures). Labels and other annotations can range from semantically complex. For example, annotations may indicate time, identify blood vessels, or outline tumors. Furthermore, annotation instructions can be provided via touchscreen, voice, or automatically. Figure 2A After the features (such as determining to add an annotation at S240 and identifying the location for the annotation at S250) are defined, the marking at S261 is performed. That is, when appropriate, now for Figure 2B The embodiments described Figure 2A Features can still be included Figure 2B In the embodiments described above.

[0051] Examples of annotable information that can be included in annotations (such as virtual labels) include markings for various anatomical features, such as lung fissures, tumors, blood vessels, airways, and other organs. Virtual labels can include multiple checklist items, in which case examples of annotable information include surgical events such as incisions, staples, resections, grasps, rotations, and stretches. Other examples of annotable information include perioperative information such as elapsed time and forces exerted. Detailed information can be provided as annotable information, such as details assessed based on intraoperative information, and sometimes even combined with external reference data that can include various forms of physiological data, models, and imaging. Postoperative information such as staples or precise tumor location can also be provided as annotable information, which can be used as input to future learning algorithms. Most of the annotable information described in this paper can be used as a live feedback in the form of a roadmap for medical personnel involved in interventional medical procedures.

[0052] Tracking the movement of anatomical structures at S271 can... Figure 2A The process occurs after the video output at S270. For example, the movement of the anatomical structure can be tracked when the anatomical structure rotates or translates in the horizontal and / or vertical directions. The anatomical structure can also move relative to the viewpoint of the imaging device, such as an endoscope camera, when the imaging device moves. At S272, the movement of the anatomical structure relative to the imaging device, such as an endoscope camera that generates interventional images, is detected. The movement of the anatomical structure inherently reflects the usefulness of the corresponding virtual tag, therefore the movement of the anatomical structure can be tracked to ensure that the virtual tag moves with the anatomical structure.

[0053] At S273, the position and orientation of the virtual tag are automatically adjusted based on the movement of the anatomical structure being tracked relative to the endoscope. The adjustment at S273 can be based on measuring the deformation of the corresponding anatomical structure, such as rotation about any of the three axes or linear translation in any of the three axes.

[0054] Figure 2C The illustration depicts another method for intraoperative virtual annotation in VATS and minimally invasive surgery, according to a representative embodiment. Where appropriate, Figure 2C The method can supplement Figure 2A The method, but Figure 2CThe individual characteristics of the method may be replaced where indicated or where otherwise appropriate. Figure 2A Individual characteristics of the method.

[0055] refer to Figure 2C At S221, the first algorithm is used to detect and track low-level features in the interventional image, which can serve as... Figure 2A The analysis at S220 is performed in this section. Low-level features include features that form relatively unique patterns in the details of interventional images so that they can be interpreted as anatomical structures or tools. Low-level features include patterns that may not be recognizable to humans, may not be observable to humans, or may otherwise be undetectable to humans by the naked eye. As previously mentioned, examples of algorithms for detecting and tracking low-level features in images such as video frames include Scale Invariant Feature Transform (SIFT) algorithms, Accelerated Robust Feature (SURF) algorithms, optical flow algorithms, or learned feature algorithms.

[0056] At S222, Figure 2C The method involves extracting low-level features as the extracted low-level features. S222 can also be used as... Figure 2A The analysis at S220 is performed.

[0057] Figure 2C The method continues at S223, where detectable features are detected by using a second algorithm to identify and classify anatomical structures as detectable features in the endoscopic image based on the extracted low-level features. S223 can be used as... Figure 2A The detection portion at S230 is executed. The second algorithm may include features such as comparing low-level features detected by the first algorithm with known properties of known anatomical features (such as by color, size, shape, and proximity to other known properties). Both low-level features and properties of known anatomical features may have patterns that are meaningless to humans, unobservable to humans, or otherwise undetectable to humans, such as by the naked eye, but still recognizable and detectable by algorithms used in image processing. The second algorithm may rank possible matches by how closely they match the properties of known anatomical features and then select the highest-ranking known anatomical feature. The highest-ranking known anatomical feature for low-level features may then be classified as relevant or irrelevant, such as based on the type of medical intervention being performed. Alternatively, the classification may be based on the type of known anatomical feature, such as bone or tissue, organ or airway, or other types. The algorithm may then selectively prompt personnel involved in the interventional medical procedure to annotate the identified anatomical features based on the type of known anatomical feature and its relevance to the medical intervention being performed.

[0058] At S224, Figure 2CThe method may include depicting the boundaries of anatomical structures. The depiction at S224 can be performed by overlaying solid or dashed lines to mark the boundaries of the anatomical structures.

[0059] As mentioned above, in Figure 2C In this context, one or more algorithms for image analysis and feature classification can be used to perform image analysis and feature detection. For example, it can be performed on each of multiple frames in an endoscopic image. Figure 2C The process. Once the first endoscopic image is generated, or once a specific command is received to begin image analysis and feature detection or to specifically add annotations, image analysis and feature detection can be performed throughout the interventional medicine procedure. That is, personnel involved in the interventional medicine procedure can dynamically decide when to annotate features during the procedure, and... Figure 2A S220 or Figure 2C The treatment starting at S221 can be initiated by personnel involved in the interventional medical process who initiate the procedure described herein.

[0060] Figure 2D The illustration depicts another method for intraoperative virtual annotation in VATS and minimally invasive surgery, according to a representative embodiment. Where appropriate, Figure 2D The method can supplement Figure 2A The method, but Figure 2D The individual characteristics of the method may be replaced where indicated or where otherwise appropriate. Figure 2A Individual characteristics of the method.

[0061] refer to Figure 2D At S241, predetermined information for detectable features is identified from an external source. For example, when in Figure 2A During the process, when it is determined at S240 that an annotation needs to be added, labels, icons, information items, or other predetermined information can be identified at S241. At S262, the predetermined information from the external source is added as an annotation to the interventional image. For example, when a fourth instruction is received at S260, predetermined information from the external source can be added as an annotation to the interventional image at S262, as a response to... Figure 2A The comments at S260 are automatically added modifications or additions.

[0062] Figure 2E The illustration depicts another method for intraoperative virtual annotation in VATS and minimally invasive surgery, according to a representative embodiment. Where appropriate, Figure 2E The method can supplement Figure 2A The method, but Figure 2E The individual characteristics of the method may be replaced where indicated or where otherwise appropriate. Figure 2A Individual characteristics of the method.

[0063] refer to Figure 2E At S211, intraoperative information from the intraoperative intervention is analyzed. Intraoperative information may include details about the setup of the interventional medical procedure performed as described herein. For example, intraoperative information may include the voice, gestures, or specific inputs of medical personnel involved in the interventional medical procedure. Intraoperative information may also include information output from medical equipment (such as monitors for heart rate and pulse), and the time when the information is output. The analysis at S211 may be for a specific predetermined type of intraoperative information, or it may be for multiple predetermined types of intraoperative information.

[0064] At S242, the content for the annotation is derived based on the analyzed intraoperative information. For example, the annotation content can be derived from intraoperative information based on conversations or sounds in the operating room, as well as from intraoperative information from images taken in the operating room (such as from a camera, endoscope, or another facility used for medical imaging). The annotation content can also be derived from intraoperative information based on electronic signals from equipment (such as indicators that a tool is turned on or off, lights are turned on or off, or adjusted up or down, or a blood pressure monitor triggers an alarm when the patient's blood pressure exceeds an upper or lower threshold). These are all examples of how the content for the annotation can be derived in real time based on the intraoperative information derived from the analysis at S241 at S242. The annotation itself can then be placed as a record on the interventional image so that it is visible to the medical personnel involved in the interventional medical procedure.

[0065] At S263, Figure 2E The process involves adding content as annotations to the endoscopic images. For example, when the fourth instruction is received at S260, content derived from the intraoperative information can be added as an annotation at S263, as a response to... Figure 2A The automatic addition of modifications or supplements at S260.

[0066] Figure 2F The illustration shows another method for intraoperative virtual annotation in VATS and minimally invasive surgery, according to a representative embodiment. Figure 2F The method can be supplemented as appropriate. Figure 2A The method, but Figure 2F The individual characteristics of the method may be replaced where indicated or where otherwise appropriate. Figure 2A Individual characteristics of the method.

[0067] refer to Figure 2F A preoperative segmentation model of the anatomical structure is obtained at S205. For example, the preoperative segmentation model can be obtained from a preoperative computed tomography (CT) scan of the anatomical structure. The preoperative CT scan is used to generate the model, which is then segmented.

[0068] At S264, the instruction to annotate the anatomical structure is received along with the preoperative segmentation model as the annotation. That is, any anatomical structure represented by the preoperative segmentation model from S205 corresponds to the instruction at S264.

[0069] At S265, the anatomical structures in the endoscopic images are virtually replaced with a preoperative segmentation model of the anatomical structures. That is, at S265, an instruction is received to replace the live interventional images of the anatomical structures with the preoperative segmentation model of the anatomical structures obtained at S205.

[0070] At S266, the preoperative segmentation model of the anatomical structure is registered to the anatomical structure. That is, at S266, the live interventional image of the anatomical structure is registered with the preoperative segmentation model of the anatomical structure obtained at S205. As a result, the annotation automatically added at S260 involves registering the live interventional image of the anatomical structure with the preoperative segmentation model of the anatomical structure obtained at S205 and replacing the live interventional image of the anatomical structure with the preoperative segmentation model of the anatomical structure obtained at S205. This form of annotation can help those involved in the interventional medicine process visualize aspects of the patient's anatomy on screen, thereby helping to improve focus. That is, like most or all forms of annotation described herein, Figure 2F The annotations can be used as real-time feedback for clinicians involved in interventional medical procedures, as an aid similar to a roadmap or navigation tool, and / or as an aid similar to a sticky note.

[0071] Figures 2A-2F Any two or more methods described herein can be integrated within the scope of this disclosure. Furthermore, Figures 2A-2F The methods described can include additional features, such as storing features detected from interventional image recognition and tracking. The stored features can be used later, for example, if the endoscopic camera view moves to a significantly different area of ​​the organ and then returns to the initial field of view, allowing annotations such as virtual labels to be automatically provided again. The above combined... Figures 2A to 2F Some or all aspects of the methods and processes of the representative embodiments described can be derived from... Figure 5A System 500 implementation, or even by Figure 5B The controller 522, which executes software instructions, is implemented independently.

[0072] The stored features and annotations can also be used in subsequent interventional procedures, even on different days. For example, when one or more anatomical features are identified in a first interventional procedure and similar anatomical features are identified in a second interventional procedure, the similar features can be used in the second interventional procedure to automatically register the location and orientation of additional labels from the first interventional procedure. This results in less processing and faster annotation of known features. For example, tumor boundaries can be defined in both the first and second interventional procedures, and the area and / or volume of the tumor can be quantified in each of the first and second interventional procedures, making it possible to calculate differences between areas.

[0073] The stored features and annotations can also be used for reporting purposes after the interventional medical procedure. For example, multiple features and feature types can be saved to the patient's electronic medical record. Alternatively, when the feature is the type of implant / tool / device used, a count can be maintained for the purchasing system to restock these implants / tools / devices.

[0074] Figure 3 The illustration depicts another method for interactive endoscopy with intraoperative virtual annotation in VATS and minimally invasive surgery, according to a representative embodiment. (The above is combined with...) Figure 3 Some or all aspects of the methods and processes of the representative embodiments described can be derived from... Figure 5A System 500 implementation, or even by Figure 5B The controller 522, which executes software instructions, is implemented independently.

[0075] refer to Figure 3 At S310, the instrument is placed in the medical intervention setup, such as in an interventional medical procedure. For the purposes of explanation herein, the instrument is placed within or around the patient's anatomy within the view of the imaging apparatus. At S320, the anatomical feature of interest or the event to be recorded is identified. S320 represents a trigger for the annotation process described herein, wherein identifying the anatomical feature of interest and / or the event to be recorded for annotation is part of the final annotation of the interventional image.

[0076] At S330, the intention to annotate is input by the clinician via a touchscreen, keyboard, button, scroll pad, mouse, or other physical interface. The intention to annotate can also be input via audible commands or visual gestures detected by a speech recognition or video recognition mechanism implemented by a processor executing software instructions. At S340, the desired annotation is reviewed when it appears in the video feed. The reviewed annotation is generated from the intention input at S330 and can be an annotation superimposed on the video feed at S340 or even visually integrated into the video feed. The annotation can be about... Figures 2A to 2FThe embodiments describe the results of various processes. At S350, the clinician continues with surgery or another form of intervention within an interventional medical procedure that provides virtual annotations. In the embodiments, the process from S320 to S350 can be performed cyclically during the interventional medical procedure.

[0077] Figure 4 Three examples of virtual annotation for interactive endoscopic examinations for intraoperative virtual annotation in VATS and minimally invasive surgery are illustrated according to representative embodiments.

[0078] refer to Figure 4 In the first example of the annotation shown on the left, the tumor is labeled "tumor," the blood vessel is labeled "blood vessel," and the airway is labeled "airway." Furthermore, the airway is overlaid as an annotation with a virtual structure to clearly identify the airway. In the depicted example, the labels are offset from the tissue image, but this is not necessary, and the labels can be overlaid onto the tissue image or onto the interventional image at the edge. In the second example of the annotation, the anatomical surface is labeled "lobular cleft," and the tumor edge is labeled "tumor edge." Moreover, the boundary of the tumor edge is overlaid with a virtual structure to clearly identify the tumor and the tumor edge. In the first and second examples, the text labels could be replaced with simpler icons such as "A," "B," "C," "D," and "E," or numbers from "1" to "5." In the third example of the annotation shown on the right, the annotation includes a freehand curved outline and the label "X" overlaid on the tissue image. The curved outline is also labeled "organ boundary," which is offset from the tissue image. Additionally, a label with a checklist is overlaid on the tissue image in the lower right corner.

[0079] therefore, Figure 4 The illustrations show several examples of how interventional images can be annotated. Various annotations can vary in size, type, location, color, text, shadow, orientation, and more. Furthermore, some annotations can be provided for certain types of medical interventions but not others, or based on input from the individual incorporated into machine learning for a specific individual.

[0080] Examples of annotation types that may be provided as virtual annotations conforming to this disclosure include markers, labels, distinguishing colors, lines and curves, computer-aided design (CAD) models, and drawings. Markers include dots, stars, squares, dashes, arrows, and other symbolic icons that convey location, direction, anatomical context, time, or other relevant information. Labels may include text and other semantic representations that convey embedded information as markers. Distinguishing colors may be used to depict areas of tissue or may be used to represent other forms of information. Lines and curves may be used to indicate locations or boundaries of clinical interest and may be input, for example, via a touchscreen. CAD models may include segmented preoperative computed tomography (CT) images representing anatomical structures such as tumors, blood vessels, airways, or other tissues of interest. Drawings may allow personnel involved in the interventional medical procedure to arbitrarily outline the procedure on a video screen and allow marker locations to be updated for each video content. Figure 4 Examples of the aforementioned tag types are shown in the figure. Furthermore, tags or other forms of annotation can be variable to convey transitions through space or time, or to convey transient information such as the interaction between tools and organizations.

[0081] exist Figure 4 In the embodiments described herein, and for other embodiments herein, the position and / or shape of the virtual annotations can be updated with each video frame based on the image content. For example, if the lung is flipped over and the corresponding tissue is not in view, the shaded area of ​​a specific color, such as green, in the second example can disappear.

[0082] Figure 5A The illustration depicts a system for interactive endoscopic examination of intraoperative virtual annotations in VATS and minimally invasive surgery, according to a representative embodiment.

[0083] like Figure 5A As shown, system 500 includes a thoracoscope 510, a computer 520, a display 530, and an input mechanism 540. Computer 520 includes a controller 522. System 500 is a simplified block diagram of an environment that could otherwise be much more complex for implementing the teachings herein. System 500 may include some or all of the components of the interactive annotation system described herein. System 500 implements the combination as described herein. Figures 2A to 3 Some or all aspects of the methods and processes of representative embodiments.

[0084] Thoracoscope 510 is an example of an endoscope. Thoracoscope 510 is a slender camera typically used for examination, biopsy, and / or resection within the thoracic cavity (pleural cavity). For example, thoracoscope 510 transmits endoscopic images (e.g., video) to computer 520 via a wired connection and / or a wireless connection such as Bluetooth.

[0085] Computer 520 includes at least controller 522, but may include, as explained below, Figure 6 The controller 522 includes at least a memory storing software instructions and a processor executing the software instructions to directly or indirectly implement some or all aspects of the various processes described herein. For example, the computer 520 may include ports or other forms of communication interfaces for engagement with the thoracoscope 510 and the display 530.

[0086] Display 530 may be a video display that displays endoscopic or other interventional images derived from thoracoscope 510 and / or any other imaging equipment present in the environment where the interventional medical procedure takes place. Display 530 may be a monitor or television that displays video in color or black and white, and may also have audio capability to output audio signals.

[0087] Input device 540 may be or include a mouse, keyboard, touchpad, tablet computer, microphone, camera (for capturing, for example, gestures), or any other equipment through which a clinician can input instructions. Instructions input to input device 540 may be used in procedures such as those described herein for annotating endoscopic or other interventional images. As shown, input device 540 may communicate with computer 520, but may also communicate with display 530, such as when the input device is on, in, or otherwise connected to display 530 or a touchscreen connected to display 530. Input device 540 may communicate via wired or wireless connection and as described above, and may be part of or integrated with display 530 and / or computer 520.

[0088] Input mechanism 540 can be provided in various ways, allowing different methods to be selectively adapted to the specific context of various types of information and annotations. Input mechanism 540 may be or include a personal computer mouse and keyboard, such that the personal computer mouse is used to point to the desired annotation location (such as a lung fissure visible on the surface of a lung) or to draw a continuum along the desired location, and the keyboard is used to type text labels as annotations. Input mechanism 540 may be or include a touchscreen to perform tasks such as location selection and data input without a keyboard and mouse. External buttons mounted on the instrument and / or elsewhere can be used as input mechanism 540 to recreate the functionality of a mouse in a convenient location for one or more clinicians involved in the interventional medical procedure.

[0089] Image / video recognition software programs can be coupled with a camera as an input mechanism 540 to recognize gestures. For example, in an example of an image / video recognition software program, movement of surgical instruments within a specific pattern in an endoscopic view can activate annotation placement. Other examples of gestures that can be recognized by the image / video recognition software program include actions such as opening and closing a clamp twice in succession, rolling an object back and forth, or tapping tissue. The recognized gestures can be used to encode the type of annotation, or the type of annotation can be provided by other means discussed herein. Input mechanism 540 can also be or include speech recognition software. In an example of speech recognition software, voice commands can activate the placement, removal, or alteration of virtual annotations while providing the type of annotation to be used. Input mechanism 540 can also, for example, utilize the image / video recognition software program to recognize physical markers on tissue. For example, the image / video recognition software program can recognize physical labels that indicate where to place sutures or cauterization marks on a tissue surface to identify specific features. As other examples of input mechanism 540, virtual annotations can be automatically placed or suggested based on models, machine learning, or deep learning or other data-driven methods. Examples of machine learning described herein can analyze video content to identify features and actions. For example, using machine learning applied to previously instantiated features, features dynamically labeled during a specific medical intervention can be analyzed and detected based on the automatic labeling of similar features that behave similarly to previously labeled features. As another example of input facility 540, partial or semi-automatic entry of virtual annotations can trigger "auto-completion" of annotations. For example, when drawing blood vessels onto an image so that a preoperative model of the blood vessels is overlaid on a video, drawing can be used as a registration guide to implement auto-completion.

[0090] Machine learning can be provided to system 500 by a central system. For example, the central system receives instantiations of virtual annotations from system 500 via the Internet or other networks, and provides the results of machine learning to system 500 via the Internet or other networks based on instantiations of virtual annotations from a number of systems including system 500. For example, machine learning can be performed in a cloud-based processing system (such as at a data center). Alternatively, machine learning can be centrally implemented at a dedicated central computer system, such as by an entity related to system 500.

[0091] Figure 5B The illustration shows a controller for interactive endoscopic examination of intraoperative virtual annotation in VATS and minimally invasive surgery, according to a representative embodiment.

[0092] Figure 5B The controller 522 includes a processor 52210, a bus 52208, and a memory 52220. The controller 522 includes components for implementing the above-described combination. Figures 2A to 3The components of some or all aspects of the methods and processes of the representative embodiments described below. Figure 6 The processor 52210 is fully explained by the description of the processor in the computer system 600. The processor 52210 executes software instructions to implement the above-mentioned combination. Figures 2A to 3 Some or all aspects of the methods and processes of representative embodiments described below. Figure 6 The memory 52220 in the computer system 600 is described in detail below. Memory 52220 stores software instructions executed by processor 52210 to implement some or all aspects of the methods and processes described herein. Bus 52208 connects processor 52210 and memory 52220. Controller 522... Figure 5B The component is shown as a separate element, and this illustrates that controller 522 does not necessarily need to be provided as... Figure 5B The controller 522 described herein is part of the computer 520. More specifically, the controller 522 described herein may be provided as a standalone component to implement some or all aspects of the methods described herein, or may be integrated into a variety of devices, such as specialized medical technology items, laptop or desktop computers, or smartphones or tablets.

[0093] Figure 6 The illustration shows a general-purpose computer system according to another representative embodiment, on which interactive endoscopic examination methods for intraoperative virtual annotation in VATS and minimally invasive surgery can be implemented.

[0094] Figure 6 The computer system 600 illustrates a complete set of components for a communication device or computer device. However, the "controller" described herein can utilize fewer than Figure 6 The computer system 600 may be implemented using this set of components, such as a combination of memory and processor. The computer system 600 may include some or all of the elements of one or more component arrangements of the interactive annotation system described herein, but any such arrangement may not necessarily include one or more of the elements described for the computer system 600, and may include other elements not described.

[0095] refer to Figure 6 The computer system 600 includes a set of software instructions that can be executed to cause the computer system 600 to perform any of the methods or computer-based functions disclosed herein. The computer system 600 can operate as a standalone device or can be connected to other computer systems or peripheral devices, for example, using a network 601. In embodiments, the computer system 600 performs logic processing based on digital signals received via an analog-to-digital converter.

[0096] In a networked deployment, computer system 600 operates as a server in a server-client user network environment or as a client user computer, or as a peer-to-peer (or distributed) computer system in a peer-to-peer (or distributed) network environment. Computer system 600 can also be implemented as or incorporated into various devices, such as fixed computers, mobile computers, personal computers (PCs), laptop computers, tablet computers, or any other machine capable of executing a set of software instructions (sequential or otherwise) specifying actions to be performed by that machine. Computer system 600 can be incorporated as or within a device, which in turn is in an integrated system including additional devices. In embodiments, computer system 600 can be implemented using electronic devices that provide voice, video, and / or data communications. Furthermore, although computer system 600 is illustrated in the singular, the term "system" should also be taken to include any collection of systems or subsystems that individually or jointly execute one or more sets of software instructions to perform one or more computer functions.

[0097] like Figure 6 As illustrated, the computer system 600 includes a processor 610. The processor 610 can be considered as... Figure 5B The processor 52210 of the controller 522 is a representative example and executes instructions to implement some or all aspects of the methods and processes described herein. The processor 610 is tangible and non-transient. As used herein, the term "non-transient" should not be interpreted as a perpetual characteristic of a state, but rather as a characteristic of a state that will persist over a period of time. The term "non-transient" explicitly negates fleeting characteristics, such as carrier waves or signals, or other forms of characteristics that exist only transiently in any place at any time. The processor 610 is an article of manufacture and / or a machine part. The processor 610 is configured to execute software instructions to perform the functions described in the various embodiments herein. The processor 610 may be a general-purpose processor or may be part of an application-specific integrated circuit (ASIC). The processor 610 may also be a microprocessor, microcomputer, processor chip, controller, microcontroller, digital signal processor (DSP), state machine, or programmable logic device. The processor 610 may also be logic circuitry, including a programmable gate array (PGA) such as a field-programmable gate array (FPGA), or another type of circuitry including discrete gate and / or transistor logic. Processor 610 may be a central processing unit (CPU), a graphics processing unit (GPU), or both. Furthermore, any processor described herein may include multiple processors, parallel processors, or both. Multiple processors may be included in a single device or multiple devices, or coupled to a single device or multiple devices.

[0098] As used herein, the term "processor" encompasses an electronic component capable of executing programs or machine-executable instructions. References to computing devices that include "processor" should be interpreted as including more than one processor or processing core, as is the case in a multi-core processor. A processor can also refer to a collection of processors within a single computer system or distributed across multiple computer systems. The term computing device should also be interpreted as including a collection or network of computing devices, each including one or more processors. A program has software instructions that are executed by one or more processors, which may be within the same computing device or distributed across multiple computing devices.

[0099] The computer system 600 also includes main memory 620 and static memory 630, wherein the memories in the computer system 600 communicate with each other and with the processor 610 via bus 608. Either or both of main memory 620 and static memory 630 can be considered as... Figure 5B The memory 52220 of the controller 522 is a representative example and stores instructions used to implement some or all aspects of the methods and processes described herein. The memory described herein is a tangible storage medium for storing data and executable software instructions, and is non-transient during the time the software instructions are stored therein. As used herein, the term "non-transient" should not be construed as a perpetual characteristic of a state, but rather as a characteristic of a state that will persist over a period of time. The term "non-transient" explicitly negates fleeting characteristics, such as carrier waves or signals, or other forms of characteristics that exist only transiently in any place at any time. Main memory 620 and static memory 630 are articles of manufacture and / or machine parts. Main memory 620 and static memory 630 are computer-readable media from which data and executable software instructions can be read by a computer (e.g., processor 610). Each of the main memory 620 and the static memory 630 may be implemented as one or more of random access memory (RAM), read-only memory (ROM), flash memory, electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, magnetic tapes, optical disc read-only memory (CD-ROM), digital versatile disks (DVDs), floppy disks, Blu-ray discs, or any other form of storage medium known in the art. The memory may be volatile or non-volatile, secure and / or encrypted, insecure and / or unencrypted.

[0100] “Memory” is an example of a computer-readable storage medium. Computer memory is any memory that can be directly accessed by a processor. Examples of computer memory include, but are not limited to, RAM, registers, and register files. The reference to “computer memory” or “memory” should be interpreted as potentially referring to multiple memories. Memory can be, for example, multiple memories within the same computer system. Memory can also be multiple memories distributed across multiple computer systems or computing devices.

[0101] As shown in the figure, the computer system 600 also includes a video display unit 650, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), a flat panel display, a solid-state display, or a cathode ray tube (CRT). Furthermore, the computer system 600 includes input devices 660 (such as a keyboard / virtual keyboard or a touch-sensitive input screen or voice input with voice recognition) and cursor control devices 670 (such as a mouse or a touch-sensitive input screen or board). The computer system 600 also optionally includes a disk drive unit 680, a signal generation device 690 (such as a speaker or remote control), and / or a network interface device 640.

[0102] In an embodiment, such as Figure 6 As depicted herein, the disk drive unit 680 includes a computer-readable medium 682 in which one or more sets of software instructions 684 (software) are embedded. The software instructions 684 are read from the computer-readable medium 682 for execution by the processor 610. Furthermore, the software instructions 684, when executed by the processor 610, perform one or more steps of the methods and processes described herein. In embodiments, the software instructions 684 reside wholly or partially within main memory 620, static memory 630, and / or processor 610 during execution by the computer system 600. Additionally, the computer-readable medium 682 may include the software instructions 684 or receive and execute the software instructions 684 in response to a propagated signal, causing a device connected to network 601 to transmit voice, video, or data through network 601. The software instructions 684 may be sent or received via network interface device 640 through network 601.

[0103] In alternative embodiments, dedicated hardware implementations, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic arrays, and other hardware components, are constructed to implement one or more of the methods described herein. One or more embodiments described herein may use two or more specific interconnected hardware modules or devices to implement functionality using associated control and data signals that can communicate between and through the modules. Therefore, this disclosure covers software, firmware, and hardware implementations. Nothing in this application should be construed as being implemented or feasible solely using software without utilizing hardware such as tangible non-transient processors and / or memory.

[0104] According to various embodiments of this disclosure, the methods described herein can be implemented using a hardware computer system that executes software programs. Furthermore, in exemplary non-limiting embodiments, implementations may include distributed processing, component / object distributed processing, and parallel processing. Virtual computer system processing can implement one or more of the methods or functions described herein, and the processors described herein can be used to support virtual processing environments.

[0105] Therefore, interactive endoscopy for intraoperative virtual annotation in VATS and minimally invasive surgery effectively transforms the interventional imaging modality from a passive to an interactive, image-based modality, providing a user interface for personnel involved in the interventional medical process, allowing for digital labeling of the surgical scene during the procedure. This allows personnel involved in the intervention to keep track of the anatomical structures seen and the tasks performed, which helps eliminate over- and / or redundant exploration. Labeling of anatomical structures during the interventional medical process also helps maintain data for retrospective review and for feedback to optimize future interventional medical procedures. In the long term, the ability to accumulate annotated (e.g., labeled) surgical videos can be used to develop machine learning to further improve the integration of information and analysis, thereby enhancing endoscopic surgery. As more surgical videos are labeled, automatic labeling becomes particularly important, which in turn is achieved through a user-friendly interface for interactive endoscopy. The ability to label intraoperatively has the potential to generate a large amount of labeled endoscopic data. Such a volume of data can then be used to improve deformable registration and accurate overlay of preoperative images onto endoscopic views.

[0106] A fixed library of virtual labels can also be used in the automation process, enabling the addition or removal of virtual annotations to raw endoscopic images even without machine learning. In one embodiment, raw aspects of the endoscopic image can be removed, such as by overlaying them with virtual annotations of a specific color.

[0107] In another example, instead of supplementing partial input, virtual annotations can be provided under a supervised automated process. For instance, a user can indicate a specific structure of interest on an endoscopic image, and the supervised automated process can identify the structure by classification, delineate its boundaries, identify the location of virtual labels near the structure, and track the structure as it moves. In this embodiment, even a single initial instruction can be used to implement virtual annotations.

[0108] Examples of places where intraoperative virtual annotation can be used include lung surgery where a target location needs to be reached and potentially removed. Intraoperative virtual annotation can also be used for tumor resection, lymph node dissection and resection, foreign body removal, etc.

[0109] Although an interactive endoscope for intraoperative virtual annotation in VATS and minimally invasive surgery has been described with reference to several exemplary embodiments, it should be understood that the language used is descriptive and illustrative, not limiting. Changes may be made within the scope of the appended claims, as presently stated and modified, without departing in any respect from the scope and spirit of the interactive endoscope for intraoperative virtual annotation in VATS and minimally invasive surgery. While an interactive endoscope for intraoperative virtual annotation in VATS and minimally invasive surgery has been described with reference to specific modules, materials, and embodiments, it is not intended to be limited to the disclosed details; rather, it extends to functionally equivalent structures, methods, and uses such as those within the scope of the appended claims.

[0110] The illustrations of the embodiments described herein are intended to provide a general understanding of the structure of various embodiments. These illustrations are not intended to be a complete description of all elements and features disclosed herein. Many other embodiments will be apparent to those skilled in the art upon viewing this disclosure. Other embodiments may be utilized and derived from this disclosure, allowing structural and logical substitutions and changes to be made without departing from the scope of this disclosure. Furthermore, the illustrations are merely representative and may not be drawn to scale. Some scales within the illustrations may be exaggerated, while others may be minimized. Therefore, this disclosure and the accompanying drawings should be considered illustrative rather than restrictive.

[0111] This document refers to one or more embodiments of the present disclosure individually and / or collectively by way of the term "invention" for convenience only, and is not intended to voluntarily limit the scope of this application to any particular invention or inventive concept. Furthermore, although specific embodiments have been illustrated and described herein, it should be appreciated that any subsequent arrangements designed to achieve the same or similar purpose may replace the specific embodiments shown. This disclosure is intended to cover any and all subsequent modifications or variations of the various embodiments. After reviewing this specification, combinations of the above embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art.

[0112] This abstract of disclosure is provided in accordance with 37 C. FR § 1.72(b) and is submitted with the understanding that it is not intended to interpret or limit the scope or meaning of the claims. Furthermore, in the foregoing detailed description, various features may be combined together or described in a single embodiment for the purpose of simplifying this disclosure. This disclosure should not be construed as reflecting an intention that the claimed embodiments require more features than expressly recited in each claim. Rather, as reflected in the following claims, the inventive subject matter may relate to fewer than all features of any of the disclosed embodiments. Therefore, the following claims are incorporated into the detailed description, each claim serving as itself to define a separately claimed subject matter.

[0113] The prior description of the disclosed embodiments is provided to enable those skilled in the art to practice the concepts described in this disclosure. Thus, the subject matter of the above disclosure is to be considered illustrative rather than restrictive, and the claims are intended to cover all such modifications, enhancements, and other embodiments falling within the true spirit and scope of this disclosure. Therefore, to the fullest extent permitted by law, the scope of this disclosure shall be determined by the broadest permissible interpretation of the claims and their equivalents, and shall not be bound or limited by the foregoing detailed description.

Claims

1. A controller (522) for live annotation of interventional imagery, comprising: a memory (52220) that stores software instructions; and a processor (52210) that executes the software instructions, wherein, when executed by the processor (52210), the software instructions cause the controller (522) to implement a process comprising: receiving (S210) interventional imagery during an intraoperative intervention; automatically analyzing (S220) the interventional imagery for a detectable feature; detecting (S230) a detectable feature; determining (S240) to add an annotation to the interventional imagery for the detectable feature; identifying (S250) a location for the annotation in the interventional imagery as an identified location; adding (S260) the annotation to the interventional imagery at the identified location to correspond to the detectable feature; outputting (S270), during the intraoperative intervention, a video based on the interventional imagery and the annotation as a video output, the annotation including the annotation overlaid on the interventional imagery at the identified location; applying machine learning to the video output; and identifying, based on the machine learning, an optimized placement of the annotation, wherein the optimized placement of the annotation is learned by applying the machine learning to at least one prior instantiation of live annotation. the process implemented when the processor (52210) executes the software instructions further comprises:

2. The controller (522) of claim 1, wherein, detecting (S230 / S231) an anatomical structure as the detectable feature; and tracing (S232) a boundary of the anatomical structure. the process implemented when the processor (52210) executes the software instructions further comprises:

3. The controller (522) of claim 1, wherein, detecting (S230) an interventional tool as the detectable feature; and tracing (S232) a boundary of the interventional tool. the process implemented when the processor (52210) executes the software instructions further comprises:

4. The controller (522) of claim 1, wherein, identifying (S240 / S241) predetermined information for the detectable feature from a source external to the controller (522); and adding (S260 / S262) the predetermined information from the source to the interventional imagery as the annotation, wherein the annotation includes a virtual label to virtually mark the detectable feature with the predetermined information. the process implemented when the processor (52210) executes the software instructions further comprises:

5. The controller (522) of claim 1, wherein, detecting (S230 / S231) an anatomical structure as the detectable feature; marking (S260 / S261) the anatomical structure with a virtual label as the annotation added to the interventional imagery; tracking (S271) movement of the anatomical structure relative to an endoscope that generates the video output; detecting (S272) movement of the anatomical structure relative to the endoscope that generates the video output; and automatically adjusting (S273) a position and orientation of the virtual label based on detecting movement of the anatomical structure relative to the endoscope. ​ 6. The controller (522) of claim 1, wherein, The processes implemented when the processor (52210) executes the software instructions further include: detecting and tracking (S221) low-level features in the interventional imagery using a first algorithm; extracting (S222) the low-level features as extracted low-level features, and identifying and classifying (S223) anatomical structures based on the extracted low-level features as the detectable features in the interventional imagery using a second algorithm.

7. The controller (522) of claim 1, wherein, The processes implemented when the processor (52210) executes the software instructions further include: automatically analyzing (S220) the interventional imagery for detectable features in response to a first instruction; automatically identifying (S250) the location for the annotation in response to a second instruction; and automatically adding (S260) the annotation to the interventional imagery at the identified location in response to a third instruction.

8. The controller (522) of claim 1, wherein, The processes implemented when the processor (52210) executes the software instructions further include: automatically adding (S240) the annotation to the interventional imagery at the identified location in response to an instruction to add the annotation based on the instruction.

9. The controller (522) of claim 1, wherein, The processes implemented when the processor (52210) executes the software instructions further include: automatically identifying (S250) the location for the annotation in response to an instruction to identify a location for the annotation.

10. The controller (522) of claim 1, wherein, The processes implemented when the processor (52210) executes the software instructions further include: automatically detecting (S220 / S230) a structure for virtual labeling in response to an instruction to detect a structure for virtual labeling.

11. The controller (522) of claim 1, wherein, The processes implemented when the processor (52210) executes the software instructions further include: detecting (S272) movement of an anatomical structure relative to an endoscope that generated the interventional imagery; and automatically adjusting (S273) a position and orientation of the annotation based on detecting the movement of the anatomical structure relative to the endoscope.

12. The controller (522) of claim 1, wherein, The processes implemented when the processor (52210) executes the software instructions further include: analyzing (S211) intraoperative information from the intraoperative intervention; deriving (S240 / S242) content for the annotation based on the intraoperative information, and adding (S260 / S263) the content to the interventional imagery as the annotation, wherein the annotation comprises a virtual label.

13. The controller (522) of claim 1, wherein, The processes implemented when the processor (52210) executes the software instructions further include: obtaining (S205) a preoperative segmentation model of an anatomical structure; receiving (S260 / S264) an instruction to annotate the anatomical structure with the preoperative segmentation model as an annotation, and virtually replacing (S260 / S265) the anatomical structure in the interventional imagery with the preoperative segmentation model of the anatomical structure.

14. The controller (522) of claim 13, wherein, The processes implemented when the processor (52210) executes the software instructions further include: registering (S266) the preoperative segmentation model of the anatomical structure to the anatomical structure, wherein the instructions include selection of an icon corresponding to the pre-operative segmentation model.

15. The controller (522) of claim 1, wherein, the process further includes, when executed by the processor (52210): detecting (S220 / S230 / S320) a trigger during the intra-operative intervention; and changing (S240 / S241 / S242) the annotation based on detecting the trigger.

16. A system for live annotation of interventional imagery, comprising: a display (530) that displays the interventional imagery; a controller (522) that includes a memory (52220) that stores software instructions; and a processor (52210) that executes the software instructions, wherein the software instructions, when executed by the processor (52210), cause the controller (522) to implement a process that includes: receiving (S210) interventional imagery during an intra-operative intervention; automatically analyzing (S220) the interventional imagery for a detectable feature; detecting (S230) a detectable feature; determining (S240) to add an annotation to the interventional imagery for the detectable feature; identifying (S250) a location for the annotation in the interventional imagery as an identified location; adding (S260) the annotation to the interventional imagery at the identified location to correspond to the detectable feature; outputting (S270) a video on the display based on the interventional imagery and the annotation during the intra-operative intervention as a video output, the annotation including the annotation overlaid on the interventional imagery at the identified location; applying machine learning to the video output; and identifying an optimized placement of the annotation based on the machine learning, wherein the optimized placement of the annotation is learned by applying the machine learning to at least one prior instantiation of live annotation.

17. The system of claim 16, wherein, the detectable feature is detected based on a result of applying the machine learning to at least one prior instantiation of live annotation, and wherein the identified location is identified based on a result of applying the machine learning to the at least one prior instantiation of live annotation.

18. A tangible, non-transitory computer-readable storage medium (52220 / 620 / 630) that stores a computer program that, when executed by a processor (52210 / 610), causes a system (500) that includes the tangible, non-transitory computer-readable storage medium (52220 / 620 / 630) to perform a process for live annotation of interventional imagery, the process, when executed by the processor (52210 / 610), includes: receiving (S210) interventional imagery from a thoracoscope during a video- assisted thoracoscopic surgery for a lung tumor resection, the thoracoscope producing the interventional imagery as a view of an inside of a chest; automatically analyzing (S220) the interventional imagery for a detectable feature by a processor (52210) executing software instructions from a memory (52220); detecting (S230) a detectable feature; determining (S240) to add an annotation to the interventional imagery for the detectable feature; identifying (S250), in the interventional imagery, a location for the annotation as an identified location; adding (S260) the annotation to the interventional imagery at the identified location to correspond to the detectable feature; outputting (S270), during the video-assisted thoracoscopic surgery, a video as a video output based on the interventional imagery and the annotation, the annotation including the annotation overlaid on the interventional imagery at the identified location; applying machine learning to the video output; and identifying, based on the machine learning, an optimized placement of the annotation, wherein the optimized placement of the annotation is learned by applying the machine learning to at least one previous instantiation of live annotation.

19. The tangible, non-transitory computer-readable storage medium (52220 / 620 / 630) of claim 18, wherein, the detectable feature is detected based on results of applying the machine learning to at least one previous instantiation of live annotation, and wherein the identified location is identified based on results of applying the machine learning to the at least one previous instantiation of live annotation.

20. The tangible, non-transitory computer-readable storage medium (52220 / 620 / 630) of claim 18, wherein, the process further includes, when the processor (52210 / 610) executes the computer program: detecting (S230 / S231) an anatomical structure as the detectable feature; labeling (S260 / S261) the anatomical structure with a virtual tag as the annotation added to the interventional imagery; tracking (S271) movement of the anatomical structure relative to a scope generating the video output; detecting (S272) movement of the anatomical structure relative to the scope generating the video output; and automatically adjusting (S273) a position and orientation of the virtual tag based on detecting movement of the anatomical structure relative to the scope.

Images