Real-time EBUS-TBNA staging guidance techniques

Abstract

EBUS-TBNA staging techniques are described that identify sampling targets and generate staging plans based on scan data, such as pre-procedural and / or intra-procedural scan data. The techniques overlay a graphical representation of the staging plan onto real¬ time ultrasound images during an EBUS procedure and then display a route for sampling the lymph node stations.

Description

REAL-TIME EBUS-TBNA STAGING GUIDANCE TECHNIQUESCLAIM OF PRIORITY

[0001] This application claims the benefit of priority to U.S. Provisional Application Serial No. 63 / 651,517, titled “REAL-TIME EBUS-TBNA STAGING GUIDANCE TECHNIQUES,” to Dawn Potts, and filed on May 24, 2024, which is incorporated by reference herein in its entirety.FIELD OF THE DISCLOSURE

[0002] This document pertains generally, but not by way of limitation, to overlay of medical image data.BACKGROUND

[0003] Endobronchial Ultrasound-Guided Transbronchial Needle Aspiration (EBUS- TBNA) is a minimally invasive medical procedure that enhances the capabilities of standard Endobronchial Ultrasound (EBUS) for diagnosing lung diseases, including lung cancer, infections, and other thoracic conditions. EBUS-TBNA combines the visual advantages of bronchoscopy with the imaging precision of ultrasound, along with the added capability of needle aspiration. This integration allows for the direct sampling of tissue from lung lesions, lymph nodes, and other structures adjacent to the airways.

[0004] EBUS-TBNA employs a specialized bronchoscope with an ultrasound transducer at its tip and a channel for needle aspiration. When the bronchoscope is navigated to the target area within the lungs, the ultrasound component generates detailed images of lung tissues, airway walls, and nearby structures, facilitating precise needle aspiration of these areas.This technique is particularly beneficial for sampling lymph nodes or masses adjacent to the airways but outside the reach of traditional bronchoscopic biopsy methods.

[0005] One of the significant advantages of EBUS-TBNA is its ability to perform real-time image-guided needle biopsies. This capability is crucial for accurately sampling tissue for diagnostic purposes, reducing the need for more invasive procedures like mediastinoscopy or thoracotomy. EBUS-TBNA significantly enhances the precision and safety of lung biopsies, leading to improved diagnosis and staging of lung cancer and other diseases.

[0006] EBUS-TBNA is particularly valuable in diagnosing conditions like tuberculosis, sarcoidosis, and lymphoma. It plays a pivotal role in the staging of lung cancer, determining the extent of the disease and guiding the treatment plan. This procedure is typically well-tolerated by patients and can be performed under moderate sedation or general anesthesia. With its low complication rate and high diagnostic yield, EBUS-TBNA represents a significant advancement in the field of pulmonary medicine, offering a less invasive, yet highly effective alternative to traditional diagnostic methods for thoracic diseases.SUMMARY OF THE DISCLOSURE

[0007] This disclosure describes EBUS-TBNA staging techniques that identify sampling targets and generate staging plans based on scan data, such as pre-procedural and / or intraprocedural scan data. The techniques overlay a graphical representation of the staging plan onto real-time ultrasound images during an EBUS procedure and then display a route for sampling the lymph node stations.

[0008] In some aspects, this disclosure is directed to a system for providing real-time EBUS-TBNA staging guidance during an endobronchial ultrasound (EBUS) procedure, the system comprising: an EBUS device configured to capture an ultrasound image of a patient's airway; a user interface including a display; and a processing unit configured for: generating a staging plan that is based on scan data indicating inflammation of a lymph node; overlaying, on the user interface, a graphical representation of the ultrasound image with a graphical representation of the staging plan including lymph node stations indicated by the staging plan; and displaying, on the user interface, a route for sampling the lymph node stations.

[0009] In some aspects, this disclosure is directed to a method for providing real-time EBUS-TBNA staging guidance during an endobronchial ultrasound (EBUS) procedure, the method comprising: capturing an ultrasound image of a patient's airway; generating a staging plan that is based on scan data indicating inflammation of a lymph node; overlaying a graphical representation of the ultrasound image with a graphical representation of the staging plan including lymph node stations indicated by the staging plan; and displaying a route for sampling the lymph node stations.

[0010] In some aspects, this disclosure is directed to a system for providing real-time EBUS-TBNA staging guidance during an endobronchial ultrasound (EBUS) procedure, the system comprising: an EBUS device configured to capture an ultrasound image of a patient's airway; a user interface including a display; and a processing unit configured for: generating a staging plan that is based on scan data indicating inflammation of a lymph node, wherein the scan data includes pre-procedural scan data including PET scan data and CT scan data, wherein the staging plan is based on a CT-PET score using a combination ofthe CT scan data and the PET scan data, and wherein the CT-PET score is indicative of disease activity; overlaying, on the user interface, a graphical representation of the ultrasound image with a graphical representation of the staging plan including lymph node stations indicated by the staging plan; and displaying, on the user interface, a route for sampling the lymph node stations.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views. Similar components in different views may be described by like numerals. The drawings illustrate generally, by way of example, but not by way of limitation, various embodiments discussed in the present document.

[0012] FIG. 1 is a schematic diagram of an example of an endoscopy system including an imaging and control system and an endoscope that may implement various techniques of this disclosure.

[0013] FIG. 2 is a schematic diagram of the endoscopy system of FIG. 1 including the endoscope connected to a control unit of the imaging and control system.

[0014] FIG. 3 is a perspective view of a transbronchial needle aspiration system including an ultrasound sensor that may used with the techniques of this disclosure.

[0015] FIG. 4 depicts an example of an ultrasound image overlaid with a label indicating identified anatomical structures in accordance with this disclosure.

[0016] FIG. 5 depicts another example of an ultrasound image overlaid with a label indicating identified anatomical structures in accordance with this disclosure.

[0017] FIG. 6 depicts another example of an ultrasound image overlaid with a label indicating identified anatomical structures in accordance with this disclosure.

[0018] FIG. 7 shows a schematic diagram of an example of a computer-based lymph node identifier.

[0019] FIG. 8 shows a schematic diagram of an example of a trained machine-learning model.

[0020] FIG. 9 illustrates an aspect of the subject matter in accordance with one embodiment.

[0021] FIG. 10 is a block diagram illustrating an example of a machine upon which one or more examples may be implemented.

[0022] FIG. 11 is a graphical representation an ultrasound imaging depicting the lymph node stations and bronchial tree anatomy in a patient along with a staging plan and recommended route in accordance with this disclosure.

[0023] FIG. 12 is a graphical representation of an example of a procedural progress 1200 of the staging plan and recommended route of FIG. 11 in accordance with this disclosure.

[0024] FIG. 13 is a graphical representation of an example of the lymph node stations and bronchial tree anatomy in a patient along with an orientation of the EBUS device in accordance with this disclosure.

[0025] FIG. 14 is a flow diagram of an example of a method for providing real-time EBUS-TBNA staging guidance during an EBUS procedure in accordance with this disclosure.DETAILED DESCRIPTION

[0026] Staging refers to determining how far a lung cancer has spread in the body. It is an important step after diagnosis as it assists in the treatment plan for patients. Staging involves evaluating the size and locations of the primary tumor, assessing if cancer cells have spread to nearby lymph nodes, and checking for metastases in more distant organs. The most common staging system for lung cancer is the TNM system, where T describes the primary tumor size and invasiveness, N indicates regional lymph node involvement, and M notes the presence or absence of distant metastases.

[0027] By assigning numeric values and letters to each category, an overall stage can be assigned, usually ranging from Stage I (early, localized cancer) to Stage IV (advanced, metastatic cancer). Accurately staging via procedures like endobronchial ultrasound (EBUS) allows the selection of personalized treatment options for the best outcome.

[0028] EBUS procedures are the current standard of care for staging non-small cell lung cancer by sampling lymph nodes to assess disease spread. However, these procedures may be long and complex, requiring sampling multiple lymph node stations across different regions of the lung. Currently, physicians mentally track which lymph nodes to sample and in what order, potentially leading to incomplete staging or unnecessary sampling. The present inventor has recognized the need for a system that generates staging plans and that provides real-time guidance during EBUS-TBNA procedures.

[0029] This disclosure describes EBUS-TBNA staging techniques that identify sampling targets and generate staging plans based on scan data, such as pre-procedural and / or intra-procedural scan data. The techniques overlay a graphical representation of the staging plan onto real-time ultrasound images during an EBUS procedure and then display a route for sampling the lymph node stations.

[0030] The techniques of this disclosure offer one or more of the following advantages: (1) ensure complete and systematic staging by displaying a route with indicators representing pre-determined lymph nodes from which samples need to be taken; (2) provide real-time feedback with indicators representing lymph node stations from which samples have been taken; (3) display a graphical representation of the EBUS device relative to the lymph node stations; (4) update the staging plan based on intra-procedural scan data including indications of lymph node size, shape, and density; and (5) generate CT-PET scores based on pre-procedural scan data to indicate disease activity and guide the staging process.

[0031] In addition, existing EBUS systems allow for imaging of anatomical vascular structures, but do not confirm or label those structures or lymph nodes. The present inventor has recognized that due to the intricate lung anatomy, variation across patients, and physician skillset, some physicians may lose orientation during EBUS procedures, such as thinking they are at one lymph node station when they are at another lymph node station, leading to inaccurate lymph node identification and staging.

[0032] To address this issue and other issues, the present inventor has recognized the desirability of providing real-time labeling and identification of key anatomical structures during EBUS procedures. This disclosure describes techniques to utilize machine learning to analyze EBUS ultrasound imagery. Models are trained on annotated samples to identify important vascular landmarks, lymph node stations, and other structures. The techniques of this disclosure have a number of advantages over existing systems, such as the following: (1) improving patient safety by clearly labeling the anatomical vascular structure, allowing the physician to sample cautiously if in proximity of the anatomic vascular structure; (2) reducing procedure time; (3) reducing procedure and technology learning curve; (4) increasing accuracy of staging by ensuring the correct lymph nodes are sampled, resulting in more accurate treatment plans; and (5) reducing procedure time related to losing position and orientation during procedure.

[0033] FIG. 1 is a schematic diagram of an example of an endoscopy system 10 including an imaging and control system 12 and an endoscope 14 that may implement various techniques of this disclosure. The endoscopy system 10 is suitable for use with the systems, devices, and methods described below, such as modular endoscopy systems, modular endoscopes, and methods for designing, building, and deconstructing endoscopes.According to some examples, the endoscope 14 may be insertable into an anatomical region for imaging and / or to provide passage of one or more sampling devices for biopsies, or one or more therapeutic devices for treatment of a disease state associated with the anatomical region. Endoscope 14 may, in advantageous aspects, interface with and connect to imaging and control system 12. In the example shown, the endoscope 14 includes a duodenoscope, though other types of endoscopes may be used with the features and teachings of the present disclosure.

[0034] The imaging and control system 12 may include a controller 16, a user interface including an output unit 18 (e.g., display) and an input unit 20, a light source 22, a fluid source 24, and a suction pump 26. The imaging and control system 12 may include various ports for coupling with the endoscopy system 10. For example, the controller 16 may include a data input / output port for receiving data from and communicating data to the endoscope 14.

[0035] The light source 22 may include an output port for transmitting light to the endoscope 14, such as via a fiber optic link. The fluid source 24 may include a port for transmitting fluid to the endoscope 14. The fluid source 24 may comprise a pump and a tank of fluid or may be connected to an external tank, vessel, or storage unit. The suction pump 26 may include a port used to draw a vacuum from the endoscope 14 to generate suction, such as for withdrawing fluid from the anatomical region into which the endoscope 14 is inserted. The output unit 18 and the input unit 20 may be used by an operator of the endoscopy system 10 to control functions of the endoscopy system 10 and view the output of the endoscope 14.

[0036] The controller 16 may additionally be used to generate signals or other outputs from treating the anatomical region into which the endoscope 14 is inserted. In some examples, the controller 16 may generate electrical output, acoustic output, a fluid output, and the like for treating the anatomical region with, for example, cauterizing, cutting, freezing, and the like.

[0037] The endoscope 14 may include an insertion section 28, a functional section 30, and a handle section 32, which may be coupled to a cable section 34 and a coupler section 36. The insertion section 28 may extend distally from the handle section 32 and the cable section 34 may extend proximally from the handle section 32. The insertion section 28 may be elongated and include a bending section, and a distal end to which functional section 30 may be attached. The bending section may be controllable (e.g., by a control knob 38 on a handle section 32) to maneuver the distal end through tortuous anatomical passageways(e.g., stomach, duodenum, kidney, ureter, etc.) The insertion section 28 may also include one or more working channels (e.g., an internal lumen) that may be elongated and support the insertion of one or more therapeutic tools of the functional section 30. The working channel may extend between the handle section 32 and the functional section 30. Additional functionalities, such as fluid passages, guide wires, and pull wires may also be provided by the insertion section 28 (e.g., via suction or irrigation passageways, and the like).

[0038] The handle module 32 may include a knob 38 as well as ports 40. The knob 38 may be coupled to a pull wire extending through the insertion section 28. The ports 40 may be configured to couple various electrical cables, fluid tubes, and the like to the handle module 32 for coupling with the insertion section 28.

[0039] The imaging and control system 12, according to some examples, may be provided on a mobile platform (e.g., a cart 41) with shelves for housing the light source 22, the suction pump 26, the image processing unit 42, etc. Alternatively, several components of the imaging and control system 12 that are shown in FIG. 1 and FIG. 2 may be provided directly on the endoscope 14 to make the endoscope “self-contained.”

[0040] FIG. 2 is a schematic diagram of the endoscopy system 10 of FIG. 1 including the imaging and control system 12 and endoscope 14. FIG. 2 schematically illustrates components of the imaging and control system 12 coupled to the endoscope 14, which in the illustrated example includes a duodenoscope. The imaging and control system 12 may include the controller 16, which may include or be coupled to the image processing unit 42, the treatment generator 44 and the drive unit 46, as well as the light source 22, the input unit 20, and the output unit 18. The output unit 18, e.g., a display unit, and the input unit 20 form part of a user interface 200. The image processing unit 42 includes one or more processors that may be distributed, such as locally and remotely, or co-located at one location.

[0041] The image processing unit 42 and the light source 22 may each interface with the endoscope 14 by wired or wireless electrical connections. The imaging and control system 12 may accordingly illuminate an anatomical region, collect signals representing the anatomical region, process signals representing the anatomical region, and display images representing the anatomical region on the display unit 18. The imaging and control system 12 may include the light source 22 to illuminate the anatomical region using light of desired spectrum (e.g., broadband white light, narrow-band imaging using preferred electromagnetic wavelengths, and the like). The imaging and control system 12 may connect (e.g., via anendoscope connector) to the endoscope 14 for signal transmission (e.g., light output from light source, video signals from imaging system in the distal end, and the like).

[0042] The fluid source 24 may include one or more sources of air, saline or other fluids, as well as associated fluid pathways (e.g., air channels, irrigation channels, suction channels) and connectors (barb fittings, fluid seals, valves and the like). The imaging and control system 12 may also include the drive unit 46, which may be an optional component. The drive unit 46 may include a motorized drive for advancing a distal section of endoscope 14, as described in PCT Pub. No. WO 2011 / 140118 Al to Frassica et al., titled “Rotate-to- Advance Catheterization System,” which is hereby incorporated in its entirety by reference.

[0043] One or more sensors are in electrical communication with the controller 16. For example, the endoscopy system 10 may include one or both of a sensor 202 and a shape sensor 204. The sensor 202 may be positioned adjacent a distal tip of an EBUS device, such as the EBUS bronchoscope 306 of FIG. 3, and configured for detecting a position of the EBUS device relative to the patient's airway and providing positioning data to the processing unit, such as the image processing unit 42. The image processing unit 42 may then cause the user interface 200 to display data representing the position of the EBUS device. In some examples, the sensor 202 is a magnetic sensor.

[0044] Alternatively or additionally, the endoscopy system 10 may also include a shape sensor 204. Shape sensors are well-known and will not be described in detail. An example of a shape sensor is described in U.S. Patent No. 10,823,627, titled “Shape sensing with multi-core fiber sensor” to Sanborn et al., the entire contents of which being incorporated herein by reference. The shape sensor 204 may be integrated into a portion of the EBUS bronchoscope 306 of FIG. 3, or some other portion of the insertion section 28 of the endoscopy system 10. In some examples, the shape sensor 204 is configured for detecting an orientation of the EBUS bronchoscope relative to the patient's airway and providing orientation data to the processing unit, such as the image processing unit 42. The image processing unit 42 may then cause the user interface 200 to display data representing the orientation of the EBUS bronchoscope. In some examples, the shape sensor 204 includes a plurality of fiber Bragg gratings.

[0045] In accordance with this disclosure, the endoscopy system 10 of FIGS. 1 and 2 is an example of a system for providing real-time EBUS-TBNA staging guidance during an endobronchial ultrasound (EBUS) procedure. As described in more detail below, using an EBUS device configured to capture an ultrasound image of a patient's airway, such as the EBUS bronchoscope 306 of FIG. 3, the system for providing real-time EBUS-TBNAstaging guidance during an EBUS procedure identifies sampling targets and generates staging plans based on scan data, such as pre-procedural and / or intra-procedural scan data. A processing unit, such as the image processing unit 42 of FIG. 2, is configured for overlaying on a user interface, such as the display unit 18 of FIG. 2, a graphical representation of the staging plan onto real-time ultrasound images during the EBUS procedure and then displaying on the user interface a route for sampling the lymph node stations.

[0046] FIG. 3 is a perspective view of a transbronchial needle aspiration system including an ultrasound sensor that may be used with the techniques of this disclosure. The EBUS system 300 is an example of an Endobronchial Ultrasound Transbronchial Needle Aspiration system (EBUS-TBNA)) that may be used with the endoscopy system 10 of FIG.1 and FIG. 2. For example, the EBUS system 300 may form part of the functional section 30 of FIG. 2.

[0047] The EBUS system 300 includes an ultrasound probe 302 that is configured to generate data to be displayed as an ultrasound image and that is situated at the distal end of a specialized EBUS bronchoscope 306. A rigid needle 304 extends at an angle from an aperture 308. The needle 304 is sheathed prior to deployment by a sheath or catheter 310 that contains coils 312. The coils 312 may surround the needle 304 to reduce the likelihood of the needle 304 perforating a working channel of the EBUS bronchoscope 306.

[0048] The techniques of this disclosure are not limited for use with an EBUS bronchoscope. Other EBUS devices that do not include a camera may be used to implement the techniques of this disclosure.

[0049] As described in more detail below, in some examples, the ultrasound image from the ultrasound probe 302 and the EBUS device is provided to a machine-learning model trained to identify a lymph node in the ultrasound image. The machine-learning model generates an identification of a lymph node in the ultrasound image, and the ultrasound image overlaid with a label indicating the identified lymph node may be displayed on a user interface.

[0050] FIG. 4 depicts an example of an ultrasound image overlaid with a label indicating identified anatomical structures in accordance with this disclosure. An ultrasound image 400 is shown displayed, such as on the user interface 200 of FIG. 2. The ultrasound image 400 is overlaid with a label “LN” indicating a lymph node. In some examples, the label further indicates the specific lymph node, such as 4L in this particular example.

[0051] In addition, in some examples, the machine-learning model is further trained to generate and overlay a label indicating an identified anatomical structure. In some examples, the anatomical structures include one or more of one or more of blood vessels, arteries, veins, or other anatomical landmarks. For example, the ultrasound image 400 is overlaid with a label “PA” indicating the pulmonary artery and another label “Ao” indicating the ascending aorta.

[0052] FIG. 5 depicts another example of an ultrasound image overlaid with a label indicating identified anatomical structures in accordance with this disclosure. An ultrasound image 500 is shown displayed, such as on the user interface 200 of FIG. 2. The ultrasound image 500 is overlaid with a label “LN” indicating a lymph node. In some examples, the label further indicates the specific lymph node, such as 4R in this particular example.

[0053] In addition, in some examples, the machine-learning model is further trained to generate and overlay a label indicating an identified anatomical structure. In some examples, the anatomical structures include one or more of one or more of blood vessels, arteries, veins, or other anatomical landmarks. For example, the ultrasound image 500 is overlaid with a label “SVC” indicating the superior vena cava.

[0054] FIG. 6 depicts another example of an ultrasound image overlaid with a label indicating identified anatomical structures in accordance with this disclosure. An ultrasound image 600 is shown displayed, such as on the user interface 200 of FIG. 2. The ultrasound image 600 is overlaid with a label “LN” indicating a lymph node. In some examples, the label further indicates the specific lymph node, such as 10R in this particular example.

[0055] In addition, in some examples, the machine-learning model is further trained to generate and overlay a label indicating an identified anatomical structure. In some examples, the anatomical structures include one or more of one or more of blood vessels, arteries, veins, or other anatomical landmarks. For example, the ultrasound image 600 is overlaid with a label “SVC” indicating the superior vena cava, a label “PA” indicating the pulmonary artery, and another label “Az” indicating the azygos vein.

[0056] FIG. 7 shows a schematic diagram of an example of a computer-based lymph node identifier. The computer-based lymph node identifier 700 is configured for, among other things, identifying a lymph node from an ultrasound image from an EBUS device, such as the EBUS bronchoscope 306 of the EBUS system 300 in FIG. 3 or another EBUS device that does not include a camera.

[0057] In some examples, the computer-based lymph node identifier 700 may include an input interface 702 through which medical information, such as age, weight, sex, that arespecific to a patient may be provided as input features to a trained machine learning (ML) or artificial intelligence (Al) model, such as trained Al model 704. One or more relevant input features 710, which may be extracted from various ultrasound images 712, are applied to the Al model to generate an output predicted from Al model inference 706. For example, one or more relevant input features 710, such as the superior vena cava and the azygos vein, may be extracted from the ultrasound images generated by the EBUS system 300 of FIG. 3 and thenapplied to the trained Al model 704, and the Al model 704.

[0058] Then, the Al model 704 identifies and labels the lymph nodes in the ultrasound images. The computer-based lymph node identifier 700 may provide one or more confidence scores 714. The confidence scores 714 and the output predicted from Al model inference 706, e.g., lymph node labels, may be displayed on a user interface 716 and communicated to a user, e.g., a clinician.

[0059] In some embodiments, the input interface 702 may be a direct data link between the computer-based lymph node identifier 700 and one or more medical devices (e.g., the EBUS system 300), that generates at least some of the input features. Additionally, or alternatively, the input interface 702 may be a classical user interface that facilitates interaction between a user and the computer-based lymph node identifier 700. For example, the input interface 702 may facilitate a user interface through which the user may manually enter information.

[0060] Based on one or more of the input features, the output predicted from Al model inference 706 performs an inference operation using the Al model 704 to identify lymph nodes of an ultrasound image. The Al model 704 may provide a computer system the ability to perform tasks, without explicitly being programmed, by making inferences based on patterns found in the analysis of data. The Al model 704 explores the study and construction of algorithms (e.g., machine-learning algorithms) that may learn from existing data and make predictions about new data. Such algorithms operate by building an Al model from example training data in order to make data-driven predictions or decisions expressed as outputs or assessments.

[0061] There are two common modes for machine learning (ML): supervised ML and unsupervised ML. Supervised ML uses prior knowledge (e.g., examples that correlate inputs to outputs or outcomes) to learn the relationships between the inputs and the outputs. The goal of supervised ML is to learn a function that, given some training data, best approximates the relationship between the training inputs and outputs so that the ML model may implement the same relationships when given inputs to generate the corresponding outputs. Unsupervised ML is the training of an ML algorithm using information that isneither classified nor labeled, and allowing the algorithm to act on that information without guidance. Unsupervised ML is useful in exploratory analysis because it may automatically identify structure in data.

[0062] Typical tasks for supervised ML are classification problems and regression problems. Classification problems, also referred to as categorization problems, aim at classifying items into one of several category values (for example, is this object an apple or an orange?). Regression algorithms aim at quantifying some items (for example, by providing a score to the value of some input). Some examples of commonly used supervised-ML algorithms are Logistic Regression (LR), Naive-Bayes, Random Forest (RF), neural networks (NN), deep neural networks (DNN), matrix factorization, and Support Vector Machines (SVM).

[0063] Some everyday tasks for unsupervised ML include clustering, representation learning, and density estimation. Some examples of commonly used unsupervised-ML algorithms are K-means clustering, principal component analysis, and auto-encoders.

[0064] Another type of ML is federated learning (also known as collaborative learning) that trains an algorithm across multiple decentralized devices holding local data, without exchanging the data. This approach stands in contrast to traditional centralized machinelearning techniques where all the local datasets are uploaded to one server, as well as to more classical decentralized approaches which often assume that local data samples are identically distributed. Federated learning enables multiple actors to build a standard, robust machine-learning model without sharing data, thus allowing it to address critical issues such as data privacy, data security, data access rights and access to heterogeneous data.

[0065] In some examples, the Al model may be trained continuously or periodically prior to the performance of the inference operation by the output predicted from Al model inference 706. Then, during the inference operation, the patient-specific input features provided to the Al model may be propagated from an input layer, through one or more hidden layers, and ultimately to an output layer. By using these techniques, a processing unit, such as the image processing unit 42 of FIG. 2, may identify and label a lymph node of an ultrasound image.

[0066] FIG. 8 shows a schematic diagram of an example of a trained machine-learning model 800. An ultrasound device and various sensors may be used to generate training data. For example, a shape sensor 802, such as the shape sensor 204 of FIG. 2 is used to generate orientation data 804. A sensor, such as a magnetic sensor 806, may be used to generatepositioning data 808. An ultrasound imaging device 810, such as the EBUS system 300, may be used to generate ultrasound image data 812.

[0067] One or more of the orientation data 804, the positioning data 808, or the ultrasound image data 812 is used to generate corresponding N sets of ultrasound image training data 814, such as one or more of positioning data N 816, orientation data N 818, or ultrasound imaging data N 820.

[0068] One or more signal processing steps may be performed on the ultrasound image training data 814, such as sampling, feature extraction, filtering, and the like before the signal data is ready to be used as training data 822. The training data 822 may include N sets of training data based on the ultrasound image training data 814. In addition, the training data 822 may include annotation training data 824. For example, annotation training data 824 may include sets of contour data N 826 and labeling data N 828 generated by a medical practitioner 830. The contour data N 826 may include contours around anatomical structures in ultrasound images, such as one or more of blood vessels, arteries, veins, or other anatomical landmarks, and associated labels. The labeling data N 828 may include location labels, for example.

[0069] The training data 822 is used to train an Al or machine-learning model, such as the trained machine-learning model 800, e.g., the Al model 704 of FIG. 7. The training data 822 may be applied to a neural network structure 834, such as a DNN, comprising an input layer, one or more hidden layers, and an output layer. The training data 822, along with labeling data N 828, may be fed into the input layer of the neural network structure 834, which propagates the input data or data features through one or more hidden layers to the output layer that outputs weights and bias to form the trained machine-learning model 800. The trained machine-learning model 800 is able to perform tasks without explicitly being programmed by making inferences based on patterns found in the analysis of data.

[0070] FIG. 9 is a flow diagram of an example of a method 900 for labeling lymph nodes during an endobronchial ultrasound (EBUS) procedure. At block 902, the method 900 includes receiving the ultrasound image from the EBUS device and providing the ultrasound image from the EBUS device to a machine-learning model trained to identify a lymph node in the ultrasound image. For example, the image processing unit 42 of FIG. 2 receives one or more images from the EBUS bronchoscope 306 of FIG. 3 and provides the data representing the ultrasound image to the trained machine-learning model 800 of FIG. 8, which is trained to identify a lymph node in the ultrasound image. The processing unit is configured for providing the data to the machine-learning model.

[0071] At block 904, the method 900 includes receiving an identification of a lymph node in the ultrasound image from the machine-learning model. For example, the image processing unit 42 of FIG. 2 receives an identification of the lymph node in the ultrasound image from the trained machine-learning model 800.

[0072] At block 906, the method 900 includes displaying, on the user interface, the ultrasound image overlaid with a label indicating the identified lymph node. For example, the image processing unit 42 of FIG. 2 may display on the display unit 18 the ultrasound image overlaid with a label indicating the identified lymph node, such as shown in FIG. 4 to FIG. 6.

[0073] In some examples, the method 900 further includes receiving an identification of an anatomical structure in the ultrasound image from the machine-learning model, and displaying, on the user interface, the ultrasound image overlaid with a label indicating the identified anatomical structure, such as one or more of one or more of blood vessels, arteries, veins, or other anatomical landmarks. For example, the image processing unit 42 of FIG. 2 receives an identification of an anatomical structure in the ultrasound image from the trained machine-learning model 800 and displays on the display unit 18 the ultrasound image overlaid with a label indicating the identified anatomical structure, such as shown in FIG. 4 to FIG. 6.

[0074] In some examples, the image processing unit 42 may display on the display unit 18 the position and / or the orientation of the EBUS device, such as relative to a patient's airway. In some examples, the identification of the lymph node in the ultrasound image is based at least partially on the positioning data.

[0075] In addition to the techniques described above, this disclosure is also directed to techniques for providing real-time EBUS TBNA staging guidance to a clinician.Specifically, techniques are described that assist with generating a staging plan for a patient and then monitor EBUS-TBNA procedure progress to provide guidance to a clinician to adhere to the staging plan. The staging techniques of this disclosure are also appliable for use with Endoscopic Ultrasound (EUS) systems.

[0076] To successfully perform EBUS-TBNA procedures clinicians need to recognize anatomical vascular landmarks, master bronchoscopic orientation, and identify hilar and mediastinal lymph nodes using ultrasound. Mastery of these skills enables clinicians to successfully puncture lymph nodes with an EBUS-TBNA needle to obtain samples.

[0077] Lung cancer staging utilizing EBUS technology may be a long procedure with clinicians sampling 4-7 lymph node stations. Further, current EBUS practice guidelinesrecommend performing 3-4 needle passes for each lymph node station, leading to 16-28 needle passes per procedure. Although EBUS-TBNA remains an integral part of pulmonology Fellowship Training, the training methods, volumes, and outcomes have unfortunately been variable. This has led to a wide variation in practice patterns, diagnostic yield, and operator confidence. The American College of CHEST Physicians published guidelines on how many EBUS-TBNA procedures should be performed. However, there remains a need for systems and practices that reinforce adherence to guidelines to increase consistency between clinicians.

[0078] Described in detail below with respect to FIG. 11 through FIG. 14 is a system for providing Real-Time EBUS TBNA Staging Guidance to a clinician, such as for assisting with generating a staging plan for a patient and then monitoring an EBUS-TBNA procedure progress to provide guidance to a clinician to adhere to the staging plan. The staging plan may be generated based on a combination of a pre-procedural scan, such as a Computed Tomography (CT) scan or Positron Emission Tomography (PET) scan, or a combined CT- PET scan.

[0079] FIG. 10 illustrates a block diagram of an example machine 1000 upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform. Examples, as described herein, may include, or may operate by, logic or a number of components, or mechanisms in the machine 1000. Circuitry (e.g., processing circuitry) is a collection of circuits implemented in tangible entities of the machine 1000 that include hardware (e.g., simple circuits, gates, logic, etc.). Circuitry membership may be flexible over time. Circuitries have members that may, alone or in combination, perform specified operations when operating. In an example, hardware of the circuitry may be immutably designed to carry out a specific operation (e.g., hardwired). In an example, the hardware of the circuitry may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) including a machine readable medium physically modified (e.g., magnetically, electrically, moveable placement of invariant massed particles, etc.) to encode instructions of the specific operation. In connecting the physical components, the underlying electrical properties of a hardware constituent are changed, for example, from an insulator to a conductor or vice versa. The instructions enable embedded hardware (e.g., the execution units or a loading mechanism) to create members of the circuitry in hardware via the variable connections to carry out portions of the specific operation when in operation. Accordingly, in an example, the machine-readable medium elements are part of the circuitry or are communicatively coupled to the other components of the circuitry when the device is operating. In an example, any of the physical components may be used in more than onemember of more than one circuitry. For example, under operation, execution units may be used in a first circuit of a first circuitry at one point in time and reused by a second circuit in the first circuitry, or by a third circuit in a second circuitry at a different time. Additional examples of these components with respect to the machine 1000 follow.

[0080] In alternative examples, the machine 1000 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine 1000 may operate in the capacity of a server machine, a client machine, or both in server-client network environments. In an example, the machine 1000 may act as a peer machine in peer-to-peer (P2P) (or other distributed) network environment. The machine 1000 may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations.

[0081] The machine 1000 may include a hardware processing unit 1002 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processing unit core, or any combination thereof), a main memory 1004, a static memory (e.g., memory or storage for firmware, microcode, a basic-input-output (BIOS), and mass storage 1008 (e.g., hard drives, tape drives, flash storage, or other block devices) some or all of which may communicate with each other via an interlink 1030 (e.g., bus). The machine 1000 may further include a display unit 1010, an alphanumeric input device 1012 (e.g., a keyboard), and a user interface (UI) navigation device 1014 (e.g., a mouse). In an example, the display unit 1010, input device 1012 and UI navigation device 1014 may be a touch screen display. The machine 1000 may additionally include a signal generation device 1018 (e.g., a speaker), a network interface device 1020, and one or more sensors 1016, such as a global positioning system (GPS) sensor, compass, accelerometer, or other sensor. The machine 1000 may include an output controller 1028, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.).

[0082] Registers of the processing unit 1002, the main memory 1004, the static memory 1006, or the mass storage 1008 may be, or include, a machine-readable medium 1022 onwhich is stored one or more sets of data structures or instructions 1024 (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein. The instructions 1024 may also reside, completely or at least partially, within any of registers of the processing unit 1002, the main memory 1004, the static memory 1006, or the mass storage 1008 during execution thereof by the machine 1000. In an example, one or any combination of the hardware processing unit 1002, the main memory 1004, the static memory 1006, or the mass storage 1008 may constitute the machine-readable media 1022. While the machine readable medium 1022 is illustrated as a single medium, the term “machine readable medium” may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store the one or more instructions 1024.

[0083] The term “machine-readable medium” may include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine 1000 and that cause the machine 1000 to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Non-limiting machine-readable medium examples may include solid-state memories, optical media, magnetic media, and signals (e.g., radio frequency signals, other photon-based signals, sound signals, etc.). In an example, a non-transitory machine-readable medium comprises a machine-readable medium with a plurality of particles having invariant (e.g., rest) mass, and thus are compositions of matter. Accordingly, non-transitory machine- readable media are machine-readable media that do not include transitory propagating signals. Specific examples of non-transitory machine-readable media may include: nonvolatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0084] In an example, information stored or otherwise provided on the machine readable medium 1022 may be representative of the instructions 1024, such as instructions 1024 themselves or a format from which the instructions 1024 may be derived. This format from which the instructions 1024 may be derived may include source code, encoded instructions (e.g., in compressed or encrypted form), packaged instructions (e.g., split into multiple packages), or the like. The information representative of the instructions 1024 in the machine readable medium 1022 may be processed by processing circuitry into the instructions to implement any of the operations discussed herein. For example, deriving the instructions 1024 from the information (e.g., processing by the processing circuitry) mayinclude: compiling (e.g., from source code, object code, etc.), interpreting, loading, organizing (e.g., dynamically or statically linking), encoding, decoding, encrypting, unencrypting, packaging, unpackaging, or otherwise manipulating the information into the instructions 1024.

[0085] In an example, the derivation of the instructions 1024 may include assembly, compilation, or interpretation of the information (e.g., by the processing circuitry) to create the instructions 1024 from some intermediate or preprocessed format provided by the machine-readable medium 1022. The information, when provided in multiple parts, may be combined, unpacked, and modified to create the instructions 1024. For example, the information may be in various compressed source code packages (or object code, or binary executable code, etc.) on one or several remote servers. The source code packages may be encrypted when in transit over a network and decrypted, uncompressed, assembled (e.g., linked) if necessary, and compiled or interpreted (e.g., into a library, stand-alone executable etc.) at a local machine, and executed by the local machine.

[0086] The instructions 1024 may be further transmitted or received over a communications network 1026 using a transmission medium via the network interface device 1020 utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), LoRa / LoRaWAN, or satellite communication networks, mobile telephone networks (e.g., cellular networks such as those complying with 3G, 4G LTE / LTE-A, or 5G standards), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) 1002.11 family of standards known as Wi-Fi®, IEEE 1002.15.4 family of standards, peer-to-peer (P2P) networks, among others. In an example, the network interface device 1020 may include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas to connect to the communications network 1026. In an example, the network interface device 1020 may include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input singleoutput (MISO) techniques. The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by the machine 1000, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software. A transmission medium is a machine-readable medium.

[0087] As mentioned above, this disclosure also describes EBUS-TBNA staging techniques that identify sampling targets and generate staging plans based on scan data, such as pre-procedural and / or intra-procedural scan data. A graphical representation of the staging plan is overlaid onto real-time ultrasound images during an EBUS procedure and then a route is displayed for sampling the lymph node stations.

[0088] FIG. 11 is a graphical representation of an ultrasound image 1100 depicting the lymph node stations and bronchial tree anatomy in a patient along with a staging plan and recommended route in accordance with this disclosure. Lymph node stations include but are not limited to the following (some of which are depicted in FIG. 11):2R (Upper paratracheal lymph nodes on the right side);4R (Lower paratracheal lymph nodes on the right side);4L (Lower paratracheal lymph nodes on the left side);5 (Subaortic (aortic-pulmonary window) lymph nodes);7 (Subcarinal lymph nodes (central location));8 (Paraesophageal lymph nodes (below the carina));9 (Pulmonary ligament lymph nodes);10R (Hilar lymph nodes on the right side);10L (Hilar lymph nodes on the left side);HR (Intrapulmonary lymph nodes on the right side);HL (Intrapulmonary lymph nodes on the left side);12R (Lobar lymph nodes on the right side); and12L (Lobar lymph nodes on the left side).

[0089] Using various techniques of this disclosure, the endoscopy system 10 of FIG. 1, for example, is configured for providing real-time EBUS-TBNA staging guidance for a patient during an EBUS procedure. Using an EBUS device configured to capture an ultrasound image of a patient’s airway, such as the EBUS bronchoscope 306 of FIG. 3, the endoscopy system 10 identifies sampling targets, e.g., lymph node stations, and generates staging plans based on scan data, such as pre-procedural and / or intra-procedural scan data. A processing unit, such as the image processing unit 42 of FIG. 2, is configured for overlaying on a user interface, such as the display unit 18 of FIG. 2, a graphical representation of the staging plan onto real-time ultrasound images during the EBUS procedure and then displaying on the user interface a route for sampling the lymph node stations. A lymph node is a singleimmune system structure that filters lymph. A lymph node station is a group of lymph nodes in a specific region used in medical staging systems.

[0090] In some examples, the scan data indicating inflammation of the lymph node includes pre-procedural scan data such as positron emission tomography (PET) scan data indicating glucose uptake levels for the lymph node and computed tomography (CT) scan data of anatomical structures adjacent to the lymph node. The PET / CT imaging process may be used to help determine locations for sampling during the EBUS-TBNA procedure. In some examples, the PET / CT imaging procedure involves injecting the patient with a radiotracer called fluoro-deoxyglucose (FDG). A PET scan detects areas where this glucose is taken up by activated inflammatory cells, which may indicate inflammation or disease activity. CT scan data is then overlaid with the PET scan data to create a combined image. This overlay technique allows clinicians to identify both the presence of inflammation (shown as “lit up” areas in the PET data where glucose uptake occurs) and precisely locate these areas anatomically using the CT structural information. This combined PET / CT imaging approach helps determine locations for sampling during the EBUS-TBNA procedure.

[0091] The processing unit is configured for generating a “CT-PET score” for a lymph node, based on a combination of the CT scan data and the PET scan data, where the CT-PET score is indicative of disease activity. The CT-PET score is a scoring system used to interpret CT-PET scan results. It quantifies the amount of radioactive tracer uptake on the PET scan — typically measured as the Standardized Uptake Value (SUV) — and is used to evaluate the severity or extent of a disease, especially cancer, as seen on the combined PET and CT images. Generally, a higher score indicates increased disease activity. An example of a basic CT-PET scoring system is as follows: Score 0: Minimal tracer uptake; Score 1 : Moderate uptake; Score 2: High uptake; and Score 3: Very high uptake.

[0092] In response to the CT-PET score of the lymph node exceeding a threshold, the processing unit is configured for applying a rule set that determines a sequence of lymph node stations to be sampled based on a location of the lymph node. In some examples, the lymph node stations include at least three different lymph node stations. As mentioned above, the most common staging system for lung cancer is the TNM system, where T describes the primary tumor size and invasiveness, N indicates regional lymph node involvement, and M notes the presence or absence of distant metastases. In some examples,the rule set includes sampling N3 lymph node stations, then N2 lymph node stations, then N1 lymph node stations. In some examples, the rule set includes sampling lymph node stations on an opposite side from the lymph node before sampling lymph node stations on a same side as the lymph node.

[0093] As a non-limiting example for purposes of explanation only, based on the CT-PET score of the 4R lymph node exceeding a threshold, e.g., score 2 or higher, the endoscopy system 10 applies the rule set to generate a staging plan that includes sampling the following lymph node stations:4R: Lower paratracheal lymph nodes on the right side;7: Subcarinal lymph nodes (central location); and10L: Hilar lymph nodes on the left side.

[0094] Below is a more generic example of a rule set that the system, such as the endoscopy system 10 of FIG. 1, may use to generate a staging plan. Although the nonlimiting example below is specific to tumors in the lung, staging plans are applicable to other diseases. The rule set may include sampling any lymph node station with a visible lymph node or with a node greater than 5 mm in its short axis.

[0095] The system first identifies whether the primary tumor site is in the right lung or left lung. Based on the disease location, the system generates a staging plan that progresses through lymph node stations in a systematic order from N3 to N2 to N1 nodes, unless rapid on-site cytologic examination indicates malignant cells in a higher station lymph node.

[0096] For right-side disease locations, the system begins with left side N3 nodes (contralateral or opposite side nodes) including stations 2L, 4L, and station 7. The system then progresses to right side N2 nodes (ipsilateral or same side nodes) including stations 2R and 4R. Finally, the system completes sampling at right side N1 nodes (ipsilateral or same side nodes) including stations 10R, HR, and 12R.

[0097] For left-side disease locations, the system begins with right side N3 nodes (contralateral or opposite side nodes) including stations 2R, 4R, and station 7. The system then progresses to left side N2 nodes (ipsilateral or same side nodes) including stations 2L and 4L. Finally, the system completes sampling at left side N1 nodes (ipsilateral or same side nodes) including stations 10L, HL, and 12L.

[0098] The image processing unit 42 of the endoscopy system 10 may display on the display unit 18 an example of a route 1102 that a clinician may use to sample the lymph node stations 4R, 7, and 10L according to the staging plan is shown in FIG. 11. The lymph node stations may be depicted relative to the lung or bronchial tree. The image processingunit 42 may display aspects of the staging plan on the display unit 18 to the clinician alongside one or both of an ultrasound image and / or an endoscopic image.

[0099] In some examples, the image processing unit causes the display unit to annotate the route, e.g., the path for sampling. The route includes an indicator representing predetermined lymph nodes from which a sample needs to be taken during the procedure. For example, the image processing unit 42 causes the display unit 18 to annotate one or both of the ultrasound and / or endoscopic images with information to guide the clinician in adhering to the staging plan of FIG. 11.

[0100] For example, using various techniques described earlier in this disclosure, the endoscopy system 10 has generated a staging plan based on scan data indicating inflammation of a lymph node that includes the 4R, 7, and 10L lymph nodes. The staging plan 1104 may be depicted using indicators (e.g., circling, highlighting, and the like) the 4R, 7, and 10L lymph nodes for sampling on the graphical representation of the ultrasound image 1100. The processing unit causes the graphical representation of the ultrasound image 1100 displayed on the user interface to be overlaid with a graphical representation of the staging plan 1104 including lymph node stations indicated by the staging plan identified and displayed the 4R, 7, and 10L lymph nodes within the ultrasound image, like in FIG. 4 through FIG. 6. The processing unit causes the user interface to display a route 1102 for sampling the lymph node stations.

[0101] In some examples, the scan data includes intra-procedural scan data acquired by the EBUS device during the EBUS procedure. For example, the intra-procedural scan data includes an indication of at least one of a size of the lymph node, a shape of the lymph node, and a density of the lymph node. The size, shape, and / or density of the lymph node may be indicative of inflammation. The staging plan may then be updated based on the intra-procedural scan data. For example, based on a particular lymph node showing up in the intra-procedural scan data acquired by the EBUS device during the EBUS procedure as being inflamed (or “hot”), the system applies a rule set, or updates an existing rule set, to generate a patient-specific staging plan.

[0102] The staging plan may also be updated based on anatomical considerations, or clinical requirements for diagnosing various thoracic conditions including but not limited to cancer, tuberculosis, sarcoidosis, and lymphoma.

[0103] In some examples, the processing unit generates the staging plan at least partially using various techniques described above with respect to FIGS 4-9. For example, the processing unit identifies vascular anatomical structures in the ultrasound image, determineslocations of lymph nodes relative to the identified vascular anatomical structures, and generates the staging plan based on the determined lymph node locations.

[0104] FIG. 12 is a graphical representation of an example of a procedural progress 1200 of the staging plan and recommended route 1102 of FIG. 11 in accordance with this disclosure. Using the various techniques of this disclosure, the processing unit is configured for displaying real-time feedback on the user interface regarding a progress of the EBUS-TBNA procedure in relation to the staging plan, including indications of the lymph node stations successfully sampled and those pending sampling as per the staging plan.

[0105] After generating the staging plan, the endoscopy system 10 may monitor progress during the staging procedure and provide guidance or feedback to a clinician to assist them with adhering to the plan. For example, the endoscopy system 10 displays an alert 1202 that informs the clinician that the first lymph node target in the staging plan has been reached and that three samples should be obtained.

[0106] In some examples, the image processing unit 42 causes the display unit 18 to dynamically indicate the currently being viewed in the ultrasound image. In some examples, the display unit 18 highlights, such as in red, lymph nodes that have not yet been sampled, and then changes the highlighting to green when sufficient samples have been obtained. In some examples, the display unit 18 is displaying a lymph node that is not a sample target (PET -), the display unit 18 still highlights that lymph node for navigation assistance but also indicates to the clinician that no samples are needed per the recommended staging plan.

[0107] In some examples, during a procedure, the clinician is shown an image stream from the endoscope along with an image stream from an ultrasound device, like in FIG. 12. The display unit 18 may also display to a clinician the staging plan and a progress graphic. For example, the image processing unit 42 causes the display unit 18 to graphically depict the lymph nodes or lymph node stations that are fully sampled and the lymph nodes or lymph node stations from which samples still need to be taken.

[0108] In some examples, the image processing unit causes the display unit to display an indicator that represents the lymph node stations from which a sample has been taken. For example, responsive to an indication that a sample has been taken, the display unit 18 displays a “check mark” 1204 (or some other indictor) by the lymph node 4R to indicate that it is fully sampled and does not display check marks by lymph node 7 or lymph node 10L in the route 1102 of the staging plan to indicate that samples need to be taken fromthose lymph nodes, as shown in FIG. 12. In some examples, a clinician manually provides the indication to the system to indicate that the sample has been taken.

[0109] FIG. 13 is a graphical representation of an example of the lymph node stations and bronchial tree anatomy in a patient along with an orientation of the EBUS device in accordance with this disclosure. As described earlier in this disclosure, in some examples, the image processing unit 42 may display on the display unit 18 the position and / or the orientation of the EBUS device, such as relative to a patient’s airway. In some examples, the identification of the lymph node in the ultrasound image is based at least partially on the positioning data.

[0110] In some examples, the image processing unit causes the display unit to display a graphical representation of the EBUS device relative to the lymph node stations. For example, as seen in FIG. 13, an orientation of an EBUS device 1300 is graphically depicted relative to the lymph node stations and bronchial tree anatomy. In particular, the display unit 18 depicts the EBUS device 1300 adjacent the 4R lymph node. Depicting the orientation of the EBUS device 1300 relative to the lymph node stations and bronchial tree anatomy may assist the clinician in performing the staging plan.[OHl] Some advantages over existing systems include the following: (1) improved patient safety by clearly labeling the anatomical vascular structure, allowing the physician to sample cautiously if in proximity of the anatomic vascular structure; (2) reduced procedure time; (3) reduced procedure and technology learning curve; (4) increased accuracy of staging by ensuring the correct lymph nodes are sampled, resulting in more accurate treatment plans; and (5) reduced procedure time related to losing position and orientation during procedure.

[0112] FIG. 14 is a flow diagram of an example of a method 1400 for providing real-time EBUS-TBNA staging guidance during an EBUS procedure in accordance with this disclosure. At block 1402, the method 1400 includes capturing an ultrasound image of a patient’s airway. For example, using the EBUS bronchoscope 306 of FIG. 3, the endoscopy system 10 of FIG. 1 captures or acquires an ultrasound image of a patient’s airway, such as the ultrasound image 1100 of FIG. 11 that depicts the lymph node stations and bronchial tree anatomy of a patient.

[0113] At block 1404, the method 1400 includes generating a staging plan that is based on scan data indicating inflammation of a lymph node. For example, the imaging processing unit 42 of FIG. 2 generates a staging plan that is based on pre-procedural and / or intraprocedural scan data indicating inflammation of a lymph node.

[0114] At block 1406, the method 1400 includes overlaying a graphical representation of the ultrasound image with a graphical representation of the staging plan including lymph node stations indicated by the staging plan. For example, the imaging processing unit 42 of FIG. 2 causes a graphical representation of the ultrasound image 1100 of FIG. 11 to be overlaid with a graphical representation of the staging plan 1104 of FIG. 11, including lymph node stations indicated by the staging plan.

[0115] At block 1408, the method 1400 includes displaying a route for sampling the lymph node stations. For example, the imaging processing unit 42 of FIG. 2 causes a route 1102 in FIG. 11 to be displayed on the display unit 18 of FIG. 2, where the route 1102 is for sampling the lymph node stations.Various Notes

[0116] Each of the non-limiting claims or examples described herein may stand on its own, or may be combined in various permutations or combinations with one or more of the other examples.

[0117] The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are also referred to herein as “examples.” Such examples may include elements in addition to those shown or described. However, the present inventors also contemplate examples in which only those elements shown or described are provided. Moreover, the present inventors also contemplate examples using any combination or permutation of those elements shown or described (or one or more claims thereof), either with respect to a particular example (or one or more claims thereof), or with respect to other examples (or one or more claims thereof) shown or described herein.

[0118] In the event of inconsistent usages between this document and any documents so incorporated by reference, the usage in this document controls.

[0119] In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of “at least one” or “one or more.” In this document, the term “or” is used to refer to a nonexclusive or, such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. In this document, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” Also, in the following claims, the terms “including” and “comprising” are open-ended, that is, a system, device, article, composition, formulation, or process that includes elements inaddition to those listed after such a term in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to impose numerical requirements on their objects.

[0120] Method examples described herein may be machine or computer-implemented at least in part. Some examples may include a computer-readable medium or machine-readable medium encoded with instructions operable to configure an electronic device to perform methods as described in the above examples. An implementation of such methods may include code, such as microcode, assembly language code, a higher-level language code, or the like. Such code may include computer readable instructions for performing various methods. The code may form portions of computer program products. Further, in an example, the code may be tangibly stored on one or more volatile, non-transitory, or nonvolatile tangible computer-readable media, such as during execution or at other times. Examples of these tangible computer-readable media may include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact discs and digital video discs), magnetic cassettes, memory cards or sticks, random access memories (RAMs), read only memories (ROMs), and the like.

[0121] The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more claims thereof) may be used in combination with each other. Other embodiments may be used, such as by one of ordinary skill in the art upon reviewing the above description. The Abstract is provided to comply with 37 C.F.R. § 1.72(b), to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the above Detailed Description, various features may be grouped together to streamline the disclosure. This should not be interpreted as intending that an unclaimed disclosed feature is essential to any claim. Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description as examples or embodiments, with each claim standing on its own as a separate embodiment, and it is contemplated that such embodiments may be combined with each other in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.Various Notes

[0122] Each of the non-limiting claims or examples described herein may stand on its own, or may be combined in various permutations or combinations with one or more of the other examples.

[0123] The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are also referred to herein as “examples.” Such examples may include elements in addition to those shown or described. However, the present inventors also contemplate examples in which only those elements shown or described are provided. Moreover, the present inventors also contemplate examples using any combination or permutation of those elements shown or described (or one or more claims thereof), either with respect to a particular example (or one or more claims thereof), or with respect to other examples (or one or more claims thereof) shown or described herein.

[0124] In the event of inconsistent usages between this document and any documents so incorporated by reference, the usage in this document controls.

[0125] In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of “at least one” or “one or more.” In this document, the term “or” is used to refer to a nonexclusive or, such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. In this document, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” Also, in the following claims, the terms “including” and “comprising” are open-ended, that is, a system, device, article, composition, formulation, or process that includes elements in addition to those listed after such a term in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to impose numerical requirements on their objects.

[0126] Method examples described herein may be machine or computer-implemented at least in part. Some examples may include a computer-readable medium or machine-readable medium encoded with instructions operable to configure an electronic device to perform methods as described in the above examples. An implementation of such methods may include code, such as microcode, assembly language code, a higher-level language code, or the like. Such code may include computer readable instructions for performing various methods. The code may form portions of computer program products. Further, in anexample, the code may be tangibly stored on one or more volatile, non-transitory, or nonvolatile tangible computer-readable media, such as during execution or at other times. Examples of these tangible computer-readable media may include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact discs and digital video discs), magnetic cassettes, memory cards or sticks, random access memories (RAMs), read only memories (ROMs), and the like.

[0127] The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more claims thereof) may be used in combination with each other. Other embodiments may be used, such as by one of ordinary skill in the art upon reviewing the above description. The Abstract is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the above Detailed Description, various features may be grouped together to streamline the disclosure. This should not be interpreted as intending that an unclaimed disclosed feature is essential to any claim. Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description as examples or embodiments, with each claim standing on its own as a separate embodiment, and it is contemplated that such embodiments may be combined with each other in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

CLAIMSWhat is claimed is:

1. A system for providing real-time EBUS-TBNA staging guidance during an endobronchial ultrasound (EBUS) procedure, the system comprising: an EBUS device configured to capture an ultrasound image of a patient's airway; a user interface including a display; and a processing unit configured for: generating a staging plan that is based on scan data indicating inflammation of a lymph node; overlaying, on the user interface, a graphical representation of the ultrasound image with a graphical representation of the staging plan including lymph node stations indicated by the staging plan; and displaying, on the user interface, a route for sampling the lymph node stations.

2. The system of claim 1, wherein the processing unit is configured for displaying realtime feedback on the user interface regarding a progress of the EBUS procedure in relation to the staging plan, including indications of the lymph node stations successfully sampled and those pending sampling as per the staging plan.

3. The system of claim 1, wherein the scan data indicating inflammation of the lymph node includes pre-procedural scan data including:PET scan data indicating glucose uptake levels for the lymph node; and CT scan data of anatomical structures adjacent to the lymph node, wherein the processing unit is further configured for: generating, for the lymph node, a CT-PET score based on a combination of the CT scan data and the PET scan data, wherein the CT-PET score is indicative of disease activity.

4. The system of claim 3, wherein, in response to the CT-PET score of the lymph node exceeding a threshold, the processing unit configured for generating the staging plan is further configured for: applying a rule set that determines a sequence of lymph node stations to be sampled based on a location of the lymph node.

5. The system of claim 1, wherein the scan data includes intra-procedural scan data acquired by the EBUS device during the EBUS procedure, and wherein the processing unit is further configured for: updating the staging plan based on the intra-procedural scan data.

6. The system of claim 5, wherein the intra-procedural scan data includes an indication of at least one of a size of the lymph node, a shape of the lymph node, and a density of a lymph node.

7. The system of claim 4, wherein the lymph node stations include at least three different lymph node stations.

8. The system of claim 4, wherein the rule set includes sampling lymph node stations on an opposite side from the lymph node before sampling lymph node stations on a same side as the lymph node.

9. The system of claim 8, wherein the rule set includes sampling N3 lymph node stations, then N2 lymph node stations, then N1 lymph node stations.

10. The system of claim 1, wherein generating the staging plan comprises: identifying vascular anatomical structures in the ultrasound image; determining locations of lymph nodes relative to the identified vascular anatomical structures; and generating the staging plan based on the determined lymph node locations.

11. The system of claim 1, wherein the processing unit is configured for: annotating, on the user interface, the route, wherein the route includes a first indicator representing pre-determined lymph nodes from which a sample needs to be taken during the procedure.

12. The system of claim 11, wherein the processing unit configured for annotating the route is configured for: responsive to an indication that a sample has been taken, displaying, on the user interface, a second indicator that represents the lymph node stations from which the sample has been taken.

13. The system of claim 11, wherein the processing unit is configured for:displaying, on the user interface, a graphical representation of the EBUS device relative to the lymph node stations.

14. A method for providing real-time EBUS-TBNA staging guidance during an endobronchial ultrasound (EBUS) procedure, the method comprising: capturing an ultrasound image of a patient's airway; generating a staging plan that is based on scan data indicating inflammation of a lymph node; overlaying a graphical representation of the ultrasound image with a graphical representation of the staging plan including lymph node stations indicated by the staging plan; and displaying a route for sampling the lymph node stations.

15. The method of claim 14, further comprising: applying a rule set that determines a sequence of lymph node stations to be sampled based on a location of the lymph node.

16. The method of claim 14, further comprising: annotating the route, wherein the route includes a first indicator representing predetermined lymph nodes from which a sample needs to be taken during the EBUS procedure; and responsive to an indication that a sample has been taken, displaying a second indicator that represents the lymph node stations from which the sample has been taken.

17. The method of claim 14, wherein the scan data indicating inflammation of the lymph node includes pre-procedural scan data, the method further comprising: generating, for the lymph node, a CT-PET score based on a combination of CT scan data and PET scan data, wherein the CT-PET score is indicative of disease activity.

18. The method of claim 14, wherein the scan data includes intra-procedural scan data acquired during the EBUS procedure, the method further comprising: updating the staging plan based on the intra-procedural scan data.

19. A system for providing real-time EBUS-TBNA staging guidance during an endobronchial ultrasound (EBUS) procedure, the system comprising: an EBUS device configured to capture an ultrasound image of a patient's airway; a user interface including a display; and a processing unit configured for: generating a staging plan that is based on scan data indicating inflammation of a lymph node, wherein the scan data includes pre-procedural scan data including PET scan data and CT scan data, wherein the staging plan is based on a CT-PET score using a combination of the CT scan data and the PET scan data, and wherein the CT-PET score is indicative of disease activity; overlaying, on the user interface, a graphical representation of the ultrasound image with a graphical representation of the staging plan including lymph node stations indicated by the staging plan; and displaying, on the user interface, a route for sampling the lymph node stations.

20. The system of claim 19, wherein, in response to the CT-PET score of the lymph node exceeding a threshold, the processing unit configured for generating the staging plan is further configured for: applying a rule set that determines a sequence of lymph node stations to be sampled based on a location of the lymph node.

Images