Ultrasound anatomical vascular structure imaging techniques

Abstract

Techniques are described to utilize machine learning to analyze EBUS ultrasound imagery. Models are trained on annotated samples to identify important vascular landmarks, lymph node stations, and other structures.

Description

ULTRASOUND ANATOMICAL VASCULAR STRUCTURE IMAGINGTECHNIQUESPRIORITY CLAIM

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 553,732, filed February 15, 2024, and U.S. Provisional Patent Application Serial No. 63 / 571,571, filed March 29, 2024, the contents of which are hereby incorporated by reference.FIELD OF THE DISCLOSURE

[0002] This document pertains generally, but not by way of limitation, to overlay of medical image data.BACKGROUND

[0003] Endobronchial Ultrasound-Guided Transbronchial Needle Aspiration (EBUS- TBNA) is a minimally invasive medical procedure that enhances the capabilities of standard Endobronchial Ultrasound (EBUS) for diagnosing lung diseases, including lung cancer, infections, and other thoracic conditions. EBUS-TBNA combines the visual advantages of bronchoscopy with the imaging precision of ultrasound, along with the added capability of needle aspiration. This integration allows for the direct sampling of tissue from lung lesions, lymph nodes, and other structures adjacent to the airways.

[0004] EBUS-TBNA employs a specialized bronchoscope with an ultrasound transducer at its tip and a channel for needle aspiration. When the bronchoscope is navigated to the target area within the lungs, the ultrasound component generates detailed images of lung tissues, airway walls, and nearby structures, facilitating precise needle aspiration of these areas.This technique is particularly beneficial for sampling lymph nodes or masses adjacent to the airways but outside the reach of traditional bronchoscopic biopsy methods.

[0005] One of the significant advantages of EBUS-TBNA is its ability to perform real-time image-guided needle biopsies. This capability is crucial for accurately sampling tissue for diagnostic purposes, reducing the need for more invasive procedures like mediastinoscopy or thoracotomy. EBUS-TBNA significantly enhances the precision and safety of lung biopsies, leading to improved diagnosis and staging of lung cancer and other diseases.

[0006] EBUS-TBNA is particularly valuable in diagnosing conditions like tuberculosis, sarcoidosis, and lymphoma. It plays a pivotal role in the staging of lung cancer,determining the extent of the disease and guiding the treatment plan. This procedure is typically well-tolerated by patients and can be performed under moderate sedation or general anesthesia. With its low complication rate and high diagnostic yield, EBUS-TBNA represents a significant advancement in the field of pulmonary medicine, offering a less invasive, yet highly effective alternative to traditional diagnostic methods for thoracic diseases.SUMMARY OF THE DISCLOSURE

[0007] This disclosure describes techniques to utilize machine learning to analyze EBUS ultrasound imagery. Models are trained on annotated samples to identify important vascular landmarks, lymph node stations, and other structures. The techniques of this disclosure have a number of advantages over existing systems, such as the following: (1) improving patient safety by clearly labeling the anatomical vascular structure, allowing the physician to sample cautiously if in proximity of the anatomic vascular structure; (2) reducing procedure time; (3) reducing procedure and technology learning curve; (4) increasing accuracy of staging by ensuring the correct lymph nodes are sampled, resulting in more accurate treatment plans; and (5) reducing procedure time related to losing position and orientation during procedure.

[0008] In some aspects, this disclosure is directed to a system for labeling lymph nodes during an endobronchial ultrasound (EBUS) procedure, the system comprising: an EBUS device configured to capture an ultrasound image of a patient's airway and surrounding structures; a user interface including a display; and a processing unit and a non-transitory computer-readable medium storing instructions that, when executed, cause the processing unit to perform operations including: receiving the ultrasound image from the EBUS device and providing the ultrasound image from the EBUS device to a machine-learning model trained to identify a lymph node in the ultrasound image; receiving an identification of the lymph node in the ultrasound image from the machine-learning model; and displaying, on the user interface, the ultrasound image overlaid with a label indicating the identified lymph node.

[0009] In some aspects, this disclosure is directed to a method for labeling lymph nodes during an endobronchial ultrasound (EBUS) procedure, the method comprising: receiving an ultrasound image from an EBUS device and providing the ultrasound image from the EBUS device to a machine-learning model trained to identify a lymph node in the ultrasound image; receiving an identification of the lymph node in the ultrasound imagefrom the machine-learning model; and displaying, on a user interface, the ultrasound image overlaid with a label indicating the identified lymph node.

[0010] In some aspects, this disclosure is directed to a system for labeling lymph nodes during an endobronchial ultrasound (EBUS) procedure, the system comprising: an EBUS device configured to capture an ultrasound image of a patient's airway and surrounding structures; a sensor configured for detecting at least one of a position or an orientation of the EBUS device relative to the patient's airway and providing at least one of positioning data or orientation data to a processing unit; a user interface including a display; and the processing unit and a non-transitory computer-readable medium storing instructions that, when executed, cause the processing unit to perform operations including: receiving the ultrasound image from the EBUS device and providing the ultrasound image from the EBUS device to a machine-learning model trained to identify a lymph node in the ultrasound image; receiving an identification of the lymph node in the ultrasound image from the machine-learning model; and displaying, on the user interface, the ultrasound image overlaid with a label indicating the identified lymph node and data representing at least one of the position or the orientation of the EBUS deviceBRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views. Like numerals having different letter suffixes may represent different instances of similar components. The drawings illustrate generally, by way of example, but not by way of limitation, various embodiments discussed in the present document.

[0012] FIG. 1 is a schematic diagram of an example of an endoscopy system including an imaging and control system and an endoscope that may implement various techniques of this disclosure.

[0013] FIG. 2 is a schematic diagram of the endoscopy system of FIG. 1 including the endoscope connected to a control unit of the imaging and control system.

[0014] FIG. 3 is a perspective view of a transbronchial needle aspiration system including an ultrasound sensor that may used with the techniques of this disclosure.

[0015] FIG. 4 depicts an example of an ultrasound image overlaid with a label indicating identified anatomical structures in accordance with this disclosure.

[0016] FIG. 5 depicts another example of an ultrasound image overlaid with a label indicating identified anatomical structures in accordance with this disclosure.

[0017] FIG. 6 depicts another example of an ultrasound image overlaid with a label indicating identified anatomical structures in accordance with this disclosure.

[0018] FIG. 7 shows a schematic diagram of an example of a computer-based lymph node identifier.

[0019] FIG. 8 shows a schematic diagram of an example of a trained machine-learning model.

[0020] FIG. 9 illustrates an aspect of the subject matter in accordance with one embodiment.

[0021] FIG. 10 is a block diagram illustrating an example of a machine upon which one or more examples may be implemented.DETAILED DESCRIPTION

[0022] Accurately staging lung cancer is critical for determining the optimal treatment path for patients. Staging refers to determining how far a lung cancer has spread in the body. It is a critical step after diagnosis as it dictates the optimal treatment plan and prognosis for patients. Staging involves evaluating the size and locations of the primary tumor, assessing if cancer cells have spread to nearby lymph nodes, and checking for metastases in more distant organs. The most common staging system for lung cancer is the TNM system, where T describes the primary tumor size and invasiveness, N indicates regional lymph node involvement, and M notes the presence or absence of distant metastases.

[0023] By assigning numeric values and letters to each category, an overall stage can be assigned, usually ranging from Stage I (early, localized cancer) to Stage IV (advanced, metastatic cancer). Accurately staging via procedures like EBUS allows selection of personalized treatment options for the best outcome.

[0024] Currently, endobronchial ultrasound (EBUS) procedures are the standard of care for staging non-small cell lung cancer by sampling lymph nodes to assess disease spread. However, these procedures can be long and complex, requiring sampling multiple lymph node stations across different regions of the lung. In addition, existing EBUS systems allow for imaging of anatomical vascular structures, but do not confirm or label those structures or lymph nodes. The present inventors have recognized that due to the intricate lung anatomy,variation across patients, and physician skillset, some physicians may lose orientation during EBUS procedures, such as thinking they are at one lymph node station when they are at another lymph node station, leading to inaccurate lymph node identification and staging.

[0025] To address this issue and other issues, the present inventors have recognized the desirability of providing real-time labeling and identification of key anatomical structures during EBUS procedures. This disclosure describes techniques to utilize machine learning to analyze EBUS ultrasound imagery. Models are trained on annotated samples to identify important vascular landmarks, lymph node stations, and other structures. The techniques of this disclosure have a number of advantages over existing systems, such as the following: (1) improving patient safety by clearly labeling the anatomical vascular structure, allowing the physician to sample cautiously if in proximity of the anatomic vascular structure; (2) reducing procedure time; (3) reducing procedure and technology learning curve; (4) increasing accuracy of staging by ensuring the correct lymph nodes are sampled, resulting in more accurate treatment plans; and (5) reducing procedure time related to losing position and orientation during procedure.

[0026] FIG. 1 is a schematic diagram of an example of an endoscopy system 10 including an imaging and control system 12 and an endoscope 14 that may implement various techniques of this disclosure. The endoscopy system 10 is suitable for use with the systems, devices, and methods described below, such as modular endoscopy systems, modular endoscopes, and methods for designing, building, and deconstructing endoscopes. According to some examples, the endoscope 14 may be insertable into an anatomical region for imaging and / or to provide passage of one or more sampling devices for biopsies, or one or more therapeutic devices for treatment of a disease state associated with the anatomical region. Endoscope 14 may, in advantageous aspects, interface with and connect to imaging and control system 12. In the example shown, the endoscope 14 includes a duodenoscope, though other types of endoscopes may be used with the features and teachings of the present disclosure.

[0027] The imaging and control system 12 may include a controller 16, a user interface including an output unit 18 (e.g., display) and an input unit 20, a light source 22, a fluid source 24, and a suction pump 26. The imaging and control system 12 may include various ports for coupling with the endoscopy system 10. For example, the controller 16 may include a data input / output port for receiving data from and communicating data to the endoscope 14.

[0028] The light source 22 may include an output port for transmitting light to the endoscope 14, such as via a fiber optic link. The fluid source 24 may include a port for transmitting fluid to the endoscope 14. The fluid source 24 may comprise a pump and a tank of fluid or may be connected to an external tank, vessel, or storage unit. The suction pump 26 may include a port used to draw a vacuum from the endoscope 14 to generate suction, such as for withdrawing fluid from the anatomical region into which the endoscope 14 is inserted. The output unit 18 and the input unit 20 may be used by an operator of the endoscopy system 10 to control functions of the endoscopy system 10 and view the output of the endoscope 14.

[0029] The controller 16 may additionally be used to generate signals or other outputs from treating the anatomical region into which the endoscope 14 is inserted. In some examples, the controller 16 may generate electrical output, acoustic output, a fluid output, and the like for treating the anatomical region with, for example, cauterizing, cutting, freezing, and the like.

[0030] The endoscope 14 may include an insertion section 28, a functional section 30, and a handle section 32, which may be coupled to a cable section 34 and a coupler section 36. The insertion section 28 may extend distally from the handle section 32 and the cable section 34 may extend proximally from the handle section 32. The insertion section 28 may be elongated and include a bending section, and a distal end to which functional section 30 may be attached. The bending section may be controllable (e.g., by a control knob 38 on a handle section 32) to maneuver the distal end through tortuous anatomical passageways (e.g., stomach, duodenum, kidney, ureter, etc.) The insertion section 28 may also include one or more working channels (e.g., an internal lumen) that may be elongated and support the insertion of one or more therapeutic tools of the functional section 30. The working channel may extend between the handle section 32 and the functional section 30. Additional functionalities, such as fluid passages, guide wires, and pull wires may also be provided by the insertion section 28 (e.g., via suction or irrigation passageways, and the like).

[0031] The handle module 32 may include a knob 38 as well as ports 40. The knob 38 may be coupled to a pull wire extending through the insertion section 28. The ports 40 may be configured to couple various electrical cables, fluid tubes, and the like to the handle module 32 for coupling with the insertion section 28.

[0032] The imaging and control system 12, according to some examples, may be provided on a mobile platform (e.g., a cart 41) with shelves for housing the light source 22, the suction pump 26, the image processing unit 42, etc. Alternatively, several components ofthe imaging and control system 12 that are shown in FIGS. 1 and 2 may be provided directly on the endoscope 14 to make the endoscope “self-contained.”

[0033] FIG. 2 is a schematic diagram of the endoscopy system 10 of FIG. 1 including the imaging and control system 12 and endoscope 14. FIG. 2 schematically illustrates components of the imaging and control system 12 coupled to the endoscope 14, which in the illustrated example includes a duodenoscope. The imaging and control system 12 may include the controller 16, which may include or be coupled to the image processing unit 42, the treatment generator 44 and the drive unit 46, as well as the light source 22, the input unit 20, and the output unit 18. The output unit 18, e.g., a display unit, and the input unit 20 form part of a user interface 200. The image processing unit 42 includes one or more processors that may be distributed, such as locally and remotely, or co-located at one location.

[0034] The image processing unit 42 and the light source 22 may each interface with the endoscope 14 by wired or wireless electrical connections. The imaging and control system 12 may accordingly illuminate an anatomical region, collect signals representing the anatomical region, process signals representing the anatomical region, and display images representing the anatomical region on the display unit 18. The imaging and control system 12 may include the light source 22 to illuminate the anatomical region using light of desired spectrum (e.g., broadband white light, narrow-band imaging using preferred electromagnetic wavelengths, and the like). The imaging and control system 12 may connect (e.g., via an endoscope connector) to the endoscope 14 for signal transmission (e.g., light output from light source, video signals from imaging system in the distal end, and the like).

[0035] The fluid source 24 may include one or more sources of air, saline or other fluids, as well as associated fluid pathways (e.g., air channels, irrigation channels, suction channels) and connectors (barb fittings, fluid seals, valves and the like). The imaging and control system 12 may also include the drive unit 46, which may be an optional component. The drive unit 46 may include a motorized drive for advancing a distal section of endoscope 14, as described in PCT Pub. No. WO 2011 / 140118 Al to Frassica et al., titled “Rotate-to- Advance Catheterization System,” which is hereby incorporated in its entirety by reference.

[0036] One or more sensors are in electrical communication with the controller 16. For example, the endoscopy system 10 may include one or both of a sensor 202 and a shape sensor 204. The sensor 202 may be positioned adjacent a distal tip of an EBUS device, such as the EBUS bronchoscope 306 of FIG. 3, and configured for detecting a position of the EBUS device relative to the patient's airway and providing positioning data to theprocessing unit, such as the image processing unit 42. The image processing unit 42 may then cause the user interface 200 to display data representing the position of the EBUS device. In some examples, the sensor 202 is a magnetic sensor.

[0037] Alternatively or additionally, the endoscopy system 10 may also include a shape sensor 204. Shape sensors are well-known and will not be described in detail. An example of a shape sensor is described in U.S. Patent No. 10,823,627, titled “Shape sensing with multi-core fiber sensor” to Sanborn et al., the entire contents of which being incorporated herein by reference. The shape sensor 204 may be integrated into a portion of the EBUS bronchoscope 306 of FIG. 3, or some other portion of the insertion section 28 of the endoscopy system 10. In some examples, the shape sensor 204 is configured for detecting an orientation of the EBUS bronchoscope relative to the patient's airway and providing orientation data to the processing unit, such as the image processing unit 42. The image processing unit 42 may then cause the user interface 200 to display data representing the orientation of the EBUS bronchoscope. In some examples, the shape sensor 204 includes a plurality of fiber Bragg gratings.

[0038] FIG. 3 is a perspective view of a transbronchial needle aspiration system including an ultrasound sensor that may be used with the techniques of this disclosure. The EBUS system 300 is an example of an Endobronchial Ultrasound Transbronchial Needle Aspiration system (EBUS-TBNA)) that may be used with the endoscopy system 10 of FIG. 1 and FIG. 2. For example, the EBUS system 300 may form part of the functional section 30 of FIG. 2.

[0039] The EBUS system 300 includes an ultrasound probe 302 that is configured to generate data to be displayed as an ultrasound image and that is situated at the distal end of a specialized EBUS bronchoscope 306. A rigid needle 304 extends at an angle from an aperture 308. The needle 304 is sheathed prior to deployment by a sheath or catheter 310 that contains coils 312. The coils 312 may surround the needle 304 to reduce the likelihood of the needle 304 perforating a working channel of the EBUS bronchoscope 306. In some examples, the EBUS system 300 includes a camera 314.

[0040] The techniques of this disclosure are not limited for use with an EBUS bronchoscope. Other EBUS devices that do not include a camera may be used to implement the techniques of this disclosure.

[0041] As described in more detail below, the ultrasound image from the ultrasound probe 302 and the EBUS device are provided to a machine-learning model trained to identify a lymph node in the ultrasound image. The machine-learning model generates anidentification of a lymph node in the ultrasound image, and the ultrasound image overlaid with a label indicating the identified lymph node may be displayed on a user interface.

[0042] FIG. 4 depicts an example of an ultrasound image overlaid with a label indicating identified anatomical structures in accordance with this disclosure. An ultrasound image 400 is shown displayed, such as on the user interface 200 of FIG. 2. The ultrasound image 400 is overlaid with a label “LN” indicating a lymph node. In some examples, the label further indicates the specific lymph node, such as 4L in this particular example.

[0043] In addition, in some examples, the machine-learning model is further trained to generate and overlay a label indicating an identified anatomical structure. In some examples, the anatomical structures include one or more of one or more of blood vessels, arteries, veins, or other anatomical landmarks. For example, the ultrasound image 400 is overlaid with a label “PA” indicating the pulmonary artery and another label “Ao” indicating the ascending aorta.

[0044] FIG. 5 depicts another example of an ultrasound image overlaid with a label indicating identified anatomical structures in accordance with this disclosure. An ultrasound image 500 is shown displayed, such as on the user interface 200 of FIG. 2. The ultrasound image 500 s overlaid with a label “LN” indicating a lymph node. In some examples, the label further indicates the specific lymph node, such as 4R in this particular example.

[0045] In addition, in some examples, the machine-learning model is further trained to generate and overlay a label indicating an identified anatomical structure. In some examples, the anatomical structures include one or more of one or more of blood vessels, arteries, veins, or other anatomical landmarks. For example, the ultrasound image 500 is overlaid with a label “SVC” indicating the superior vena cava.

[0046] FIG. 6 depicts another example of an ultrasound image overlaid with a label indicating identified anatomical structures in accordance with this disclosure. An ultrasound image 600 is shown displayed, such as on the user interface 200 of FIG. 2. The ultrasound image 600 is overlaid with a label “LN” indicating a lymph node. In some examples, the label further indicates the specific lymph node, such as 10R in this particular example.

[0047] In addition, in some examples, the machine-learning model is further trained to generate and overlay a label indicating an identified anatomical structure. In some examples, the anatomical structures include one or more of one or more of blood vessels, arteries, veins, or other anatomical landmarks. For example, the ultrasound image 600 isoverlaid with a label “SVC” indicating the superior vena cava, a label “PA” indicating the pulmonary artery, and another label “Az” indicating the azygos vein.

[0048] FIG. 7 shows a schematic diagram of an example of a computer-based lymph node identifier. The computer-based lymph node identifier 700 is configured for, among other things, identifying a lymph node from an ultrasound image from an EBUS device, such as the EBUS bronchoscope 306 of the EBUS system 300 in FIG. 3 or another EBUS device that does not include a camera.In some examples, the computer-based lymph node identifier 700 may include an input interface 702 through which medical information, such as age, weight, sex, that are specific to a patient may be provided as input features to a trained machine learning (ML) or artificial intelligence (Al) model, such as trained Al model 704. One or more relevant input features 710, which may be extracted from various ultrasound images 712, are applied to the Al model to generate an output predicted from Al model inference 706. For example, one or more relevant input features 710, which may be extracted from the ultrasound images generated by the EBUS system 300 of FIG. 3, may be applied to the trained Al model 704, and the Al model 704. In an example, an ultrasound image is applied to a separate Al model, which may be used to identify one or more anatomical structures in the image, such as the superior vena cava, the pulmonary artery, the azygos vein, etc. The anatomical structure(s) identified by the Al model are examples of relevant input features 710 that are applied to the input interface 702 and used as input features, such as the 1stinput feature.

[0049] Then, the Al model 704 identifies and labels the lymph nodes in the ultrasound images. The computer-based lymph node identifier 700 may provide one or more confidence scores 714. The confidence scores 714 and the output predicted from Al model inference 706, e.g., lymph node labels, may be displayed on a user interface 716 and communicated to a user, e.g., a clinician.

[0050] In some embodiments, the input interface 702 may be a direct data link between the computer-based lymph node identifier 700 and one or more medical devices (e.g., the EBUS system 300), that generates at least some of the input features. Additionally, or alternatively, the input interface 702 may be a classical user interface that facilitates interaction between a user and the computer-based lymph node identifier 700. For example, the input interface 702 may facilitate a user interface through which the user may manually enter information.

[0051] Based on one or more of the input features, the output predicted from Al model inference 706 performs an inference operation using the Al model 704 to identify lymphnodes of an ultrasound image. The Al model 704 may provide a computer system the ability to perform tasks, without explicitly being programmed, by making inferences based on patterns found in the analysis of data. The Al model 704 explores the study and construction of algorithms (e.g., machine-learning algorithms) that may learn from existing data and make predictions about new data. Such algorithms operate by building an Al model from example training data in order to make data-driven predictions or decisions expressed as outputs or assessments.

[0052] There are two common modes for machine learning (ML): supervised ML and unsupervised ML. Supervised ML uses prior knowledge (e.g., examples that correlate inputs to outputs or outcomes) to learn the relationships between the inputs and the outputs. The goal of supervised ML is to learn a function that, given some training data, best approximates the relationship between the training inputs and outputs so that the ML model may implement the same relationships when given inputs to generate the corresponding outputs. Unsupervised ML is the training of an ML algorithm using information that is neither classified nor labeled, and allowing the algorithm to act on that information without guidance. Unsupervised ML is useful in exploratory analysis because it may automatically identify structure in data.

[0053] Typical tasks for supervised ML are classification problems and regression problems. Classification problems, also referred to as categorization problems, aim at classifying items into one of several category values (for example, is this object an apple or an orange?). Regression algorithms aim at quantifying some items (for example, by providing a score to the value of some input). Some examples of commonly used supervised-ML algorithms are Logistic Regression (LR), Naive-Bayes, Random Forest (RF), neural networks (NN), deep neural networks (DNN), matrix factorization, and Support Vector Machines (SVM).

[0054] Some everyday tasks for unsupervised ML include clustering, representation learning, and density estimation. Some examples of commonly used unsupervised-ML algorithms are K-means clustering, principal component analysis, and auto-encoders.

[0055] Another type of ML is federated learning (also known as collaborative learning) that trains an algorithm across multiple decentralized devices holding local data, without exchanging the data. This approach stands in contrast to traditional centralized machinelearning techniques where all the local datasets are uploaded to one server, as well as to more classical decentralized approaches which often assume that local data samples are identically distributed. Federated learning enables multiple actors to build a standard, robustmachine-learning model without sharing data, thus allowing it to address critical issues such as data privacy, data security, data access rights and access to heterogeneous data.

[0056] In some examples, the Al model may be trained continuously or periodically prior to the performance of the inference operation by the output predicted from Al model inference 706. Then, during the inference operation, the patient-specific input features provided to the Al model may be propagated from an input layer, through one or more hidden layers, and ultimately to an output layer. By using these techniques, a processing unit, such as the image processing unit 42 of FIG. 2, may identify and label a lymph node of an ultrasound image.

[0057] FIG. 8 shows a schematic diagram of an example of a trained machine-learning model 800. An ultrasound device and various sensors may be used to generate training data. For example, a shape sensor 802, such as the shape sensor 204 of FIG. 2 is used to generate orientation data 804. A sensor, such as a magnetic sensor 806, may be used to generate positioning data 808. An ultrasound imaging device 810, such as the EBUS system 300, may be used to generate ultrasound image data 812.

[0058] One or more of the orientation data 804, the positioning data 808, or the ultrasound image data 812 is used to generate corresponding N sets of ultrasound image training data 814, such as one or more of positioning data N 816, orientation data N 818, or ultrasound imaging data N 820.

[0059] One or more signal processing steps may be performed on the ultrasound image training data 814, such as sampling, feature extraction, filtering, and the like before the signal data is ready to be used as training data 822. The training data 822 may include N sets of training data based on the ultrasound image training data 814. In addition, the training data 822 may include annotation training data 824. For example, annotation training data 824 may include sets of contour data N 826 and labeling data N 828 generated by a medical practitioner 830. The contour data N 826 may include contours around anatomical structures in ultrasound images, such as one or more of blood vessels, arteries, veins, or other anatomical landmarks, and associated labels. The labeling data N 828 may include location labels, for example.

[0060] The training data 822 is used to train an Al or machine-learning model, such as the trained machine-learning model 800, e.g., the Al model 704 of FIG. 7. The training data 822 may be applied to a neural network structure 834, such as a DNN, comprising an input layer, one or more hidden layers, and an output layer. The training data 822, along with labeling data N 828, may be fed into the input layer of the neural network structure 834,which propagates the input data or data features through one or more hidden layers to the output layer that outputs weights and bias to form the trained machine-learning model 800. The trained machine-learning model 800 is able to perform tasks without explicitly being programmed by making inferences based on patterns found in the analysis of data.

[0061] FIG. 9 is a flow diagram of an example of a method 900 for labeling lymph nodes during an endobronchial ultrasound (EBUS) procedure. At block 902, the method 900 includes receiving the ultrasound image from the EBUS device and providing the ultrasound image from the EBUS device to a machine-learning model trained to identify a lymph node in the ultrasound image. For example, the image processing unit 42 of FIG. 2 receives one or more images from the EBUS bronchoscope 306 of FIG. 3 and provides the data representing the ultrasound image to the trained machine-learning model 800 of FIG. 8, which is trained to identify a lymph node in the ultrasound image. The processing unit is configured for providing the data to the machine-learning model.

[0062] At block 904, the method 900 includes receiving an identification of a lymph node in the ultrasound image from the machine-learning model. For example, the image processing unit 42 of FIG. 2 receives an identification of the lymph node in the ultrasound image from the trained machine-learning model 800.

[0063] At block 906, the method 900 includes displaying, on the user interface, the ultrasound image overlaid with a label indicating the identified lymph node. For example, the image processing unit 42 of FIG. 2 may display on the display unit 18 the ultrasound image overlaid with a label indicating the identified lymph node, such as shown in FIGS. 4- 6.

[0064] In some examples, the method 900 further includes receiving an identification of an anatomical structure in the ultrasound image from the machine-learning model, and displaying, on the user interface, the ultrasound image overlaid with a label indicating the identified anatomical structure, such as one or more of one or more of blood vessels, arteries, veins, or other anatomical landmarks. For example, the image processing unit 42 of FIG. 2 receives an identification of an anatomical structure in the ultrasound image from the trained machine-learning model 800 and displays on the display unit 18 the ultrasound image overlaid with a label indicating the identified anatomical structure, such as shown in FIGS. 4-6.

[0065] In some examples, the image processing unit 42 may display on the display unit 18 the position and / or the orientation of the EBUS device, such as relative to a patient's airway.In some examples, the identification of the lymph node in the ultrasound image is based at least partially on the positioning data.

[0066] FIG. 10 illustrates a block diagram of an example machine 1000 upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform. Examples, as described herein, may include, or may operate by, logic or a number of components, or mechanisms in the machine 1000. Circuitry (e.g., processing circuitry) is a collection of circuits implemented in tangible entities of the machine 1000 that include hardware (e.g., simple circuits, gates, logic, etc.). Circuitry membership may be flexible over time. Circuitries have members that may, alone or in combination, perform specified operations when operating. In an example, hardware of the circuitry may be immutably designed to carry out a specific operation (e.g., hardwired). In an example, the hardware of the circuitry may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) including a machine readable medium physically modified (e.g., magnetically, electrically, moveable placement of invariant massed particles, etc.) to encode instructions of the specific operation. In connecting the physical components, the underlying electrical properties of a hardware constituent are changed, for example, from an insulator to a conductor or vice versa. The instructions enable embedded hardware (e.g., the execution units or a loading mechanism) to create members of the circuitry in hardware via the variable connections to carry out portions of the specific operation when in operation. Accordingly, in an example, the machine-readable medium elements are part of the circuitry or are communicatively coupled to the other components of the circuitry when the device is operating. In an example, any of the physical components may be used in more than one member of more than one circuitry. For example, under operation, execution units may be used in a first circuit of a first circuitry at one point in time and reused by a second circuit in the first circuitry, or by a third circuit in a second circuitry at a different time. Additional examples of these components with respect to the machine 1000 follow.

[0067] In alternative examples, the machine 1000 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine 1000 may operate in the capacity of a server machine, a client machine, or both in server-client network environments. In an example, the machine 1000 may act as a peer machine in peer-to-peer (P2P) (or other distributed) network environment. The machine 1000 may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term“machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations.

[0068] The machine 1000 may include a hardware processing unit 1002 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processing unit core, or any combination thereof), a main memory 1004, a static memory (e.g., memory or storage for firmware, microcode, a basic-input-output (BIOS), and mass storage 1008 (e.g., hard drives, tape drives, flash storage, or other block devices) some or all of which may communicate with each other via an interlink 1030 (e.g., bus). The machine 1000 may further include a display unit 1010, an alphanumeric input device 1012 (e.g., a keyboard), and a user interface (UI) navigation device 1014 (e.g., a mouse). In an example, the display unit 1010, input device 1012 and UI navigation device 1014 may be a touch screen display. The machine 1000 may additionally include a signal generation device 1018 (e.g., a speaker), a network interface device 1020, and one or more sensors 1016, such as a global positioning system (GPS) sensor, compass, accelerometer, or other sensor. The machine 1000 may include an output controller 1028, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.).

[0069] Registers of the processing unit 1002, the main memory 1004, the static memory 1006, or the mass storage 1008 may be, or include, a machine-readable medium 1022 on which is stored one or more sets of data structures or instructions 1024 (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein. The instructions 1024 may also reside, completely or at least partially, within any of registers of the processing unit 1002, the main memory 1004, the static memory 1006, or the mass storage 1008 during execution thereof by the machine 1000. In an example, one or any combination of the hardware processing unit 1002, the main memory 1004, the static memory 1006, or the mass storage 1008 may constitute the machine-readable media 1022. While the machine readable medium 1022 is illustrated as a single medium, the term “machine readable medium” may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store the one or more instructions 1024.

[0070] The term “machine-readable medium” may include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine 1000 and that causethe machine 1000 to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Non-limiting machine-readable medium examples may include solid-state memories, optical media, magnetic media, and signals (e.g., radio frequency signals, other photon-based signals, sound signals, etc.). In an example, a non-transitory machine-readable medium comprises a machine-readable medium with a plurality of particles having invariant (e.g., rest) mass, and thus are compositions of matter. Accordingly, non-transitory machine- readable media are machine-readable media that do not include transitory propagating signals. Specific examples of non-transitory machine-readable media may include: nonvolatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0071] In an example, information stored or otherwise provided on the machine readable medium 1022 may be representative of the instructions 1024, such as instructions 1024 themselves or a format from which the instructions 1024 may be derived. This format from which the instructions 1024 may be derived may include source code, encoded instructions (e.g., in compressed or encrypted form), packaged instructions (e.g., split into multiple packages), or the like. The information representative of the instructions 1024 in the machine readable medium 1022 may be processed by processing circuitry into the instructions to implement any of the operations discussed herein. For example, deriving the instructions 1024 from the information (e.g., processing by the processing circuitry) may include: compiling (e.g., from source code, object code, etc.), interpreting, loading, organizing (e.g., dynamically or statically linking), encoding, decoding, encrypting, unencrypting, packaging, unpackaging, or otherwise manipulating the information into the instructions 1024.

[0072] In an example, the derivation of the instructions 1024 may include assembly, compilation, or interpretation of the information (e.g., by the processing circuitry) to create the instructions 1024 from some intermediate or preprocessed format provided by the machine-readable medium 1022. The information, when provided in multiple parts, may be combined, unpacked, and modified to create the instructions 1024. For example, the information may be in various compressed source code packages (or object code, or binary executable code, etc.) on one or several remote servers. The source code packages may be encrypted when in transit over a network and decrypted, uncompressed, assembled (e.g.,linked) if necessary, and compiled or interpreted (e.g., into a library, stand-alone executable etc.) at a local machine, and executed by the local machine.

[0073] The instructions 1024 may be further transmitted or received over a communications network 1026 using a transmission medium via the network interface device 1020 utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), LoRa / LoRaWAN, or satellite communication networks, mobile telephone networks (e.g., cellular networks such as those complying with 3G, 4G LTE / LTE-A, or 5G standards), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) 1002.11 family of standards known as Wi-Fi®, IEEE 1002.15.4 family of standards, peer-to-peer (P2P) networks, among others. In an example, the network interface device 1020 may include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas to connect to the communications network 1026. In an example, the network interface device 1020 may include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input singleoutput (MISO) techniques. The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by the machine 1000, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software. A transmission medium is a machine-readable medium.Various Notes

[0074] Each of the non-limiting claims or examples described herein may stand on its own, or may be combined in various permutations or combinations with one or more of the other examples.

[0075] The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are also referred to herein as “examples.” Such examples may include elements in addition to those shown or described. However, the present inventors also contemplate examples in which only those elements shown or described are provided. Moreover, the present inventors also contemplate examples using any combination or permutation of thoseelements shown or described (or one or more claims thereof), either with respect to a particular example (or one or more claims thereof), or with respect to other examples (or one or more claims thereof) shown or described herein.

[0076] In the event of inconsistent usages between this document and any documents so incorporated by reference, the usage in this document controls.

[0077] In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of “at least one” or “one or more.” In this document, the term “or” is used to refer to a nonexclusive or, such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. In this document, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” Also, in the following claims, the terms “including” and “comprising” are open-ended, that is, a system, device, article, composition, formulation, or process that includes elements in addition to those listed after such a term in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to impose numerical requirements on their objects.

[0078] Method examples described herein may be machine or computer-implemented at least in part. Some examples may include a computer-readable medium or machine-readable medium encoded with instructions operable to configure an electronic device to perform methods as described in the above examples. An implementation of such methods may include code, such as microcode, assembly language code, a higher-level language code, or the like. Such code may include computer readable instructions for performing various methods. The code may form portions of computer program products. Further, in an example, the code may be tangibly stored on one or more volatile, non-transitory, or nonvolatile tangible computer-readable media, such as during execution or at other times. Examples of these tangible computer-readable media may include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact discs and digital video discs), magnetic cassettes, memory cards or sticks, random access memories (RAMs), read only memories (ROMs), and the like.

[0079] The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more claims thereof) may be used in combination with each other. Other embodiments may be used, such as by one of ordinary skill in the art upon reviewing the above description. The Abstract is provided to allow thereader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the above Detailed Description, various features may be grouped together to streamline the disclosure. This should not be interpreted as intending that an unclaimed disclosed feature is essential to any claim. Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description as examples or embodiments, with each claim standing on its own as a separate embodiment, and it is contemplated that such embodiments may be combined with each other in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

CLAIMSWhat is claimed is:

1. A system for labeling lymph nodes during an endobronchial ultrasound (EBUS) procedure, the system comprising: an EBUS device configured to capture an ultrasound image of a patient's airway and surrounding structures; a user interface including a display; and a processing unit and a non-transitory computer-readable medium storing instructions that, when executed, cause the processing unit to perform operations including: receiving the ultrasound image from the EBUS device and providing the ultrasound image from the EBUS device to a machine-learning model trained to identify a lymph node in the ultrasound image; receiving an identification of the lymph node in the ultrasound image from the machine-learning model; and displaying, on the user interface, the ultrasound image overlaid with a label indicating the identified lymph node.

2. The system of claim 1, wherein the machine-learning model is further trained to identify anatomical structures adjacent to lymph nodes, and wherein the non-transitory computer- readable medium includes further instructions that, when executed, cause the processing unit to perform operations including: receiving an identification of an anatomical structure in the ultrasound image from the machine-learning model; and displaying, on the user interface, the ultrasound image overlaid with a label indicating the identified anatomical structure.

3. The system of claim 2, wherein the anatomical structures include one or more of one or more of blood vessels, arteries, veins, or other anatomical landmarks.

4. The system of claim 1, wherein the EBUS device includes a sensor configured for detecting a position of the EBUS device relative to the patient's airway and providing positioning data to the processing unit, and wherein the non-transitory computer-readable medium includes further instructions that, when executed, cause the processing unit to perform operations including: displaying, on the user interface, data representing the position of the EBUS device.

5. The system of claim 4, wherein the processing unit is configured for providing the data to the machine-learning model, and wherein the identification of the lymph node in the ultrasound image is based at least partially on the positioning data.

6. The system of claim 4, wherein the sensor includes a magnetic sensor.

7. The system of claim 4, wherein the sensor is positioned adjacent a distal tip of the EBUS device.

8. The system of claim 1, wherein the EBUS device includes a sensor configured for detecting an orientation of the EBUS device relative to the patient's airway and providing orientation data to the processing unit, and wherein the non-transitory computer-readable medium includes further instructions that, when executed, cause the processing unit to perform operations including: displaying, on the user interface, data representing the orientation of the EBUS device.

9. The system of claim 8, wherein the processing unit is configured for providing the orientation data to the machine-learning model, and wherein the identification of the lymph node in the ultrasound image is based at least partially on the orientation data.

10. The system of claim 8, wherein the sensor includes a plurality of fiber Bragg gratings.

11. The system of claim 1, wherein the EBUS device includes a EBUS bronchoscope.

12. A method for labeling lymph nodes during an endobronchial ultrasound (EBUS) procedure, the method comprising: receiving an ultrasound image from an EBUS device and providing the ultrasound image from the EBUS device to a machine-learning model trained to identify a lymph node in the ultrasound image; receiving an identification of the lymph node in the ultrasound image from the machine-learning model; and displaying, on a user interface, the ultrasound image overlaid with a label indicating the identified lymph node.

13. The method of claim 12, wherein the machine-learning model is further trained to identify anatomical structures adjacent to lymph nodes, the method further comprising:receiving an identification of an anatomical structure in the ultrasound image from the machine-learning model; and displaying, on the user interface, the ultrasound image overlaid with a label indicating the identified anatomical structure.

14. The method of claim 13, wherein displaying, on the user interface, the ultrasound image overlaid with the label indicating the identified anatomical structure includes: displaying, on the user interface, the ultrasound image overlaid with the label indicating one or more of blood vessels, arteries, veins, or other anatomical landmarks.

15. The method of claim 12, comprising: displaying, on the user interface, data representing a position of the EBUS device.

16. The method of claim 12, comprising: displaying, on the user interface, data representing an orientation of the EBUS device.

17. The method of claim 12, wherein receiving an ultrasound image from the EBUS device includes: receiving an ultrasound image from an EBUS bronchoscope.

18. A system for labeling lymph nodes during an endobronchial ultrasound (EBUS) procedure, the system comprising: an EBUS device configured to capture an ultrasound image of a patient's airway and surrounding structures; a sensor configured for detecting at least one of a position or an orientation of the EBUS device relative to the patient's airway and providing at least one of positioning data or orientation data to a processing unit; a user interface including a display; and the processing unit and a non-transitory computer-readable medium storing instructions that, when executed, cause the processing unit to perform operations including: receiving the ultrasound image from the EBUS device and providing the ultrasound image from the EBUS device to a machine-learning model trained to identify a lymph node in the ultrasound image; receiving an identification of the lymph node in the ultrasound image from the machine-learning model; anddisplaying, on the user interface, the ultrasound image overlaid with a label indicating the identified lymph node and data representing at least one of the position or the orientation of the EBUS device.

19. The system of claim 18, wherein the machine-learning model is further trained to identify anatomical structures adjacent to lymph nodes, and wherein the non-transitory computer-readable medium includes further instructions that, when executed, cause the processing unit to perform operations including: receiving an identification of an anatomical structure in the ultrasound image from the machine-learning model; and displaying, on the user interface, the ultrasound image overlaid with a label indicating the identified anatomical structure.

20. The system of claim 19, wherein the anatomical structures include one or more of one or more of blood vessels, arteries, veins, or other anatomical landmarks.

21. The system of claim 18, wherein the processing unit is configured for providing the data to the machine-learning model, and wherein the identification of the lymph node in the ultrasound image is based at least partially on the positioning data.

22. The system of claim 18, wherein the sensor includes a plurality of fiber Bragg gratings.

23. The system of claim 18, wherein the EBUS device includes a EBUS bronchoscope.

Images