Needle trajectory prediction

Abstract

Systems, devices, and methods for planning an ultrasound-guided tissue acquisition process are disclosed. An example system includes an ultrasound imaging device for generating an ultrasound image of an anatomical target in a real-time ultrasound field of view (FOV), a display for displaying the ultrasound image, and a controller circuit. Prior to extending the tissue sampling device into the ultrasound FOV, the controller circuit displays a graphical user interface element (UIE) indicating a predicted nominal sampling device trajectory that intersects an anatomical target in the ultrasound FOV. As the tissue sampling device extends into the ultrasound FOV, the controller circuit determines one or more real-time trajectory parameters and updates the UIE. The controller circuit displays the updated UIE to provide visual feedback as to whether the tissue sampling device will intersect the anatomical target.

Description

[0001] Priority requirements

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 581,084, filed September 7, 2023, and U.S. Provisional Patent Application Serial No. 63 / 607,654, filed December 8, 2023, the contents of which are incorporated herein by reference. Technical Field

[0003] This article generally relates to ultrasound equipment, and more specifically, but not limited to, ultrasound-guided tissue acquisition equipment, which includes a graphical user interface to assist in navigating the tissue sampling equipment to a target for collecting tissue samples. Background Technology

[0004] Endoscopes have been used in a variety of clinical procedures, including, for example, illumination, imaging, detection and diagnosis of one or more disease states; providing fluid delivery toward anatomical areas (e.g., saline or other preparations via a fluid channel); providing channels (e.g., via a working channel) for allowing tissue sampling or biopsy equipment to pass through the channel to collect tissue samples from anatomical targets or for allowing diagnostic or therapeutic equipment to pass through the channel for medical diagnosis or treatment; or providing aspiration channels to collect fluids and unwanted objects (e.g., tissue or stone structures) from anatomical areas, and so on. Anatomical areas or targets to be intervened with include the gastrointestinal tract, respiratory tract and lungs, renal system organs and tissues, sinuses, submucosal regions, reproductive system organs, and so on. Some endoscopes can be used with energy sources such as lasers or plasma systems to deliver therapeutic energy (e.g., laser pulses) to anatomical targets such as soft or hard tissues or stone structures to achieve a variety of therapeutic goals. For example, endoscopic lasers have been used in applications including tissue ablation, coagulation, vaporization, fragmentation, and lithotripsy to break down stones in the kidneys, gallbladder, ureters, and other stone-forming areas, or to ablate large stones into smaller fragments.

[0005] Endoscopic ultrasound (EUS) is a specialized endoscopic procedure that combines routine endoscopy with ultrasound imaging to obtain ultrasound images of anatomical targets or regions of interest. Such specialized endoscopes, also known as echo endoscopes, include an ultrasound transducer typically located in the distal portion of the slender endoscope body. The ultrasound transducer emits ultrasound waves toward the anatomical target and converts the ultrasound echoes into ultrasound images (e.g., a live image stream).

[0006] Endobronchial ultrasound (EBUS) has been used to diagnose lung diseases. EBUS is a minimally invasive and efficient procedure used to diagnose lung cancer, infections, and other conditions that cause swelling of the thoracic lymph nodes. EBUS is typically performed using a specialized bronchoscope associated with an ultrasound transducer or removable ultrasound probe, delivered through a working channel and exiting from a port located distal to the bronchoscope. EBUS has also been used for tissue sampling or biopsy procedures called endobronchial ultrasound-guided transbronchial fine needle aspiration (EBUS-TBNA), in which a specialized sampling device, such as a biopsy needle, can be pushed down and then extended from the bronchoscope, and under real-time guidance of the EBUS, to collect tissue samples from peribronchial masses, such as peribronchial nodules, or from peribronchial lymph nodes. The sampled tissue can be analyzed to aid in the diagnosis of various diseases, such as tuberculosis, sarcoidosis, or cancer. Summary of the Invention

[0007] The inventors have recognized several technical problems that need to be solved using conventional EUS systems and techniques, particularly those related to the EBUS-TBNA procedure. One such technical problem is that existing endobronchial sampling devices are generally designed for procedures involving shallow and large-diameter airways of the respiratory system, where clinicians can freely advance, retract, or tilt the transbronchial aspiration or sampling device (e.g., a needle) to align with a nominal trajectory. The sampling device can be manipulated to travel along the nominal trajectory and eventually intersect with an anatomical target of interest, such as a peribronchial nodule. However, existing endobronchial sampling devices are generally not designed (and therefore may not be ideal) to collect tissue samples from targets located deep within the peripheral regions of the lungs (e.g., primary, secondary, or tertiary bronchi or bronchioles), which can only be accessed via very distal and narrow airways. As the inner diameter (ID) of the airway decreases toward the distal end, the distance the sampling device can extend into the airway is generally limited by the ID of the airway and the outer diameter (OD) of the sampling device. If the target nodule becomes clearly imaged at a location below a nominal trajectory line, clinicians may find it difficult to advance the sampling device further into the airway to align the target nodule with the nominal trajectory (e.g., pushing or forcing a sampling device with a 1.9 mm OD into or into a portion of the airway with an ID less than 1.9 mm), because further advancement of the sampling device may cause trauma to the airway walls or other unintended anatomical structures.

[0008] In EBUS-TBNA procedures involving targets close to shallow and large-diameter airways, juxtaposition is typically achieved using a saline-filled balloon surrounding the ultrasound transducer. This refers to the appropriate level of direct or gapless contact and pressure required for ultrasound energy to propagate from and return to the transducer. However, the presence of an air gap between the balloon and the contacting tissue can lead to high impedance mismatch at the typical frequencies of the ultrasound transducer. This can result in degraded image quality or even complete loss of images beyond the air gap.

[0009] In EBUS-TBNA procedures involving targets deep within the peripheral lung region, balloons are typically not used; instead, the ultrasound transducer is in direct contact with the airway wall, a process known as “direct” juxtaposition. Endobronchial sampling devices used in such direct juxtaposition scenarios face several technical challenges, including aligning the target (e.g., a peribronchial nodule) with a predetermined nominal trajectory while maintaining proper juxtaposition between the ultrasound transducer and the airway wall to produce high-quality ultrasound images. For example, to ensure correct alignment along the predetermined nominal trajectory, existing EBUS-TBNA systems typically rely on the operator visually confirming that the tissue sampling needle has penetrated the desired lesion (or a portion thereof) by guiding the needle toward and ultimately into the target within the ultrasound field of view (FOV). However, existing EBUS-TBNA systems often do not provide sufficient visual assistance to help clinicians navigate the sampling needle to the target, such as precisely aligning the target with the nominal needle trajectory before extending the needle out of the echo endoscope and before the needle enters the ultrasound FOV.

[0010] Another technical issue with this EBUS-TBNA system involves uncertainty or deviation from the initial nominal needle trajectory. Obtaining sufficient biopsy samples during the EBUS-TBNA procedure requires accurate targeting of the target nodule with the sampling needle. However, during the EBUS-TBNA procedure, the needle trajectory can vary at least depending on the initial needle posture and puncture angle when the airway wall is first penetrated after exiting the lateral exit bevel. For example, in an example where the nominal needle trajectory is determined as the linear axis of the sampling needle exiting the lateral exit bevel of the echo endoscope, even if the clinician directly aligns the target nodule with such a determined nominal needle trajectory in the ultrasound FOV, a small deviation in the puncture angle at which the sampling needle penetrates the airway wall can lead to a substantial deviation from the nominal needle trajectory as the sampling needle moves further through the ultrasound FOV and deeper into the patient's tissue.

[0011] Furthermore, existing EBUS-TBNA systems typically do not consider the actual needle trajectory or real-time needle trajectory parameters to provide real-time feedback on whether the extending needle will intersect or miss the anatomical target, let alone provide quantitative assessments such as the probability of intersection. In cases where the target lesion is very large and essentially occupies the entire ultrasound FOV generated by a linear or curved ultrasound transducer, even coarse alignment within the ultrasound FOV is sufficiently accurate due to the size of the lesion. However, in the peripheral regions of the lung, solitary pulmonary nodules can be very small (e.g., 3 mm or less in diameter) and can be 10 mm to 20 mm or more deep within the tissue. Therefore, a higher level of precision than the mere coarse alignment typically performed with existing systems is required.

[0012] At least for the reasons stated above, the inventors have recognized an unmet need for an improved technique that can efficiently generate clear ultrasound images of anatomical targets during EBUS-TBNA procedures while maintaining and dynamically adjusting the proper alignment of the anatomical target with the nominal trajectory as needed when extending the sampling device to the periphery of the lung (e.g., primary, secondary, or tertiary bronchi or bronchioles). This disclosure describes systems, apparatus, and methods for automatically planning ultrasound-guided tissue acquisition procedures. An exemplary system includes an ultrasound imaging apparatus for generating ultrasound images of anatomical targets in a real-time ultrasound field of view (FOV) of a region of interest, a display for displaying the ultrasound images in the real-time ultrasound FOV, and controller circuitry. Before extending the tissue sampling device into the real-time ultrasound FOV, the controller circuitry may display a graphical user interface element (UIE) indicating a predicted nominal sampling device trajectory intersecting with the anatomical target in the real-time ultrasound FOV. As the tissue sampling device is extended into the real-time ultrasound FOV, the controller circuitry may determine one or more real-time trajectory parameters based on analysis of the ultrasound images and update the UIE to represent the actual sampling device trajectory. The controller circuitry can display an updated UIE on the real-time ultrasound FOV to provide visual feedback on whether the tissue sampling device will intersect the target. The controller circuitry can determine the intersection probability and dynamically update the sampling device trajectory, at least in part, based on one or more determined real-time trajectory parameters.

[0013] Example 1 is a system for planning an ultrasound-guided tissue acquisition process. The system includes: an ultrasound imaging device configured to generate an ultrasound image of an anatomical target in a real-time ultrasound field of view (FOV) of a region of interest; a display configured to display the ultrasound image in the real-time ultrasound FOV; and controller circuitry configured to: display a graphical user interface element (UIE) indicating a predicted trajectory of the sampling device intersecting the anatomical target on the real-time ultrasound FOV before the tissue sampling device extends into the real-time ultrasound FOV; determine one or more real-time trajectory parameters based on analysis of the ultrasound image as the tissue sampling device extends into the real-time ultrasound FOV; update the UIE based on the determined one or more real-time trajectory parameters to represent at least one of the actual sampling device trajectory or the updated predicted sampling device trajectory; and display the updated UIE on the real-time ultrasound FOV to provide visual feedback on whether the tissue sampling device will intersect the anatomical target.

[0014] In Example 2, the subject of Example 1 may optionally include controller circuitry that can be configured to: determine the location of the anatomical target within the real-time ultrasound FOV; and determine the probability of a tissue sampling device intersecting with the anatomical target based at least in part on the determined target location within the real-time ultrasound FOV.

[0015] In Example 3, the subject matter of Example 2 may optionally include, in order to determine the location of an anatomical target, controller circuitry is configured to: receive user input via a user interface identifying an anatomical target within a real-time ultrasound FOV; and analyze ultrasound images within the real-time ultrasound FOV to determine the location of the anatomical target identified by the user.

[0016] In Example 4, the subject matter described in any one or more of Examples 1 to 3 may optionally include one or more real-time trajectory parameters, which may include one or more of the position, orientation, or direction of advance or angle of entry of the tissue sampling device when entering the real-time ultrasound FOV.

[0017] In Example 5, the subject of Example 4 may optionally include controller circuitry that can be configured to determine the predicted trajectory of the tissue sampling device based on the posture or entry angle of the tissue sampling device as it enters the real-time ultrasound FOV.

[0018] In Example 6, the subject matter described in any one or more of Examples 1 to 5 may optionally include controller circuitry that can be configured to determine a predicted sampling device trajectory based on the hardware configuration of the tissue sampling device, which includes a pre-formed curvature or bending angle of the tissue sampling device.

[0019] In Example 7, the subject matter described in any one or more of Examples 1 to 6 may optionally include controller circuitry that can be configured to use population data to determine at least one of a predicted sampling device trajectory or an updated predicted sampling device trajectory, said population data including sampling device trajectories collected from a similar ultrasound-guided tissue acquisition procedure performed on a patient population.

[0020] In Example 8, the subject matter described in any one or more of Examples 1 to 7 may optionally include controller circuitry, which may also be configured to: determine an upper and lower trajectory limit that defines a trajectory range within which an intersection with an anatomical target may occur at a specific confidence level; and display a UIE indicating the determined upper and lower trajectory limits on the real-time ultrasound FOV before the tissue sampling device extends into the real-time ultrasound FOV.

[0021] In Example 9, the subject of Example 8 optionally includes controller circuitry that can be configured to use population data to determine upper and lower limits of the trajectory, the population data including sampling device trajectories collected from a similar ultrasound-guided tissue acquisition procedure performed on a patient population.

[0022] In Example 10, the subject matter described in any one or more of Examples 1 to 9 may optionally include controller circuitry that can be configured to determine the probability of a tissue sampling device intersecting with an anatomical target based at least in part on the actual sampling device trajectory relative to the internal anatomical target of the real-time ultrasound FOV or one or more real-time trajectory parameters.

[0023] In Example 11, the subject of Example 10 may optionally include controller circuitry that can be configured to: provide suggestions for withdrawing the tissue sampling device or adjusting the position or orientation of the tissue sampling device relative to the anatomical target within the real-time ultrasound FOV in response to a hit probability below a probability threshold; and provide a notification to the user to extend the tissue sampling device according to the predicted sampling device trajectory in response to a hit probability greater than the probability threshold.

[0024] In Example 12, the subject matter described in any one or more of Examples 1 to 11 may optionally include controller circuitry, which may also be configured to: dynamically update the predicted sampling device trajectory based at least in part on one or more determined real-time trajectory parameters; and update the UIE to represent the updated predicted sampling device trajectory.

[0025] In Example 13, the subject of Example 12 may optionally include controller circuitry that can also be configured to dynamically update one or more of the upper or lower trajectory limits, which define a trajectory range within which an intersection with the anatomical target may occur at a specific confidence level.

[0026] In Example 14, the subject matter described in any one or more of Examples 12 to 13 may optionally include controller circuitry that may also be configured to determine an updated probability of a tissue sampling device intersecting with an anatomical target as it travels along an updated predicted sampling device trajectory.

[0027] In Example 15, the subject matter described in any one or more of Examples 1 to 14 may optionally include an ultrasound imaging device that may be located distal to an endobronchial sampling device configured to be inserted into the patient’s airway during an endobronchial ultrasound-guided transbronchial fine needle aspiration (EBUS-TBNA) procedure.

[0028] Example 16 is an intraluminal imaging device system comprising: an intraluminal imaging device including a tubular device body having an inner lumen and a side exit port; an ultrasound transducer configured to generate an ultrasound scan of an anatomical target and generate an ultrasound image in a real-time ultrasound field of view (FOV) of a region of interest; a display configured to display the ultrasound image in the real-time ultrasound FOV; and a tissue sampling device configured to operatively pass through the lumen under ultrasound guidance and exit from the side exit port of the tubular device body, and advance toward the anatomical target according to a predicted nominal trajectory and subsequently intersect with the anatomical target; The controller circuit is configured to: display a graphical user interface element (UIE) indicating a predicted trajectory of the sampling device intersecting with an anatomical target on the real-time ultrasound FOV before the tissue sampling device extends into the real-time ultrasound FOV; analyze ultrasound images to determine one or more real-time trajectory parameters as the tissue sampling device extends into the real-time ultrasound FOV; update the UIE based on the determined one or more real-time trajectory parameters to represent at least one of the actual sampling device trajectory or the updated predicted sampling device trajectory; and display the updated UIE on the real-time ultrasound FOV to provide visual feedback on whether the tissue sampling device will intersect with the anatomical target.

[0029] In Example 17, the subject of Example 16 may optionally include a tissue sampling device, which may include a needle, brush, snare, aspiration device, tweezers, or an auxiliary insertion device.

[0030] In Example 18, the subject matter described in any one or more of Examples 16 to 17 may optionally include a tissue sampling device comprising a transbronchial needle for sampling tissue from a peripheral lung target during an intrabronchial ultrasound-guided transbronchial fine needle aspiration (EBUS-TBNA) procedure.

[0031] In Example 19, the subject matter described in any one or more of Examples 16 to 18 may optionally include a tissue sampling device that may include a needle-stylet combination comprising a needle and a stylet insertable into the needle, wherein one or more of the stylet or needle have a preformed curvature or bending angle that substantially conforms to the predicted nominal sampling device trajectory.

[0032] In Example 20, the subject matter described in any one or more of Examples 16 to 19 may optionally include controller circuitry that can be configured to: determine the location of the anatomical target within the real-time ultrasound FOV; and determine, at least in part, the probability of a tissue sampling device intersecting with the anatomical target based on the determined target location within the real-time ultrasound FOV.

[0033] In Example 21, the subject matter described in any one or more of Examples 16 to 20 may optionally include controller circuitry, which may also be configured to: determine an upper and lower trajectory limit that defines a trajectory range within which an intersection with an anatomical target may occur at a specific confidence level; and display a UIE indicating the determined upper and lower trajectory limits on the real-time ultrasound FOV before the tissue sampling device extends into the real-time ultrasound FOV.

[0034] In Example 22, the subject matter described in any one or more of Examples 26 to 21 may optionally include controller circuitry that can be configured to: determine, at least in part, the probability of a tissue sampling device intersecting with an anatomical target based on the actual sampling device trajectory relative to the anatomical target within the real-time ultrasound FOV; and display the probability of a hit on a display.

[0035] In Example 23, the subject of Example 22 may optionally include controller circuitry that can be configured to: provide suggestions to withdraw the tissue sampling device or adjust the position or orientation of the tissue sampling device relative to the anatomical target within the FOV in response to a hit probability below a probability threshold; and to provide a notification to the user to extend the tissue sampling device according to the predicted sampling device trajectory in response to a hit probability greater than the probability threshold.

[0036] In Example 24, the subject matter described in any one or more of Examples 16 to 23 may optionally include controller circuitry, which may also be configured to: determine, based on analysis of ultrasound images, whether a tissue sampling device has intersected with an anatomical target within the real-time ultrasound FOV; dynamically update a predicted sampling device trajectory based on one or more determined real-time trajectory parameters in response to an analysis indicating that the tissue sampling device has not yet intersected with the anatomical target; and update the UIE to represent the updated predicted sampling device trajectory.

[0037] In Example 25, the subject of Example 24 may optionally include controller circuitry that may also be configured to dynamically update one or more of an upper or lower trajectory limit in response to an analysis indicating that the tissue sampling device has not yet intersected with the anatomical target. The upper and lower trajectory limits define a trajectory range within which an intersection with the anatomical target may occur at a specific confidence level.

[0038] In Example 26, the subject matter described in any one or more of Examples 24 to 25 may optionally include controller circuitry that may also be configured to determine an updated probability of a tissue sampling device intersecting with an anatomical target as it travels along an updated predicted sampling device trajectory.

[0039] Example 27 is a method for planning an ultrasound-guided tissue acquisition process. The method includes the following steps: generating and displaying an ultrasound image of an anatomical target in a real-time ultrasound field of view (FOV) of a region of interest; displaying a graphical user interface element (UIE) indicating a predicted sampling device trajectory intersecting the anatomical target on the real-time ultrasound FOV before the tissue sampling device extends into the real-time ultrasound FOV; determining one or more real-time trajectory parameters based on analysis of the ultrasound image as the tissue sampling device extends into the real-time ultrasound FOV; updating the UIE based on the determined one or more real-time trajectory parameters to represent at least one of an actual needle trajectory or an updated predicted sampling device trajectory; and displaying the updated UIE on the real-time ultrasound FOV to provide visual feedback on whether the needle will intersect the anatomical target.

[0040] In Example 28, the subject matter of Example 27 may optionally include: determining the location of an anatomical target within a real-time ultrasound FOV; and determining the probability of a tissue sampling device intersecting with the anatomical target based at least in part on the determined target location within the real-time ultrasound FOV.

[0041] In Example 29, the subject matter described in any one or more of Examples 27 to 28 may optionally include: determining an upper and lower trajectory limit that defines a trajectory range within which an intersection with an anatomical target may occur at a specific confidence level; and displaying a UIE indicating the determined upper and lower trajectory limits on a real-time ultrasound FOV.

[0042] In Example 30, the subject matter described in any one or more of Examples 27 to 29 may optionally include: determining the probability of a tissue sampling device intersecting with an anatomical target based at least in part on the actual needle trajectory relative to the anatomical target within the real-time ultrasound FOV; and displaying the probability of a hit.

[0043] In Example 31, the subject of Example 30 may optionally include: providing suggestions to withdraw the tissue sampling device or adjust the position or orientation of the tissue sampling device relative to the anatomical target within the FOV in response to a hit probability below a probability threshold; and providing a notification to the user to extend the tissue sampling device according to the predicted sampling device trajectory in response to a hit probability greater than a probability threshold.

[0044] In Example 32, the subject matter described in any one or more of Examples 27 to 31 may optionally include: dynamically updating the predicted sampling device trajectory based on one or more determined real-time trajectory parameters; and updating the UIE to represent the updated predicted sampling device trajectory.

[0045] In Example 33, the subject of Example 32 may optionally include dynamically updating one or more of the upper or lower trajectory limits, which define a trajectory range within which intersection with the anatomical target may occur at a specific confidence level.

[0046] In Example 34, the subject matter described in any one or more of Examples 32 to 33 may optionally include determining an updated hit probability that the tissue sampling device intersects with the anatomical target as it travels along an updated predicted sampling device trajectory.

[0047] The techniques presented are described in accordance with health-related processes, but are not limited thereto. This disclosure is an overview of some of the teachings of this application and is not intended to be exclusive or exhaustive of the subject matter. Further details regarding the subject matter can be found in the detailed description and the appended claims. Other aspects of this disclosure will become apparent to those skilled in the art upon reading and understanding the following detailed description and viewing the accompanying drawings, which form a part thereof, and each of the drawings should not be considered limiting. The scope of this disclosure is defined by the appended claims and their legal equivalents. Attached Figure Description

[0048] Figure 1This is a schematic diagram illustrating an example of an echo endoscope system used in endoscopic ultrasound (EUS) procedures.

[0049] Figure 2 It shows things like Figure 1 A perspective view of the distal portion of the echo endoscope shown.

[0050] Figure 3 An example of an endobronchial ultrasound-guided transbronchial fine needle aspiration (EBUS-TBNA) procedure is shown, along with a portion of the EBUS-TBNA system used in the procedure.

[0051] Figure 4 This is a block diagram illustrating an example of an EUS-guided tissue acquisition (EUS-TA) planning system that can automatically generate EUS-TA plans for medical procedures such as EBUS-TBNA.

[0052] Figures 5A to 5B , Figures 6A to 6B and Figures 7A to 7B This is a diagram illustrating an example endobronchial ultrasound (EBUS) system with an associated computing system that determines and displays graphical representations of the nominal sampling device trajectory and user interface elements (UIEs) at different operating states of an exemplary operational flow during the EBUS-TBNA process on the user interface.

[0053] Figures 8A to 8C This is a graph showing the UIE of the hit probability, which can be estimated and dynamically updated in association with the determination and dynamic updates of the nominal needle trajectory and the associated UTL and LTL, as described above regarding... Figures 5A to 5B , Figures 6A to 6B and Figures 7A to 7B The various operating states described above.

[0054] Figure 9 This is a flowchart illustrating an example method for generating EUS-TA plans and presenting visual aids on a user interface to assist in the navigation of tissue sampling devices during medical procedures such as EBUS-TBNA.

[0055] Figure 10 This is a block diagram illustrating an example machine on which any or more of the techniques (e.g., methods) discussed herein can be performed. Detailed Implementation

[0056] This document describes systems, apparatus, and methods for automatically planning ultrasound-guided tissue acquisition processes. Exemplary systems include an ultrasound imaging apparatus for generating ultrasound images of an anatomical target within a real-time ultrasound field of view (FOV) of a region of interest, a display for displaying the ultrasound images within the real-time ultrasound FOV, and controller circuitry. Before extending a tissue sampling device (e.g., a biopsy needle) into the real-time ultrasound FOV, the controller circuitry may display a graphical user interface element (UIE) indicating a predicted nominal sampling device trajectory intersecting with the anatomical target in the real-time ultrasound FOV. As the tissue sampling device is extended into the real-time ultrasound FOV, the controller circuitry may determine one or more real-time trajectory parameters based on analysis of the ultrasound images and update the UIE to represent the actual sampling device trajectory. The controller circuitry may display the updated UIE to provide visual feedback on whether the tissue sampling device will intersect with the anatomical target. The controller circuitry may determine the intersection probability and dynamically update the predicted sampling device trajectory and the intersection probability based on the determined one or more real-time trajectory parameters.

[0057] Figure 1 This is a schematic diagram illustrating an example of an echo-endoscopic system 100 used in an endoscopic ultrasound (EUS) procedure for diagnostic or therapeutic purposes, such as EUS-guided tissue acquisition. In this example, the echo-endoscopic system 100 can be configured to collect tissue samples from a target located deep within a peripheral region of the lung (e.g., primary, secondary, or tertiary bronchi or bronchioles) during an EBUS-TBNA procedure. The echo-endoscopic system 100 includes an echo endoscope 120, a light source device 130, a video processor 140, a first monitor 150 for displaying optical images, an ultrasound viewing device 160, and a second monitor 170 for displaying ultrasound images.

[0058] The echo endoscope 120 includes an insertion portion 111, an operating portion 112 extending therefrom the insertion portion 111, and a universal flexible cable 113 extending from the operating portion 112. The insertion portion 111 extends in the length direction and is configured for insertion into a living organism. The universal flexible cable 113 can be connected to a light source device 130 via a scope connector 113A located at its proximal portion. A coiled scope cable 114 and an ultrasound signal cable 115 extend from the scope connector 113A. An electrical connector portion 114A is provided at one end of the scope cable 114. The electrical connector portion 114A can be connected to a video processor 140. An ultrasound connector portion 115A is provided at one end of the ultrasound signal cable 115. The ultrasound connector portion 115A can be connected to an ultrasound observation device 160.

[0059] The insertion portion 111 of the echo endoscope 120 can be configured to sequentially connect the distal portion 121, the curved portion 122, and the flexible tube portion 123, starting from the distal end. The channel opening, optical observation window, optical illumination window, and ultrasonic transducer are arranged on one side of the distal portion 121, as shown in reference [reference needed]. Figure 2 Further description.

[0060] The operating section 112 may include a bend prevention section 124 extending from the insertion section 111, a channel opening section 125, an operating section body 126 constituting the holding section, a bendable operating section 127 including two bendable operating knobs 127A and 127B overlapping each other on one upper side of the operating section body 126, a plurality of switches 128 indicating the execution of various endoscopic functions, and a lifting rod 129 for operating the lifting platform. Examples of switches 128 include an air / water supply button, a suction button, and a freeze button.

[0061] A channel opening portion 125 is located on one side of the lower part of the operating portion body 126 and has one or more ports, each configured to receive a corresponding therapeutic instrument. By way of example and not limitation, two instrument ports 125A and 125B are located at the channel opening portion 125. Such instrument ports can communicate with two corresponding channel opening portions located at the distal portion 121 of the insertion portion 111 via two corresponding therapeutic instrument channels (not shown) inside the insertion portion 111. In the example, instrument port 125A can receive a tissue collection tool, such as a fine needle for EUS-guided tissue collection, such as EUS-guided fine needle aspiration (FNA) or fine needle biopsy (FNB). In one example, instrument port 125B can receive a cannula for endoscopic retrograde cholangiopancreatography (ERCP). A puncture needle handle portion Nh, shown by a single dotted line, is fitted into instrument port 125A.

[0062] Two instrument ports 125A and 125B can be arranged at the channel opening portion 125, such that when the operator moves their right hand RH closer to the channel opening portion 125, the instrument port closer to the right hand RH becomes instrument port 125B, and the instrument port further away from the right hand RH becomes instrument port 125A. More specifically, as... Figure 1As shown by the dotted lines, the operator holds the main body 126 of the operating section with their left hand (LH) while using their right hand (RH) to manipulate the therapeutic instruments inserted into each instrument port. Operation with therapeutic instruments such as ERCP cannulas is more challenging than operation with EUS-FNA puncture devices. Therefore, instrument ports 125B for therapeutic instruments such as cannulas, which require fine manipulation while the operator holds the main body 126 of the operating section with their left hand (LH), are positioned on the upper right side of the channel opening portion 125 compared to instrument ports 125A when viewed from the operator's perspective.

[0063] The bending knob 127A is a vertical bending knob, and the bending knob 127B is a horizontal bending knob. A bending fixing rod 127A1 for fixing the vertical bending state is provided on the proximal side of the bending knob 127A, and a bending fixing rod 127B1 for fixing the horizontal bending state is provided on the distal side of the bending knob 127B.

[0064] An image pickup unit for acquiring optical images of the interior of an object, and an illumination unit and an ultrasonic transducer unit for acquiring ultrasonic tomographic images of the interior of an object (see reference). Figure 2 The endoscope is positioned at the distal portion 121 of the echo endoscope 120. This allows the operator to insert the echo endoscope 120 into the object and allows the monitors 150 and 170 to display optical and ultrasonic tomographic images of the object's interior at the desired locations within the object, respectively.

[0065] Figure 2 As shown Figure 1 The diagram shows a perspective view of the distal portion 121 of the insertion portion 111 of the echo endoscope 120. The distal portion 121 may include a metallic distal rigid member 131 and a cylindrical synthetic resin covering member 132 therein into which the distal rigid member 131 is inserted, such that the covering member 132 partially covers the distal rigid member 131. An ultrasonic transducer portion 133 is housed within the distal portion 121. The ultrasonic transducer portion 133 may include an ultrasonic transducer configured to emit ultrasonic waves laterally at a predetermined angle relative to the insertion axis of the insertion portion 111. The ultrasonic transducer may have a linear array, curved, or phased array configuration. The cylindrical synthetic resin covering member 132 provides isolation for the distal portion 121 and allows the ultrasonic transducer portion 133 to be reliably secured therein.

[0066] When the cover member 132 is attached to the distal rigid member 131, a portion of the opening of the cylindrical cover member 132 is covered by a portion of the distal rigid member 131 on which the illumination window 141 and the optical observation window 142 are arranged. Optical light emitted from a light source, such as that located at the distal portion 121 of the echo endoscope 120 and coupled to the light source device 130, can pass through the illumination window 141 and be incident on the anatomical target and the surrounding environment. The optical observation window 142 allows an imaging device (e.g., a camera lens, not shown) at the distal portion 121 of the echo endoscope 120 to observe the target tissue. The remaining portion of the opening that is not covered by a portion of the distal rigid member 131 forms the opening 144, from which the lifting platform 151 protrudes when the lifting platform 151 is raised.

[0067] One or more diagnostic or therapeutic instruments can be activated to protrude from the opening portion 144. For example... Figure 2 As shown, when the lifting platform 151 is in its maximum lifting position, the instrument 240 can protrude from the opening 144. The instrument 240 can be inserted into one of the instrument ports on the operating portion 112 of the echo endoscope 120, such as instrument port 125A, through a channel within the echo endoscope 120, and protrude from the opening 144 at the distal portion 121 (see [reference]). Figure 1 Depending on the device used with instrument 240, it can be used to achieve different diagnostic or therapeutic goals. By way of example and not limitation, instrument 240 is a puncture device from which needle 242 protrudes. Needle 242 can be used to sample tissue from anatomical targets such as peripheral lung targets during EBUS-TBNA procedures. In addition to or as an alternative to puncture devices such as needle 242, other types of tissue sampling devices can be used and protrude from instrument 240, such as brushes, snares, forceps, aspiration devices, etc. Other devices that can be used with instrument 240 may include, for example, object retrieval devices for retrieving biological material (e.g., soft or hard tissue samples, cancerous tissue, or stone structures), resection devices for surgically removing tissue, diagnostic devices for performing in vivo analysis and diagnosis on sampled tissue, or therapeutic devices for delivering therapeutic agents (e.g., cancer treatment drugs) or different forms of energy (e.g., ultrasound, radiofrequency, laser, or thermal energy) to anatomical targets. Examples of EUS-guided treatment devices may include ablation devices, drainage devices (such as needles for draining pancreatic cysts or pseudocysts), or stenosis management devices for opening or dilating narrowed or blocked portions of catheters in the pancreatobiliary system.

[0068] In some examples, device 240 may accommodate an auxiliary insertion device, such as a needle, guidewire, catheter, probe, or cannula capable of endoscopic insertion and passage through the periphery of the lung during an EBUS-TBNA procedure. In some examples, device 240 may accommodate a needle-needle assembly comprising a needle and a needle insertable within the needle for manipulation (e.g., advance and retraction) and positioning of the needle during the procedure. Tissue sampling devices, needles, or other auxiliary insertion devices or combinations thereof used with device 240 may have different stiffnesses. In some examples, such devices may be configured with pre-shaped features, such as pre-shaped curvature or bending angles. The stiffness and pre-shaped configuration of such devices define the nominal needle trajectory that the respective device may follow within the ultrasound FOV.

[0069] In some examples, the echo endoscope 120 can be robotically controlled, such as via a robotic arm attached thereto. The robotic arm can automatically or semi-automatically (e.g., with a degree of user manual control or command) locate and navigate the instrument, such as the echo endoscope 120, within an anatomical target via actuators, or position the device in a desired posture to facilitate manipulation of the anatomical target (e.g., collecting tissue samples from the anatomical target using a brush, snare, forceps, or aspiration device). Depending on the various examples discussed herein, the controller can use artificial intelligence (AI) or machine learning (ML) techniques to determine navigation parameters and / or tool manipulation parameters (e.g., position, angle, posture, force, and navigation path) and generate control signals to the actuators of the robotic arm to facilitate manipulation of such an instrument or tool according to the determined navigation and manipulation parameters during robot-assisted processes.

[0070] Figure 3 An example of an endobronchial ultrasound-guided transbronchial fine needle aspiration (EBUS-TBNA) procedure is shown, along with a portion of the EBUS-TBNA system used in this procedure. EBUS-TBNA uses a dedicated bronchoscope 320 equipped with ultrasound capabilities to image outside the airway wall to detect the precise location of anatomical targets of interest, such as lymph nodes, in real time. During the procedure, the clinician inserts the bronchoscope 320 into the peripheral region 301 of the lung, such as primary, secondary, or tertiary bronchi or bronchioles. Similar to... Figure 2The echo endoscope 120 and bronchoscope 320 shown include an ultrasound transducer 333 that can be located at the distal end of the bronchoscope. The ultrasound transducer 333 can have a linear array, curved, or phased array configuration. The ultrasound transducer 333 can simultaneously generate a live ultrasound image stream of a target 302 (e.g., a peribronchial nodule) within a real-time ultrasound field of view (FOV) 350 in the region of interest. An intrabronchial sampling device 342 (e.g., a biopsy needle, such as needle 242, or a needle-core combination) can be inserted from a proximal port of the bronchoscope 320, traversing the entire length of the bronchoscope 320, and exiting from a side outlet port 344 (also referred to as a side outlet ramp) located at the distal end of the bronchoscope 320, slightly closer to the ultrasound transducer 333. Figure 2 (Example of an opening port 144) extends out. Then, the endobronchial sampling device 342 operatively protrudes from the side outlet port 344 and enters the ultrasound FOV 350. Under real-time ultrasound guidance, the endobronchial sampling device 342 can travel toward the target 302 (e.g., a peribronchial nodule) and eventually intersect with the target 302, and collect tissue samples from the target 302.

[0071] Figure 4 This is a block diagram illustrating an example of an EUS-Guided Tissue Acquisition (EUS-TA) planning system 400, which can automatically generate EUS-TA plans to navigate the sampling device during medical procedures such as EBUS-TBNA. The EUS-TA plan may include a path from a lateral exit port 344 at the distal portion of an endoscope (such as via a bronchoscope 320). Figure 3 The defined navigation path (also known as the nominal trajectory) extending into the anatomical target within the ultrasound FOV (field of view) is shown. The nominal trajectory can be initially predicted as the sampling device extends out of the endoscope but before entering the ultrasound FOV. In some cases, the nominal trajectory can be initially predicted even before the sampling device extends out of the endoscope. Under ultrasound guidance, a selected sampling device with a specific hardware configuration (e.g., pre-shaped curvature or bending angle) can be manually navigated by a clinician or robotically by a robotic system along the nominal trajectory and intersect (or miss) anatomical targets from which a sufficient amount of biopsy tissue has been sampled. As will be discussed further below, the nominal trajectory can be dynamically updated during the procedure to ensure that the sampling device will intersect with the anatomical target with a high probability.

[0072] System 400 may include controller circuitry 410, device controller 420, input interface 430, and user interface 440. Controller circuitry 410 may include a circuit set comprising one or more other circuits or sub-circuits that may individually or in combination perform the functions, methods, or techniques described herein. In an example, controller circuitry 410 and the circuit set therein may be implemented as part of a microprocessor circuit, which may be a special-purpose processor, such as a digital signal processor, application-specific integrated circuit (ASIC), microprocessor, or other type of processor for processing information including bodily activity information. Alternatively, the microprocessor circuit may be a general-purpose processor that can receive and execute a set of instructions for performing the functions, methods, or techniques described herein. In an example, the hardware of the circuit set may be immutably designed to perform a specific operation (e.g., hardwired). In an example, the hardware of the circuit set may include dynamically connected physical components (e.g., execution units, transistors, simple circuits, etc.) including computer-readable media that are physically modified (e.g., magnetically modified, electrically modified, movable placement of massless particles, etc.) to encode instructions for a specific operation. When connecting physical components, the fundamental electrical characteristics of the hardware composition change, for example, from insulator to conductor, and vice versa. Instructions enable embedded hardware (e.g., an execution unit or loading mechanism) to create components of a circuit set within the hardware via variable connections to perform specific operations during operation. Thus, when the device is operational, a computer-readable medium is communicatively coupled to other components of the circuit set component. In the example, any physical component can be used in more than one component of more than one circuit set. For example, under operation, an execution unit can be used at one point in time in a first circuit of a first circuit set and reused by a second circuit of the first circuit set or by a third circuit of a second circuit set at a different time.

[0073] The controller circuitry 410 can use image data received from the input interface 430 to generate an EUS-TA plan. In some embodiments, the input interface 430 may be a direct data link between the system 400 and one or more medical devices that generate at least some of the input features. For example, the input interface 430 may optionally send the EUS image 431 directly to the system 400 along with an endoscopic image 432, an external image source 433, or other information such as that collected by physiological sensors during the medical procedure. In some embodiments, the input interface 430 may be a part of the user interface 440 that facilitates interaction between the user and the system 400. For example, the user may manually provide input data to the system 400 via the user interface 440. In some examples, the input interface 430 may provide the system 400 with access to an electronic patient record from which one or more data features can be extracted. In any of these cases, the input interface 430 may collect patient information from one or more sources before and during the procedure.

[0074] EUS images 431 may include perioperative EUS images of the anatomical target and its surrounding environment during the EUS-guided procedure (e.g., a live stream of real-time EUS images). EUS images may be generated by an ultrasound transducer, such as an ultrasound transducer associated with echo endoscope 120 or an ultrasound transducer 333 associated with bronchoscope 320. Depending on the target's position relative to the ultrasound transducer, a real-time ultrasound field of view (FOV) of the target may be continuously generated and presented to the user via display 443 of user interface 440. Endoscopic images 432 may include perioperative endoscopic images or videos of the anatomical target and its surrounding environment captured by a camera device associated with an echo endoscope. External image sources 433 may include preoperative or perioperative images of the anatomical target acquired by external imaging devices other than an echo endoscope, which may include, for example, X-ray or fluoroscopic images, potential maps or impedance maps, CT images, or MRI images.

[0075] In addition to images, input interface 430 can receive other information, including, for example, information about endotherapy devices. This information includes, for example, the size, dimensions, shape, and structure of tissue sampling devices (e.g., needles, forceps, brushes, snares, knives, aspiration devices) or auxiliary insertion devices (e.g., core needles, guidewires, probes, catheters, or cannulas) to assist in the insertion and manipulation of the tissue sampling devices. The tissue sampling devices and / or auxiliary insertion devices may have pre-formed hardware configurations, such as pre-formed curvatures or bending angles. Device information (e.g., size, shape, and pre-formed configuration) of the tissue sampling devices and / or auxiliary insertion devices can be used to assist in selecting an appropriate tissue sampling device and determining device operating parameters to accurately and efficiently navigate the selected sampling device to the anatomical target. Device information can be received directly from input interface 430. Alternatively or additionally, device information can be stored in memory accessible to controller circuitry 410. In this example, input interface 430 may receive information from sensors coupled to or through a treatment device attached to an echo endoscope or through the endoscope, or otherwise associated with the patient. In the example, a proximity sensor located at the distal portion of the echo endoscope can sense information including the position, orientation, or proximity of the distal portion of the echo endoscope relative to the anatomical target.

[0076] The controller circuitry 410 may include an image processor 411, a target recognition and localization circuitry 412, a trajectory prediction circuitry 413, a sampling device tracker 414, and a trajectory evaluation and adjustment circuitry 415. The image processor 411 can process images received from the input interface 430 and extract image features characterizing an anatomical target of interest (e.g., a peribronchial nodule). The target recognition and localization circuitry 412 can use the extracted image features to detect the presence and location of an anatomical target within the real-time ultrasound field of view (FOV). The location of the anatomical target can be represented by its longitudinal position and depth relative to the surface of the ultrasound transducer in a coordinate system within the ultrasound of the FOV. In this example, a user (e.g., a clinician) can provide input for identifying the anatomical target via the input unit 445 of the user interface 440, such as by highlighting or clicking on the anatomical target displayed on the display 443, or by drawing a bounding box around the anatomical target on the display 443. The target recognition and localization circuitry 412 can analyze the image to determine the location of the anatomical target identified by the user. In some examples, the target recognition and localization circuit 412 may use real-time in vivo tissue diagnostics to determine additional characteristics of the anatomical target, including, for example, the size, shape, or structure of the anatomical target, and / or to identify pathophysiological properties such as lesions, inflammatory states, level of stenosis, or malignancy of the anatomical target (e.g., the extent or area of ​​cancer invasion). For example, the target recognition and localization circuit 412 may perform image processing techniques, such as edge detection operations, to identify the boundary edges of isolated lung nodules (or other abnormalities of the target of interest) within a real-time ultrasound image stream.

[0077] Trajectory prediction circuit 413 can extend an endoscope (e.g., via a sampling device (e.g., a biopsy needle or needle-core combination)) from a sampling device (e.g., via a biopsy needle or needle-core combination). Figure 3 When the bronchoscope 320 is shown at its side outlet port 344, but before the sampling device enters the ultrasonic FOV or even before the sampling device extends out of the endoscope, the nominal sampling device trajectory (S) is predicted. Nominal trajectory S Initial predictions can be made, at least in part, based on the detected target position in the real-time ultrasonic FOV, such that the predicted nominal trajectory S... It begins at the entry point into the real-time ultrasound FOV and intersects with the anatomical target at the detected target location visible in the real-time ultrasound FOV. In some examples, other information, such as the characteristics of the anatomical target determined by the target recognition and localization circuit 412, can also be used to predict the nominal trajectory S. In the example, the trajectory prediction circuit 413 may also predict the nominal trajectory S based on the posture and puncture angle of the sampling device (e.g., a needle) when piercing the airway wall, or the posture and entry angle when entering the real-time ultrasound FOV. In some examples, the nominal trajectory S can be predicted additionally or alternatively based on the hardware configuration of the available tissue sampling device (e.g., pre-formed curvature or bending angle). Therefore, the determined predicted nominal trajectory S Corresponding to tissue sampling devices X with specific pre-forming configurations such as pre-forming curvature or bending angle. .

[0078] In some examples, trajectory prediction circuit 413 can further determine the predicted nominal trajectory S in the real-time ultrasonic FOV. The upper limit of the trajectory (UTL) and the predicted nominal trajectory S The lower limit of the trajectory (LTL) is defined below. The UTL and LTL define the range or “region” within which intersection with the anatomical target is likely to occur with a sufficiently high confidence level (e.g., 95%). The UTL and LTL can each be determined based on uncertainties associated with the determination of device posture and puncture angle during airway wall penetration and / or device posture and entry angle during entry into the real-time ultrasound FOV. For example, in some cases, the nominal trajectory S... In one example, the initial prediction can be based on the axis of the side exit ramp 344 (where the sampling needle extends out of the endoscope relative to the FOV) or in another example, based on a population-based estimate of the puncture angle (e.g., before the sampling device enters the ultrasound FOV, or even before the sampling device extends out of the endoscope). For example, the nominal trajectory S can be initially determined based on the relative orientation of the side exit ramp relative to the FOV before the needle enters the FOV. However, when the sampling device punctures the lung membrane or airway wall, a slight deflection may occur, which can introduce a bias in hardware-based or population-based estimations of the puncture angle, thus introducing a deviation from the predicted nominal trajectory S. Associated uncertainties. UTL and LTL can also be determined based on uncertainties associated with measurements of the target location from images in a real-time ultrasound FOV.

[0079] In some cases, the trajectory prediction circuit 413 can use process data collected from the patient population to predict the nominal sampling device trajectory S. For example, the nominal sampling device trajectory S This can represent a best-fit line or curve, obtained, for example, from linear or nonlinear regression analysis of needle trajectories collected in a real-life or modeling setting from similar procedures performed on a selected patient population under similar conditions (e.g., similar type or model of bronchoscopy). For example, an ultrasound image stream can be collected from N (e.g., 1000) procedures where the needle extends from a lateral exit bevel into patient tissue within the FOV. UTL 511 and LTL 512 can represent the amount of variation in the trajectory based on the population, away from the nominal needle trajectory 510. For example, UTL 511 and LTL 512 can be determined based on the variance of the needle puncture angle in similar procedures performed on a selected patient population (e.g., UTL 511 and LTL 512 can be determined based on the standard deviation between procedures).

[0080] Including the prediction of the nominal trajectory S Graphical representation and S A graphical user interface element indicating the intersection with the anatomical target can be overlaid on the real-time ultrasound FOV and displayed on the monitor 443 during the sampling process. The UIE may additionally include graphical representations of the UTL and LTL overlaid on the real-time ultrasound FOV. In some examples, such a UIE may be displayed before the tissue sampling device extends into the real-time ultrasound FOV, or even before the sampling device extends beyond the endoscope. The UIE can be updated continuously or periodically as the tissue sampling device extends into the ultrasound FOV and moves toward the anatomical target. Such a UIE can serve as real-time feedback to the user regarding the status of the sampling device, and as a predictive nominal trajectory S for the sampling device tracking. Indicators of the degree of goodness and the likelihood of intersection with the anatomical target. Below are some... Figures 5A to 5B , Figures 6A to 6B and Figures 7A to 7B Further description of the indication predicts the nominal trajectory S And examples of UIEs for associated UTL and LTL.

[0081] The sampling device tracker 414 can track the actual state and trajectory of the sampling device as it enters the real-time ultrasound FOV. In this example, the sampling device tracker 414 can determine one or more real-time trajectory parameters based on analysis of ultrasound images or image features generated by the image processor 411. Examples of real-time trajectory parameters may include the sampling device's position, orientation, or direction or angle of travel within the real-time ultrasound FOV. The actual tracked trajectory (S) up to time t (and before intersection with the target) is... t The graphical representation can be overlaid on the real-time ultrasound FOV and displayed on the monitor 443 during the sampling process.

[0082] The trajectory evaluation and adjustment circuit 415 can evaluate the trajectory tracking performance during the process before the intersection occurs. In this example, the tracking performance can be quantified by a "hit probability," which can be estimated using a hit probability estimator 416. St The hit probability represents the actual tracking trajectory (S) given one or more real-time trajectory parameters and / or determined by the sampling device tracker 414. t The conditional probability of the sampling device intersecting the anatomical target is given in the case of [missing information]. In the example, the hit probability estimator 416 can be based on the actual tracking trajectory S relative to the target position within the real-time ultrasound FOV. t Or one or more real-time trajectory parameters to estimate the hit probability P St When the actual tracking trajectory S t With the predicted nominal trajectory S When the corresponding parts are different, the estimated hit probability can be lower than S. t Basically conforms to S The corresponding part of the situation. In the example, the hit probability estimator 416 can be based on the actual tracking trajectory (S t ) and the predicted nominal trajectory S The hit probability P is estimated by the difference level between corresponding parts. St In another example, the hit probability estimator 416 can estimate the hit probability P based on the distance between the current position of the sampling device (which is one of the real-time trajectory parameters generated by the sampling device tracker 414) and the target position within the real-time ultrasonic FOV. St In some examples, the hit probability estimator 416 can also estimate the hit probability P based on the homogeneity of the medium in the region of interest within the FOV. St .

[0083] Based on trajectory tracking performance (e.g., estimated hit probability P) St The controller circuit 410 can generate feedback or suggestions to adjust subsequent tracking strategies to ensure a high probability that the sampling device will intersect with the dissected target. The estimated hit probability P St This can be displayed on the user interface. In one example, when the hit probability is greater than the probability threshold P... TH At this point, the trajectory tracking performance is considered satisfactory, and the user can be notified (e.g., via output unit 442) to continue along the existing predicted nominal trajectory S. Advance the sampling equipment. When the hit probability P St Below the probability threshold P THAt this point, the trajectory tracking performance is considered unsatisfactory, and the controller circuit 410 can provide suggestions to the user (e.g., via output unit 442) to adjust the position or attitude of the sampling device relative to the target location within the FOV, such as withdrawing the sampling device from its current position by a specific amount. In some examples, the UIE (e.g., the actual tracked trajectory S) t Different visual indicators, such as different colors, can be used to distinguish the value or range of hit probabilities. For example, on a user interface, if P... St <P TH The actual tracking trajectory (S) t ) can be displayed in red, or if P St ≥P TH Then it can be displayed in green.

[0084] The trajectory evaluation and adjustment circuit 415 may include a trajectory update circuit 417, which may be based on one or more real-time trajectory parameters generated by the sampling device tracker 414 or the actual track S. t With the predicted nominal trajectory S The difference level between corresponding parts is used to dynamically update the predicted nominal trajectory S. In the example, this can be in response to trajectory tracking performance meeting specific conditions—such as estimating the hit probability P. St Below the probability threshold P TH —Initiate an update. The trajectory update circuit 417 can dynamically update and predict the nominal trajectory S. One or more associated UTLs or LTLs. As mentioned above, the UTL and LTL define a trajectory range within which intersections with the anatomical target may occur with a specific confidence level. Updates to the UTL and LTL can be performed by satisfying specific conditions, such as P. St <P TH The trajectory tracking performance is triggered. The controller circuit 410 can update the UIE to represent the sampling device trajectory (S). The updated predictions for ') and the updated UTL' and updated LTL'. All (S Some of the 'UTL', 'LTL', and 'LTL' can be superimposed on the real-time ultrasound FOV. The original predicted nominal trajectory S The associated UTL and LTL can be erased from the FOV, or with the updated counterpart (i.e., (S The nominal trajectory S, UTL, and LTL are displayed distinguishably, for example, in different colors. Such updates to the associated UTL and LTL are "dynamic" because they can occur continuously, periodically, or on demand as the sampling device moves toward the anatomical target, but before intersecting with it. Examples are given below regarding... Figures 7A to 7B and Figures 8A to 8C Examples describing updated UIEs include nominal trajectories (S The updated predictions of ')' and the associated UTL' and LTL'.

[0085] In various examples, one or more of the trajectory prediction circuit 413, the hit probability estimator 416, or the trajectory update circuit 417 may use artificial intelligence (AI) or machine learning (ML) techniques to perform the corresponding tasks as described above. One or more ML models may provide system 400 with the ability to perform tasks by making inferences based on patterns discovered in data analysis, without being explicitly programmed. ML model exploration involves the research and construction of algorithms (e.g., ML algorithms) that can learn from existing data and make predictions on new data. Such algorithms operate by building ML models from training data to make data-driven predictions or decisions represented as outputs or evaluations.

[0086] ML models can be trained using either supervised or unsupervised learning. Supervised learning uses prior knowledge (e.g., examples that associate inputs with outputs or outcomes) to learn the relationship between inputs and outputs. The goal of supervised learning is to learn a function that, given some training data, best approximates the relationship between the training inputs and outputs, so that the ML model can achieve the same relationship to generate the corresponding output given the input. Unsupervised learning is the training of an ML algorithm using information that is neither classified nor labeled, enabling the algorithm to act on that information without guidance. Unsupervised learning is useful in exploratory analytics because it can automatically identify structures in the data.

[0087] Common tasks in supervised learning are classification and regression problems. Classification problems, also known as categorization problems, aim to classify items into one of several class values. Regression algorithms aim to quantify some items (e.g., by assigning scores to some input values). Some examples of commonly used supervised ML algorithms are logistic regression (LR), Naive Bayes, random forest (RF), neural networks (NN), deep neural networks (DNN), matrix factorization, and support vector machines (SVM). Examples of DNNs include convolutional neural networks (CNN), recurrent neural networks (RNN), deep belief networks (DBN), or hybrid neural networks that include two or more neural network models of different types or different model configurations.

[0088] Some common tasks in unsupervised learning include clustering, representation learning, and density estimation. Examples of commonly used unsupervised learning algorithms include K-means clustering, principal component analysis, and autoencoders.

[0089] Another type of machine learning is federated learning (also known as collaborative learning), which trains algorithms on multiple distributed devices that store local data without exchanging data. This approach contrasts with traditional centralized machine learning techniques that upload all local datasets to a single server, as well as more classic distributed methods that typically assume a similar distribution of local data samples. Federated learning enables multiple participants to build general, robust machine learning models without sharing data, thus enabling the resolution of critical issues such as data privacy, data security, data access permissions, and access to heterogeneous data.

[0090] As an example and not a limitation, the trained ML model may have a neural network structure including an input layer, one or more hidden layers, and an output layer. Information fed to the input layer may include image data or image features derived therefrom generated by the image processor 411, the location and / or characteristics of the anatomical target generated by the target recognition and localization circuit 412, the actual device trajectory detected by the sampling device tracker 414, etc. The input information may propagate through one or more hidden layers to the output layer, which outputs a predicted nominal trajectory S. And associated UTL or LTL, hit probability P St or updated predicted nominal trajectory (S )' and one or more of the associated updated UTL' and LTL', etc.

[0091] In some examples, multiple ML models can be trained, validated, and used independently in different applications (during the inference phase). For instance, a first ML model (or a first set of ML models) can be trained to predict or update S. A second ML model (or a set of second ML models) can be trained to predict or update the UTL or LTL, and a third ML model (or a set of third ML models) can be trained to estimate the hit probability P. St .

[0092] Device controller 420 can generate control signals to one or more actuators 450, such as motors actuating a robotic arm. One or more actuators 450 can be coupled to one or more devices, such as a nominal sampling device (or an auxiliary insertion device to facilitate the insertion and manipulation of the sampling device, EUS probe, or echo endoscope 120 or bronchoscope 320). In response to the control signals, one or more actuators 450 can predict a nominal trajectory S based on the (initial) forecast. or updated predicted nominal trajectory (S The robot adjusts the position, posture, orientation, and navigation path of the devices coupled to it.

[0093] User interface 440 may include output unit 442 and input unit 445. Input unit 445 may receive input from a user or other data source. In this example, input interface 430 may be included in input unit 445. Output unit 442 may include display 443 to display an ultrasound field of view (FOV) containing images of the anatomical target and real-time ultrasound images of the tracking trajectory, as well as information on the position and motion of the sampling device during the procedure, such as advancing toward or retracting from the target anatomical structure within the ultrasound FOV. The user can adjust display settings via input unit 445. Visual indicators may take the form of markers, annotations (icons, text, or graphics), highlights, animations, or other visual indicators. For example, markers of different shapes, colors, forms, or sizes may be displayed on the reconstructed or integrated images to distinguish different tissues, anatomical regions, their accessibility, or criticality.

[0094] Output unit 442 may include alarm and feedback generator 444, which can generate alarms, notifications, or other forms of human-perceptible feedback to the operator regarding the status or progress of the cannula or navigation, with reference to the navigation plan. For example, an alarm may be generated to indicate the risk of tissue damage. Feedback may be one or more of audio, visual, or tactile feedback. For example, when the endoscope tip enters or approaches a “critical zone” (e.g., a proximity sensor detects that the distance to a critical anatomical structure of interest is less than a threshold distance), the critical zone may be shown in different colors to indicate such a distance (e.g., green, yellow, and red zones shown as the endoscope approaches the critical zone). Additionally or alternatively, tactile feedback such as touch or vibration may be generated and felt by the operator. In the example, alarm and feedback generator 444 may automatically adjust the vibration intensity based on the distance to the critical zone. For example, a low vibration may be generated when the endoscope tip is in the green zone. If the system predicts, based on the endoscope's current forward speed and direction, that the endoscope tip will reach the critical area within a predetermined threshold time, the alarm and feedback generator 444 can apply moderate vibration when the endoscope tip reaches the yellow zone and high vibration when the endoscope tip reaches the red zone to indicate the risk of tissue damage. Real-time alarms and feedback in image-guided medical procedures, as described herein, can improve the efficiency of cannula and endoscope navigation, especially for inexperienced physicians, and can improve procedure success rates and patient outcomes.

[0095] Figures 5A to 5B , Figures 6A to 6B and Figures 7A to 7BA schematic diagram of an exemplary EBUS-TBNA system with an associated computing system is shown. This computing system determines and displays graphical representations of one or more nominal sampling device trajectories and user interface elements (UIEs) under different operating states of an exemplary operational flow during the EBUS-TBNA process. The graphical representations of one or more nominal sampling device trajectories and UIEs can be overlaid onto ultrasound images in a real-time ultrasound field of view (FOV). Figure 5A As shown, the EBUS-TBNA system includes an EBUS sampling device 520 located within an airway surrounded by airway walls 501. During the EBUS-TBNA procedure, a predicted sampling device trajectory 510 can be displayed on an ultrasound image of the peripheral lung region in a real-time ultrasound FOV 550. The real-time ultrasound FOV 550 can be generated by an ultrasound transducer 533 located at the distal end of the EBUS sampling device 520. The EBUS sampling device 520 has a side outlet port 544 (also referred to as a needle bevel) near the ultrasound transducer 533 along the length of the tubular device body of the EBUS sampling device 520. During EUS-TA procedures such as the EBUS-TBNA procedure, such as Figure 3 The tissue sampling device (not shown here) of the endobronchial sampling device 342 shown can pass through the lumen of the EBUS sampling device 520 and extend from the side outlet port 544, puncturing the patient's airway wall 501 to enter the ultrasound FOV 550, and proceeding toward and ultimately reaching the target nodule 540, which is an isolated peribronchial nodule such as the desired biopsy sample. The FOV 550 can be displayed to the clinician on a monitor throughout the tissue sampling process.

[0096] Figure 5A , Figure 6A and Figure 7A Each is illustrated with aspects of the UIE within the real-time ultrasound FOV 550, as well as the EBUS sampling device 520, airway, and target nodule 540. The UIE is a computer-generated graphical element displayed as an overlay on the ultrasound image within the real-time ultrasound FOV 550 to indicate possible needle trajectories associated with the target nodule 540 visible within the ultrasound image. Aspects of the UIE may include a graphical representation of the nominal needle trajectory 510, the upper limit of trajectory (UTL) 511 above the nominal needle trajectory 510, and the lower limit of trajectory (LTL) 512 below the nominal needle trajectory 510. The nominal needle trajectory 510 and the associated UTL 511 and LTL 512 can be predicted by trajectory prediction circuitry 413. However, it should be understood that in a real-world implementation, such a UIE will be synthesized (i.e., overlaid / covered) on the real-time ultrasound image displayed to the clinician during the sampling process, such as... Figure 5B , Figure 6B and Figure 7BEach of the illustrations depicted in the diagram is shown in the diagram.

[0097] Figures 5A to 5B A first operational state of an exemplary operational flow associated with the use of the EUS-TA planning system 400 according to an embodiment as described herein is illustrated. In this state, no sampling device (e.g., sampling needle) extends from the side outlet ramp 344. Therefore, the EBUS sampling device 520 can move forward and / or backward within the airway surrounded by the airway wall 501 to align the target nodule 540 with the nominal needle trajectory 510. In the example, the nominal needle trajectory 510 may be an extension of the axis of the side outlet ramp 344, or any other trajectory that a needle leaving the ramp is most likely to follow as it extends. In some examples, the nominal needle trajectory 510 may represent a best-fit line or curve of needle trajectory samples collected from a similar procedure performed on a selected patient population in a real-life or modeling setting. For example, ultrasound image streams of N procedures (e.g., 1000 procedures) can be collected, where the needle extends from the lateral exit ramp into the patient tissue within the real-time ultrasound FOV 550, and the nominal needle trajectory can represent a best-fit line or curve, obtained, for example, based on linear or nonlinear regression analysis or the average of all collected data. UTL 511 and LTL 512 can represent some predetermined amount of variability (e.g., standard deviation) away from the nominal needle trajectory 510. (See above regarding...) Figure 4 The nominal needle trajectory 510 can also depend on the sampling device configuration (e.g., a combination of needle and core needle). Therefore, the displayed trajectory can also depend on the selected sampling device and can be updated using different options from clinicians.

[0098] In some examples, the nominal needle trajectory 510 and the associated UTL 511 and LTL 512 can be determined as a whole and are not specific to a single procedure. Therefore, the UIE indicating the nominal needle trajectory 510 and the associated UTL 511 and LTL 512 can be displayed before the sampling needle extends into the tissue or even before it is inserted into the proximal end (outside the patient) of the EBUS sampling device 520. For example, these UIEs can be displayed while the user is still free to slide the EBUS sampling device along the airway and / or select different combinations of needles and cardioid needles. Thus, the UIE displayed in this procedure state can serve as a useful visual aid for the user to align the target nodule 540 with the nominal needle trajectory 510.

[0099] Figures 6A to 6B The second operational state of an exemplary operational flow associated with the use of the EUS-TA planning system 400 according to the implementation described herein is shown. (As above regarding...) Figures 5A to 5BThe second operating state can occur immediately following the first operating state. In the second state, the user can slightly extend the sampling needle 642 from the side outlet 544 (e.g., Figure 3 (Example of the endobronchial sampling device 342 shown), but not extending to the point where the sampling needle 642 has entered the real-time ultrasound FOV 550. Therefore, the sampling needle 642 is not yet visible in the ultrasound image displayed to the user. In this state, the nominal needle trajectory 510 and the associated UTL 511 and LTL 512 can remain unchanged because the absence of the sampling needle 642 within the real-time ultrasound FOV 550 indicates that the system does not have real-time needle trajectory information that can be used to determine whether to update the nominal needle trajectory 510 or the associated UTL 511 and LTL 512.

[0100] Figures 7A to 7B The third operational state of an exemplary operational flow associated with the use of the EUS-TA planning system 400 according to the implementation described herein is shown. (As above regarding...) Figures 6A to 6B The third operating state can occur immediately following the second operating state. In this second state, the sampling needle 642 has entered the real-time ultrasound FOV 550. The actual needle trajectory can be tracked by the sampling device tracker 414, and one or more real-time needle trajectory parameters can be derived from the image. Tracking performance can be evaluated. Based on the tracking performance, the nominal needle trajectory 510 and the associated UTL 511 and LTL 512 can be dynamically updated. For example, when the sampling needle 642 becomes visible within the real-time ultrasound FOV 550, the trajectory evaluation and adjustment circuitry 415 can analyze the ultrasound image to distinguish the needle from the tissue and determine the real-time needle trajectory parameters, which may include, for example, the height of the needle within the FOV, the angle of the needle within the FOV, the amount and / or direction of the needle curvature, and any other parameters that can have some predictive value for the trajectory that the sampling needle 642 may follow. Then, based on the real-time needle trajectory parameters, the hit probability estimator 416 can estimate the hit probability. When the estimated hit probability is below a probability threshold, the trajectory update circuit 417 can update one or more of the nominal needle trajectory 510 or the associated UTL 511 and LTL 512. In this way, if the sampling needle 642 enters the real-time ultrasound FOV 550 and advances in a manner indicating that the sampling needle 642 may deviate from the original nominal needle trajectory 510 (potentially leading to a lower hit probability), a visual indication for this purpose can be provided to the user. (See below.) Figure 7B As shown, the updated UTL 711 and LTL 712 are narrower than the original displayed UTL 511 and LTL 512, even though the updated nominal pin track 710 is not significantly different from the original nominal pin track 510 (therefore...). Figure 5AThe label in the text is “510 (710)”. This is because when the sampling needle 642 travels through the tissue toward the target nodule 540, the possible trajectories within the FOV converge around the actual needle trajectory being followed.

[0101] In some implementations, the system may receive indications of where the target tissue is located within the field of view (FOV) and may use these indications to further enhance the visual aids presented to the clinician (e.g., UIE). For example, the system may receive user input indicating the location of the target nodule relative to the FOV. Additionally or alternatively, the system may perform image processing techniques to automatically identify a portion of the ultrasound image that possesses characteristics indicative of the target tissue (e.g., cancerous lesions, solitary pulmonary nodules (SPNs), etc.).

[0102] Figures 8A to 8C This is a graph showing the UIE, displayed as coverage on a real-time ultrasound FOV of 550. It indicates the probability of hits associated with the determination and dynamic updating of the nominal needle trajectory and associated UTL and LTL under various operating conditions, as described above regarding... Figures 5A to 5B , Figures 6A to 6B and Figures 7A to 7B As stated above. Figure 8A The diagram illustrates that, prior to the sampling needle 642 entering the FOV (corresponding to the first and second operating states) and after the system has received some indication of the target nodule 540's position within the real-time ultrasound FOV 550, if the sampling needle 642 follows the original nominal needle trajectory 510, the system determines that the probability of a hit on the target nodule 540 is 74%. Such an initial hit probability (P0) can be determined based on, for example, the distance the sampling needle 642 would need to travel to reach the target nodule 540, the homogeneity of the tissue within the FOV, or any other factor that could indicate the likelihood of the sampling needle 642 deviating from the original nominal needle trajectory 510. In some examples, the initial hit probability (P0) can be estimated using data collected from similar procedures performed on a selected patient population under similar conditions (e.g., with a similar type or model of bronchoscopy). UIE 810 can be displayed along with such an initial estimate of the hit probability.

[0103] In some examples, to ensure a satisfactory high P0 (e.g., above 95%) associated with the nominal needle trajectory 510, the user can advance, retract, or rotate the EBUS sampling device 520 (or the EUS probe, if operable separately from the EBUS sampling device 520) to change the position of the target nodule 540 when it appears in the real-time ultrasound FOV 550. Since the initial hit probability P0 can be affected by the sampling device hardware configuration and the puncture angle relative to the target location, changing the position of the bronchoscope or EUS probe relative to the target can result in different P0 values. In the examples, the system can estimate P0 = 10% at the first bronchoscope or EUS probe position, P0 = 40% at the second bronchoscope or EUS probe position, ..., P0 = 95% at the Nth bronchoscope or EUS probe position, P0 = 50% at the (N+1)th bronchoscope or EUS probe position, and so on. Each estimate can be made using data collected from similar procedures performed on a selected patient population under similar conditions (e.g., similar type or model of bronchoscope, and the same position of the bronchoscope or EUS probe relative to the corresponding target during the procedure). An “optimal point” corresponding to the highest or most satisfactory P0 value can be identified from the position of the bronchoscope or EUS probe that has been tested. The bronchoscope or EUS probe can then be positioned at the “optimal point,” and the sampling needle can extend from the side outlet port 544 and puncture into the tissue.

[0104] Figures 8B to 8C The diagram shows a second time corresponding to the third operating state, at which point the sampling needle 642 has entered the real-time ultrasound FOV 550 and is traveling approximately one-third of the original nominal needle trajectory 510 toward the target nodule 540. Figure 8B An example is shown where the sampling needle 642 has been kept very closely aligned with the original nominal needle trajectory 510 (e.g., the updated nominal needle trajectory 710 is almost aligned with the original nominal needle trajectory 510, hence labeled "510 (710)"), and the trajectory evaluation and adjustment circuitry 415 has determined an increase in the hit probability 820 of up to 95%. The UIE can be updated to reflect this new estimate of the hit probability. In some examples, the updated UIE can be distinguishably displayed on top of the previous UIE, such as through coloring (e.g., green) to indicate the determined "feasible" or "hit" scenario for the current sampling attempt. Figure 8C An example is shown where the sampling needle 642 deviates substantially from the original nominal needle trajectory 510 or diverges substantially away from the target nodule 540. The trajectory evaluation and adjustment circuitry 415 can determine the reduced probability of hit 830 as 14%. The updated UIE can include such a reduced probability of hit value, optionally indicated by a different color (e.g., yellow or red) to indicate the lower likelihood of hitting the target.

[0105] Figure 9 This is a flowchart illustrating an example method 900 for generating an EUS-TA plan and presenting visual aids on a graphical user interface to assist in the navigation of tissue sampling devices during medical procedures such as EBUS-TBNA. Method 900 can be implemented in and executed by system 400. Although the processes of method 900 are depicted in a flowchart, they do not need to be executed in a specific order. In various examples, some processes may be executed in a different order than shown.

[0106] At step 910, during an EUS-TA procedure such as an EBUS-TBNA procedure, an ultrasound transducer associated with an endoscope (such as an ultrasound transducer associated with an echo endoscope 120 or an ultrasound transducer 333 associated with a bronchoscope 320) can be used to generate an ultrasound image of the anatomical target. Depending on the position of the anatomical target relative to the ultrasound transducer, a real-time ultrasound field of view (FOV) of the target can be presented to the user on a graphical user interface. In addition to the ultrasound image, other images can be generated or otherwise received at 910, including, for example, perioperative endoscopic images or videos of the anatomical target and its surrounding environment captured by a camera device associated with an echo endoscope, or preoperative or perioperative images of the anatomical target acquired by external imaging equipment, such as X-ray or fluoroscopic images, potential maps or impedance maps, CT images, or MRI images.

[0107] At step 920, before extending the tissue sampling device (e.g., a biopsy needle) into the real-time ultrasound FOV, the predicted nominal sampling device trajectory S can be determined. It can also display the predicted trajectory S on a graphical user interface. A graphical user interface element (UIE) intersecting with the anatomical target on the real-time ultrasound field of view (FOV). Predicting the nominal trajectory S It begins at the entry point of the real-time ultrasound FOV and intersects with the anatomical target at the detected target location within the real-time ultrasound FOV. Nominal sampling device trajectory S The prediction can be based at least in part on the position of the anatomical target in the real-time ultrasound field of view (FOV), which can be detected by the target recognition and localization circuit 412 using image features extracted from the ultrasound image received at step 910. In predicting the nominal trajectory S... Other factors to consider may include, for example, the orientation and puncture angle of the sampling device (e.g., a needle) when piercing the airway wall, or the orientation and entry angle when entering the real-time ultrasound field of view (FOV). In some examples, the nominal trajectory S may also be predicted based on the hardware configuration of the available tissue sampling device (e.g., pre-formed curvature or bending angle). Therefore, the determined predicted nominal trajectory S Corresponding to tissue sampling devices X with specific pre-forming configurations such as pre-forming curvature or bending angle. Therefore, the determined predicted nominal trajectory S Corresponding to tissue sampling devices X with specific pre-forming configurations such as pre-forming curvature or bending angle. In some other examples, process data collected from patient populations can be used to predict the nominal sampling device trajectory S. Nominal sampling equipment trajectory S This can represent a best-fit line or curve, which is obtained, for example, from linear or nonlinear regression analysis of needle trajectories collected in a real-life or modeling setting from similar procedures performed on a selected patient population under similar conditions (e.g., similar type or model of bronchoscopy).

[0108] In addition to the nominal sampling equipment trajectory S In addition to the prediction, the predicted nominal trajectory S can be determined at step 920. The upper limit of the trajectory (UTL) and the predicted nominal trajectory S The lower limit of the trajectory (LTL) is below. The UTL and LTL define the trajectory range or "region" within which intersection with the anatomical target is likely to occur with a sufficiently high confidence level (e.g., 95%). The UTL and LTL can each be determined based on uncertainties associated with the determination of device posture and puncture angle during airway wall penetration and / or device posture and entry angle during entry into the real-time ultrasound FOV, as described above regarding... Figure 4 The above includes the prediction of the nominal trajectory S. and S Graphical user interface elements (UIEs) representing the indicators intersecting with the anatomical target, as well as the graphical representations of the UTL and LTL, can be overlaid on the real-time ultrasound FOV and displayed on the user interface during the procedure.

[0109] At step 930, as the tissue sampling device is extended into the real-time ultrasound FOV, one or more real-time trajectory parameters can be determined based on analysis of the ultrasound images. Examples of real-time trajectory parameters may include the position, orientation, or direction or angle of the sampling device within the real-time ultrasound FOV.

[0110] At step 940, the UIE appearing on the user interface can be updated based on one or more determined real-time trajectory parameters. The updated UIE may include the actual tracking trajectory (S) overlaid on the real-time ultrasonic FOV at a cutoff time t (and before intersecting with the target) and displayed on the user interface during the process. t The graphical representation of ).

[0111] In some examples, UIE updates can be triggered by trajectory tracking performance meeting specific conditions. The trajectory evaluation and adjustment circuit 415 can be used to evaluate trajectory tracking performance during the process prior to intersection. Tracking performance can be assessed via the hit probability P. St To quantify, the hit probability P St This indicates that given one or more real-time trajectory parameters and / or the actual tracking trajectory (S... t The conditional probability of the sampling device intersecting the anatomical target under certain conditions. In the example, this can be based on the actual tracking trajectory S relative to the target position within the real-time ultrasound field of view. t Or one or more real-time trajectory parameters to estimate the hit probability P St In the example, it can be based on the actual tracking trajectory (S). t ) and the predicted nominal trajectory S The hit probability P is estimated by the difference level between corresponding parts. St In another example, the hit probability P can be estimated based on the distance between the current position of the sampling device (which is one of the real-time trajectory parameters generated in step 930) and the target position within the real-time ultrasonic FOV. St In some examples, the hit probability P can also be determined based on the homogeneity of the medium within the region of interest within the FOV. St .

[0112] UIE updates can include updates based on one or more real-time trajectory parameters or the actual tracked trajectory S. t With the predicted nominal trajectory S The difference level between corresponding parts is used to dynamically update the predicted nominal trajectory S. In the example, this can be done in response to the estimated hit probability P. St Below the probability threshold P TH Initiate an update. Besides predicting the nominal trajectory S In addition to dynamic updates, it can also update and predict the nominal trajectory S One or more associated UTLs or LTLs. Updates to UTLs and LTLs can be performed by satisfying specific conditions, such as P. St <P TH The trajectory tracking performance is triggered. UIE updates can also include dynamic updates to the hit probability.

[0113] In some examples, one or more artificial intelligence (AI) or machine learning (ML) models can be trained separately to predict or update S. Predicting or updating UTL or LTL, or estimating the hit probability P St As mentioned above Figure 4 As described.

[0114] At step 950, the updated UIE (including the updated prediction of the sampling device trajectory (S)) can be... The original predicted nominal trajectory S is overlaid on the real-time ultrasound FOV with the updated UTL and updated LTL. This provides visual feedback on whether the tissue sampling device will intersect the anatomical target. The associated UTL and LTL can be erased from the FOV, or with the updated counterpart (i.e., (S The UTL and LTL are displayed, for example, in different colors. Such updates to the UIE are "dynamic" because they can occur continuously, periodically, or on demand as the sampling device moves toward the anatomical target but before intersecting with it.

[0115] Figure 10 A block diagram of an exemplary machine 1000 on which any or more of the techniques (e.g., methods) discussed herein can be generally shown. Parts of this description can be applied to the computational framework of various parts of the EUS-TA planning system 400, such as components 411 to 417 of the controller circuit 410.

[0116] In alternative implementations, machine 1000 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, machine 1000 may operate as a server machine, a client machine, or both in a server-client network environment. In the example, machine 1000 may act as a peer-to-peer (P2P) (or other distributed) network environment. Machine 1000 may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, network device, network router, switch, or bridge, or any machine capable of executing instructions (sequential instructions or other instructions) specifying actions to be taken by that machine. Furthermore, although only a single machine is shown, the term "machine" should also be considered as any collection of machines that individually or jointly execute a set (or more) of instructions to perform any or more of the methods discussed herein (such as cloud computing, Software as a Service (SaaS), other computer cluster configurations).

[0117] As described herein, examples may include logic or multiple components or mechanisms, or may be operated by logic or multiple components or mechanisms. A circuit set is a collection of circuits implemented in a tangible entity including hardware (e.g., simple circuits, gates, logic, etc.). Circuit set components can be flexible with time and the variability of the underlying hardware. A circuit set includes components that can perform a specified operation individually or in combination during operation. In the examples, the hardware of the circuit set may be immutably designed to perform a specific operation (e.g., hardwired). In the examples, the hardware of the circuit set may include variable-connected physical components (e.g., execution units, transistors, simple circuits, etc.), which include computer-readable media that are physically modified (e.g., magnetically modified, electrically modified, movable placement of massless particles, etc.) to encode instructions for a specific operation. When connecting physical components, the basic electrical properties of the hardware composition change, for example, from an insulator to a conductor, and vice versa. Instructions enable embedded hardware (e.g., an execution unit or loading mechanism) to create components of the circuit set in the hardware via variable connections to perform a specific operation during operation. Therefore, when the device is operational, the computer-readable medium is communicatively coupled to other components of the circuit set. In the example, any one of the physical components can be used in more than one component of more than one circuit set. For example, under operation, an execution unit can be used at one point in time in a first circuit of a first circuit set and reused by a second circuit of the first circuit set or by a third circuit of a second circuit set at a different time.

[0118] Machine (e.g., computer system) 1000 may include a hardware processor 1002 (e.g., a central processing unit (CPU), graphics processing unit (GPU), hardware processor core, or any combination thereof), main memory 1004, and static memory 1006, some or all of which may communicate with each other via an interconnect link (e.g., bus) 1008. Machine 1000 may also include a display unit 1010 (e.g., a raster display, vector display, holographic display, etc.), an alphanumeric input device 1012 (e.g., a keyboard), and a user interface (UI) navigation device 1014 (e.g., a mouse). In this example, display unit 1010, input device 1012, and UI navigation device 1014 may be a touchscreen display. Machine 1000 may additionally include a storage device (e.g., a drive unit) 1016, a signal generation device 1018 (e.g., a speaker), a network interface device 1020, and one or more sensors 1021, such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. Machine 1000 may include output controller 1028, such as serial (e.g., Universal Serial Bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection, to communicate with or control one or more peripheral devices (e.g., printers, card readers, etc.).

[0119] Storage device 1016 may include machine-readable medium 1022 on which one or more sets of data structures or instructions 1024 (e.g., software) are stored, said set of data structures or instructions 1024 being embodied or utilized by any one or more of the techniques or functions described herein. Instructions 1024 may also reside wholly or at least partially within main memory 1004, static memory 1006, or hardware processor 1002 during execution by machine 1000. In this example, one or any combination of hardware processor 1002, main memory 1004, static memory 1006, or storage device 1016 may constitute a machine-readable medium.

[0120] Although machine-readable medium 1022 is shown as a single medium, the term "machine-readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store one or more instructions 1024.

[0121] The term "machine-readable medium" can include any medium capable of storing, encoding, or carrying instructions for use by machine 1000 and causing machine 1000 to perform any one or more of the techniques disclosed herein, or any medium capable of storing, encoding, or carrying data structures used by or associated with such instructions. Examples of non-limiting machine-readable media can include solid-state memory as well as optical and magnetic media. In examples, mass machine-readable media includes machine-readable media having a massive number of particles that are constant (e.g., stationary). Therefore, mass machine-readable media are not transiently propagating signals. Specific examples of mass machine-readable media can include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory, electrically erasable programmable read-only memory (EPSOM)) and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0122] Instruction 1024 can also be used to send or receive on communication network 1026 via a transmission medium through network interface device 1020 using any of a variety of transmission protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Example communication networks may include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile phone networks (e.g., cellular networks), conventional telephone (POTS) networks, and wireless data networks (e.g., IEEE 802.11 series standards known as WiFi®, IEEE 802.16 series standards known as WiMax®), IEEE 802.15.4 series standards, peer-to-peer (P2P) networks, etc. In the example, network interface device 1020 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas to connect to communication network 1026. In the example, network interface device 1020 may include multiple antennas to perform wireless communication using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technologies. The term "transmission medium" should be considered to include any intangible medium capable of storing, encoding, or carrying instructions for execution by machine 1000, and includes digital or analog communication signals or other intangible media to facilitate communication of such software.

[0123] Additional notes

[0124] The above detailed description includes reference to the accompanying drawings, which form a part of the detailed description. The drawings illustrate specific embodiments in which the invention can be practiced. These embodiments are also referred to herein as “examples.” Such examples may include elements other than those shown or described. However, the inventors also contemplate examples in which only those elements shown or described are provided. Furthermore, the inventors envision examples using any combination or arrangement of those elements (or one or more aspects thereof) shown or described relative to a particular example (or one or more aspects thereof) or relative to other examples (or one or more aspects thereof) shown or described herein.

[0125] In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more, independent of any other instances or uses of “at least one” or “one or more.” In this document, unless otherwise stated, the term “or” is used to mean non-exclusive or, such that “A or B” includes “A but not B,” “B but not A,” and “A and B.” In this document, the terms “comprising” and “wherein” are used as simple English equivalents of the corresponding terms “comprising” and “wherein.” Furthermore, in the appended claims, the terms “comprising” and “including” are open-ended, meaning that a system, apparatus, article, composition, formulation, or treatment that includes elements other than those listed after such terms in the claims is still considered to fall within the scope of the claims. Additionally, in the appended claims, the terms “first,” “second,” and “third,” etc., are used merely as labels and are not intended to impose numerical requirements on their objects.

[0126] The above description is intended to be illustrative and not restrictive. For example, the examples above (or one or more aspects thereof) may be used in combination with each other. Other embodiments may be used by those skilled in the art upon review of the above description, for example. An abstract is provided to allow the reader to quickly determine the nature of the technical disclosure. It is understood that the abstract is not intended to interpret or limit the scope or meaning of the claims. Furthermore, in the detailed description above, various features may be grouped together to simplify the disclosure. This should not be construed as meaning that any disclosed feature not claimed is essential to any claim. Rather, the subject matter of the invention may lie in fewer than all features of a particular disclosed embodiment. Therefore, the appended claims are incorporated herein by way of example or embodiment, wherein each claim is an independent, separate embodiment, and such embodiments are contemplated to be combined with each other in various combinations or arrangements. The scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents.

Claims

1. A system for planning an ultrasound-guided tissue acquisition process, the system comprising: An ultrasound imaging device configured to generate an ultrasound image of an anatomical target in a real-time ultrasound field of view (FOV) of a region of interest; A display configured to display the ultrasound image in the real-time ultrasound field of view (FOV); as well as The controller circuit is configured to: Before the tissue sampling device extends into the real-time ultrasound FOV, the following graphical user interface element (UIE) is displayed: the graphical user interface element (UIE) indicates the predicted sampling device trajectory intersecting with the anatomical target on the real-time ultrasound FOV; As the tissue sampling device extends into the real-time ultrasound field of view (FOV), one or more real-time trajectory parameters are determined based on the analysis of the ultrasound images; The UIE is updated based on one or more determined real-time trajectory parameters to represent at least one of the actual sampling device trajectory or the updated predicted sampling device trajectory; as well as The updated UIE is displayed on the real-time ultrasound FOV to provide visual feedback on whether the tissue sampling device will intersect with the anatomical target.

2. The system according to claim 1, wherein, The controller circuit is configured to: Determine the location of the anatomical target within the real-time ultrasound field of view; and The probability of the tissue sampling device intersecting the anatomical target is determined at least in part based on the identified target location within the real-time ultrasound FOV.

3. The system according to claim 2, wherein, To determine the location of the anatomical target, the controller circuit is configured to: Receive user input via a user interface to identify the anatomical target within the real-time ultrasound field of view; and Analyze the ultrasound images within the real-time ultrasound field of view (FOV) to determine the location of the anatomical target identified by the user.

4. The system according to claim 1, wherein, The one or more real-time trajectory parameters include one or more of the position, posture, direction of travel, or angle of entry of the tissue sampling device when it enters the real-time ultrasound FOV.

5. The system according to claim 4, wherein, The controller circuit is configured to determine the predicted sampling device trajectory based on the posture or entry angle of the tissue sampling device when it enters the real-time ultrasound FOV.

6. The system according to claim 1, wherein, The controller circuit is configured to: further determine the predicted sampling device trajectory based on the hardware configuration of the tissue sampling device, the hardware configuration including the pre-formed curvature or bending angle of the tissue sampling device.

7. The system according to claim 1, wherein, The controller circuitry is configured to use population data to determine at least one of the predicted sampling device trajectory or the updated predicted sampling device trajectory, the population data including sampling device trajectories collected from similar ultrasound-guided tissue acquisition procedures performed on a patient population.

8. The system according to claim 1, wherein, The controller circuit is configured to: Determine an upper and lower bound for the trajectory, which define a range within which intersection with the anatomical target is likely to occur with a specific confidence level; and Before the tissue sampling device extends into the real-time ultrasound FOV, a UIE indicating the determined upper and lower limits of the trajectory is displayed on the real-time ultrasound FOV.

9. The system according to claim 8, wherein, The controller circuitry is configured to use population data to determine the upper and lower limits of the trajectory, the population data including sampling device trajectories collected from similar ultrasound-guided tissue acquisition procedures performed on a patient population.

10. The system according to claim 1, wherein, The controller circuit is configured to determine, at least in part, the probability of the tissue sampling device intersecting the anatomical target based on the actual sampling device trajectory relative to the anatomical target within the real-time ultrasound FOV or the one or more real-time trajectory parameters.

11. The system according to claim 10, wherein, The controller circuit is configured to: In response to the hit probability being lower than a probability threshold, a suggestion is provided to withdraw the tissue sampling device or adjust the position or orientation of the tissue sampling device relative to the anatomical target within the real-time ultrasound FOV; as well as In response to the hit probability being greater than the probability threshold, a notification is provided to the user to extend the tissue sampling device according to the predicted sampling device trajectory.

12. The system according to claim 1, wherein, The controller circuit is also configured to: The predicted sampling device trajectory is dynamically updated, at least in part, based on one or more determined real-time trajectory parameters; and Update the UIE to represent the updated predicted sampling device trajectory.

13. The system according to claim 12, wherein, The controller circuit is also configured to dynamically update one or more of an upper or lower trajectory limit, the upper and lower trajectory limits defining a trajectory range within which an intersection with the anatomical target may occur with a specific confidence level.

14. The system according to claim 12, wherein, The controller circuit is also configured to determine an updated hit probability of the tissue sampling device intersecting the anatomical target as it travels along the updated predicted sampling device trajectory.

15. The system according to claim 1, wherein, The ultrasound imaging device is located distal to the endobronchial sampling device, which is configured to be inserted into the patient's airway during an ultrasound-guided transbronchial fine needle aspiration (EBUS-TBNA) procedure.

16. An intraluminal imaging device system, comprising: An intraluminal imaging device, comprising: a tubular device body having an inner lumen and side outlet ports; and an ultrasonic transducer configured to generate an ultrasonic scan of an anatomical target and to generate an ultrasonic image in a real-time ultrasonic field of view (FOV) of a region of interest. A display configured to display the ultrasound image in the real-time ultrasound field of view (FOV); A tissue sampling device configured to: operably pass through the lumen under ultrasound guidance and exit from the side outlet port of the tubular device body, and advance toward the anatomical target according to a predicted nominal trajectory and subsequently intersect the anatomical target; and The controller circuit is configured to: Before the tissue sampling device extends into the real-time ultrasound FOV, the following graphical user interface element (UIE) is displayed: the graphical user interface element (UIE) indicates the predicted sampling device trajectory intersecting with the anatomical target on the real-time ultrasound FOV; As the tissue sampling device extends into the real-time ultrasound field of view (FOV), the ultrasound images are analyzed to determine one or more real-time trajectory parameters; The UIE is updated based on one or more determined real-time trajectory parameters to represent at least one of the actual sampling device trajectory or the updated predicted sampling device trajectory; and The updated UIE is displayed on the real-time ultrasound FOV to provide visual feedback on whether the tissue sampling device will intersect the anatomical target.

17. The system according to claim 16, wherein, The tissue sampling equipment includes: needles, brushes, snares, aspiration devices, tweezers, or auxiliary insertion devices.

18. The system according to claim 16, wherein, The tissue sampling device includes a transbronchial needle for sampling tissue from a peripheral lung target during an intrabronchial ultrasound-guided transbronchial fine needle aspiration (EBUS-TBNA) procedure.

19. The system according to claim 16, wherein, The tissue sampling device includes a needle-core assembly comprising a needle and a core insertable into the needle, wherein one or more of the core insert or the needle have a preformed curvature or bending angle substantially conforming to the predicted nominal sampling device trajectory.

20. The system according to claim 16, wherein, The controller circuit is configured to: Determine the location of the anatomical target within the real-time ultrasound field of view; and The probability of the tissue sampling device intersecting the anatomical target is determined at least in part based on the identified target location within the real-time ultrasound FOV.

21. The system according to claim 16, wherein, The controller circuit is also configured to: Determine an upper and lower bound for the trajectory, which define a range within which intersection with the anatomical target is likely to occur with a specific confidence level; and Before the tissue sampling device extends into the real-time ultrasound FOV, a UIE indicating the determined upper and lower limits of the trajectory is displayed on the real-time ultrasound FOV.

22. The system according to claim 16, wherein, The controller circuit is configured to: The probability of the tissue sampling device intersecting with the anatomical target is determined at least in part based on the actual sampling device trajectory relative to the anatomical target within the real-time ultrasound FOV; as well as The hit probability is displayed on the display.

23. The system according to claim 22, wherein, The controller circuit is configured to: In response to the hit probability being lower than a probability threshold, a suggestion is provided to withdraw the tissue sampling device or adjust the position or orientation of the tissue sampling device relative to the anatomical target within the FOV; as well as In response to the hit probability being greater than the probability threshold, a notification is provided to the user to extend the tissue sampling device according to the predicted sampling device trajectory.

24. The system according to claim 16, wherein, The controller circuit is also configured to: The analysis of the ultrasound images determines whether the tissue sampling device has intersected with the anatomical target within the real-time ultrasound field of view (FOV). In response to the analysis indicating that the tissue sampling device has not yet intersected with the anatomical target, the predicted sampling device trajectory is dynamically updated based on one or more determined real-time trajectory parameters; as well as Update the UIE to represent the updated predicted sampling device trajectory.

25. The system according to claim 24, wherein, The controller circuitry is also configured to dynamically update one or more of an upper or lower trajectory limit in response to an analysis indicating that the tissue sampling device has not yet intersected with the anatomical target. The upper and lower trajectory limits define a trajectory range within which an intersection with the anatomical target may occur with a specific confidence level.

26. The system according to claim 24, wherein, The controller circuit is also configured to determine an updated hit probability of the tissue sampling device intersecting the anatomical target as it travels along the updated predicted sampling device trajectory.

27. A method for planning an ultrasound-guided tissue acquisition process, the method comprising: Generate and display ultrasound images of the anatomical target in a real-time ultrasound field of view (FOV) within the region of interest; Before the tissue sampling device extends into the real-time ultrasound FOV, the following graphical user interface element (UIE) is displayed: The graphical user interface element (UIE) indicates the predicted sampling device trajectory intersecting with the anatomical target on the real-time ultrasound FOV; As the tissue sampling device extends into the real-time ultrasound field of view (FOV), one or more real-time trajectory parameters are determined based on the analysis of the ultrasound images; The UIE is updated based on one or more determined real-time trajectory parameters to represent at least one of the actual needle trajectory or the updated predicted sampling device trajectory; as well as The updated UIE is displayed on the real-time ultrasound FOV to provide visual feedback on whether the needle will intersect the anatomical target.

28. The method of claim 27, further comprising: Determine the location of the anatomical target within the real-time ultrasound field of view; as well as The probability of the tissue sampling device intersecting the anatomical target is determined at least in part based on the identified target location within the real-time ultrasound FOV.

29. The method of claim 27, further comprising: Determine an upper and lower bound for the trajectory, which define a range within which intersection with the anatomical target is likely to occur with a specific confidence level; and The UIE indicating the determined upper and lower limits of the trajectory is displayed on the real-time ultrasound FOV.

30. The method of claim 27, further comprising: The probability of the tissue sampling device intersecting the anatomical target is determined at least in part based on the actual needle trajectory relative to the anatomical target within the real-time ultrasound FOV; as well as This displays the hit probability.

31. The method of claim 30, further comprising: In response to the hit probability being lower than a probability threshold, a suggestion is provided to withdraw the tissue sampling device or adjust the position or orientation of the tissue sampling device relative to the anatomical target within the FOV; as well as In response to the hit probability being greater than the probability threshold, a notification is provided to the user to extend the tissue sampling device according to the predicted sampling device trajectory.

32. The method of claim 27, further comprising: The predicted sampling device trajectory is dynamically updated based on one or more determined real-time trajectory parameters; as well as Update the UIE to represent the updated predicted sampling device trajectory.

33. The method of claim 32, further comprising: Dynamically update one or more of the upper or lower trajectory limits, which define a trajectory range within which an intersection with the anatomical target may occur with a specific confidence level.

34. The method of claim 32, further comprising: Determine the updated hit probability of the tissue sampling device intersecting the anatomical target as it travels along the updated predicted sampling device trajectory.

Images