Persistent indication of biopsy target

Abstract

Various aspects of the methods, systems, and use cases can be used to generate and display real-time composite images, such as by combining a first US image in which the deep target is within the field of view (FOV) of the EBUS device with one or more subsequently acquired second US images in which the deep target is no longer within the FOV of the EBUS device. In this manner, the EBUS system persistently displays the imaged biopsy target even after the biopsy target is no longer within the FOV of the US transducer.

Description

[Technical Field]

[0001] Priority claim This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 489,392, filed March 9, 2023, the contents of which are incorporated herein by reference in their entirety. [Background technology]

[0002] A solitary pulmonary nodule (SPN) is an isolated mass in the lung that is smaller than 3 centimeters in diameter and surrounded by normal tissue. SPNs can be identified through common medical imaging techniques, such as computed tomography (CT) scans and positron emission tomography (PET) scans. In most cases, SPNs and other pulmonary nodules are simply benign tumors. In other cases, these SPNs are malignant cancers that must be treated to prevent premature death.

[0003] Diagnosis of an identified SPN cannot be performed solely through medical imaging but instead requires a biopsy, which can be performed via endobronchial ultrasound (EBUS) equipped with real-time sampling functionality. Real-time sampling using an EBUS device typically involves navigating the EBUS device through a patient's airway to a target nodule previously identified in a medical image, such that the target nodule is visible in a real-time ultrasound (US) image. Then, while the target nodule is visible in the US image, a sampling needle is extended from the working channel of the EBUS device into the target nodule to obtain a sample. Summary of the Invention [Means for solving the problem]

[0004] Modern EBUS devices include a lamp through which the biopsy needle can be extended into the field of view (FOV) of the US transducer. Specifically, the lamp is positioned proximally from the US transducer and guides the biopsy needle into the FOV along a needle path axis that sharply diverges away from the longitudinal axis of the EBUS device. Depending on the depth at which the SPN resides relative to the airway from which the biopsy is to be collected, it may not be possible to collect a sample while the SPN remains within the FOV of the US transducer. Specifically, the EBUS device may need to be moved proximally from where the SPN resides within the FOV in order for the needle path axis to converge on the SPN. To address these challenges, the inventors have developed real-time US imaging techniques that alleviate the above-mentioned issues. The following figures and specification text describe examples of these imaging techniques that facilitate the continuous visualization of SPNs that fall outside the real-time FOV of the US transducer.

[0005] In the drawings, which are not necessarily drawn to scale, like numerals may describe like components in different views. Like numerals with different letter suffixes may represent different instances of like components. The drawings illustrate generally, by way of example, but not by way of limitation, various embodiments discussed in this document. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a schematic diagram of EBUS targeting a nodule at a first distance, according to at least one example of the present disclosure. [Figure 2] 2 is a schematic diagram of the tracking of a sampling needle of the EBUS of FIG. 1 targeting a nodule at a first distance, in accordance with at least one example of the present disclosure. [Figure 3A] 2 is a schematic diagram of the EBUS of FIG. 1 targeting a node at a second distance, in accordance with at least one example of the present disclosure. [Figure 3B] 3B is a schematic diagram of the EBUS of FIG. 3A targeting a node at a second distance, according to at least one example of the present disclosure. [Figure 4] 3B is a schematic diagram of tracking of the sampling needle of the EBUS of FIG. 3A targeting a nodule at a second distance, according to at least one example of the present disclosure. [Figure 5A] 3B is a schematic diagram of a tracking of the sampling needle of the EBUS of FIG. 3A at a first position and a nodule at a second distance, according to at least one example of the present disclosure. [Figure 5B] 3B is a schematic diagram of tracking of the sampling needle of the EBUS of FIG. 3A in a second position and the nodule at a second distance, according to at least one example of the present disclosure. [Figure 5C] 3B is a schematic diagram of a tracking of the sampling needle of the EBUS of FIG. 3A in a third position and a nodule at a second distance, according to at least one example of the present disclosure. [Figure 5D] 3B is a schematic diagram of the tracking of the sampling needle of the EBUS of FIG. 3A at a fourth position intersecting a node at a second distance, according to at least one example of the present disclosure. [Figure 6] 10A-10C illustrate an example of a method for displaying a nodule when the nodule is no longer in the EBUS field of view without navigation, in accordance with at least one example of the present disclosure. [Figure 7] 10A-10C illustrate an example of a method for displaying a nodule when the nodule is no longer in the EBUS field of view with navigation, in accordance with at least one example of the present disclosure. [Figure 8] FIG. 1 is a block diagram of an example machine upon which any one or more of the techniques discussed herein may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0007] The present disclosure is directed to combining a first US image, in which a deep target is present within the field of view (FOV) of an EBUS device, with one or more subsequently acquired second US images, in which the deep target is no longer present within the FOV of the EBUS device (e.g., due to the EBUS device being moved proximally away from the target). In this manner, the EBUS system persistently displays the imaged biopsy target even after the biopsy target is no longer within the FOV of the US transducer. As described in more detail below, such a technique provides advantages over existing EBUS sampling systems by persistently displaying a biopsy target that is distal to the US FOV due to the EBUS sampling device being positioned proximally from the biopsy target to align the needle trajectory with the biopsy target.

[0008] In one example, the EBUS is a non-navigated EBUS (i.e., lacking a 6DoF sensor or other surgical navigation technology), and the images combined can be continuously monitored and updated to provide an estimate of the target's real-time location relative to the needle path axis. Because the distal tip of the needle will be outside the FOV as it penetrates the target, in some examples, the EBUS can track needle extension length and generate a virtual representation of the needle and / or needle tip on the stitched US images displayed together. Needle extension length can be determined based on data collected from a linear encoder in the needle actuation handle or any other suitable technique for measuring sampled needle extension.

[0009] In one example, the EBUS is a navigated EBUS, and the location of the target relative to the current FOV can be calculated even after the target is no longer present in the current FOV based on a comparison of first navigation data collected when the target is present in the FOV and second navigation data collected after the target is no longer present in the FOV (and, in some cases, when the needle path axis converges with the target). The first navigation data indicates a first device attitude (i.e., position and orientation), and the second navigation data indicates a second device attitude. In this example, a combination of the second navigation data and the first navigation data can be used to determine the placement of a previously acquired ultrasound image (e.g., in which the target is present in the FOV) relative to a real-time image stream being captured by the EBUS. In this manner, the previously acquired ultrasound image can be appropriately displayed in real time adjacent to the real-time image stream in a location that pinpoints the target location relative to the real-time image stream. In this manner, multiple images can be combined (e.g., stitched together) based on one or more identified locations of the SPN to provide a real-time display that includes past imaging of the SPN. In one example, the system can generate an overlay that includes the needle trajectory and / or estimated needle position.

[0010] Determining the current location of the targeted tissue (e.g., SPN) relative to the current US FOV after the targeted tissue is no longer present in the current US FOV can be based on combining images, based on navigation data, or based on a combination thereof.

[0011] 1 illustrates a schematic diagram 100 of an EBUS 102 (e.g., an EBUS sampling device) targeting a nodule 124 at a first distance (D1), in accordance with at least one example of the present disclosure. In the illustrated example, the EBUS 102 is positioned within an airway channel 120. The airway channel 120 is defined by an airway tissue wall 122. The nodule 124 is located within the airway tissue wall 122. In one example, the EBUS is designed for single use.

[0012] The EBUS 102 includes an imaging sensor 104. In this example, the imaging sensor 104 is an ultrasound transducer (e.g., a Piezoelectric Micromachined Ultrasonic Transducer (PMUT)). In one example, the imaging sensor 104 is positioned at the distal end of the EBUS 102. In one example, the imaging sensor 104 extends along a portion of the distal end of the EBUS 102. In one example, the imaging sensor 104 has a substantially planar surface on which multiple ultrasound elements are disposed, the substantially planar surface being substantially perpendicular to the longitudinal axis of the EBUS device 102. In another example, the imaging sensor 104 is curved away from the EBUS 102. The imaging sensor 104 captures medical images within a field of view (FOV) 106. The FOV 106 can be transmitted to an imaging display.

[0013] The EBUS 102 includes a sampling needle 108. In one example, the EBUS 102 also includes a sheath that surrounds a portion of the sampling needle 108. For example, the sheath is partially extended out of the EBUS 102 before the sampling needle 108 is extended from the EBUS 102. Such a configuration can protect the exit port of the EBUS 102 through which the sampling needle 108 extends from the EBUS 102.

[0014] The sampling needle 108 extends out of the EBUS 102 in a sampling needle trajectory 110. In the illustrated example, the sampling needle trajectory 110 is predictable based on the angle at which the sampling needle 108 exits the exit port of the EBUS 102 and the known length of the sampling needle 108. In the illustrated example, the sampling needle 108 and sampling needle trajectory 110 pass through the knot 124.

[0015] For example, the EBUS 102 is used to navigate to and image the nodule 124 prior to and during sampling of the nodule 124. In one example, the nodule 124 is a pre-identified tissue from which a biopsy sample is desired. In one example, ultrasound imaging (using the imaging sensor 104) is used in the respiratory area (e.g., the airway channel 120 and airway tissue wall 122) to reliably identify the target area from which to obtain a biopsy sample of the nodule 124 because ultrasound imaging shows both the tissue and the instrument (e.g., the sample needle 108 and / or sheath) in real time.

[0016] The FOV 106 extends from the airway channel 120 into the airway tissue wall 122. The FOV 106 may be angled slightly outward from the edge of the imaging sensor 104, depending on the curvature of the imaging sensor 104. In the illustrated example, the nodule 124 is positioned a first distance (D1) from the EBUS. As such, the FOV 106 is sized sufficiently to capture the nodule 124 at the first distance (D1). Additionally, because the first distance (D1) of the nodule 124 is relatively shallow within the airway tissue wall 122, the EBUS device 102 may be positioned such that the sampling needle trajectory 110 intersects the target nodule 124 within the FOV 106. Thus, the sampling needle 108 may be used to perform real-time visualization of the target nodule 124 while the needle trajectory 110 intersects the target 124 within the FOV 106. Thus, a clinician can collect a biopsy sample from the nodule 124 while the nodule 124 and sampling needle 108 are imaged in real time.

[0017] FIG. 2 illustrates a schematic diagram 150 of tracking of the sampling needle 108 of the EBUS 102 of FIG. 1 targeting the nodule 124 at a first distance (D1), in accordance with at least one example of the present disclosure. In the illustrated example, the EBUS 102 is positioned within an airway channel 120. The airway channel 120 is defined by an airway tissue wall 122. The nodule 124 is positioned within the airway tissue wall 122. The EBUS 102 is substantially similar to or the same as the EBUS 102 of FIG. 1. For example, the EBUS 102 of FIG. 2 can include an imaging sensor 104 and an exit port including a ramp that guides the sampling needle 108 along the sampling needle trajectory 110 into the FOV 106. The imaging sensor 104 captures the FOV 106. In one example, the imaging sensor 104 and FOV 106 are substantially similar to the imaging sensor 104 and FOV 106 of FIG.

[0018] The FOV 106 is transmitted and displayed on the imaging display. In the illustrated example, a sampling needle representation 152 is generated on the imaging display to assist the clinician in collecting a sample of the nodule 124 with the sampling needle 108. For example, the sampling needle representation 152 is a graphical representation of the extension of the sampling needle 108. In one example, the sampling needle representation 152 dynamically changes as the sampling needle 108 continues to be extended or retracted within the FOV 106. In one example, the sampling needle representation 152 appears substantially similar to the sampling needle 108 to assist in identifying the sampling needle 108.

[0019] In the illustrated example, a sampling needle trajectory representation 154 is generated on the imaging display to assist the clinician in collecting a sample of the nodule 124 with the sampling needle 108. For example, the sampling needle trajectory representation 152 is a graphical representation of the predicted trajectory of the sampling needle 108 (e.g., the sampling needle trajectory 110). In one example, the sampling needle trajectory representation 154 can be an arrow (as shown), a line, or other representation of the sampling needle trajectory 110. In one example, the sampling needle trajectory representation 154 is positioned at the edge of the FOV 106. For example, it is positioned at the top or distal edge of the FOV 106 based on the angle of the sampling needle trajectory 110. In another example, the sampling needle trajectory representation 154 is positioned a predetermined distance from the sampling needle representation 152. For example, as the sampling needle 108 (and thus the sampling needle representation 152) is extended or retracted, the sampling needle trajectory representation 154 moves along the sampling needle trajectory 110 at a fixed distance from the sampling needle representation 152. In some examples, multiple sampling needle trajectory representations 154 are generated on the imaging display.

[0020] FIG. 3A illustrates a schematic diagram 200 of the EBUS 102 of FIG. 1 targeting a nodule 202 at a second distance (D2), in accordance with at least one example of the present disclosure. In the illustrated example, the EBUS 102 is positioned within an airway channel 120. The airway channel 120 is defined by an airway tissue wall 122. The nodule 202 is positioned within the airway tissue wall 122. The EBUS 102 is substantially similar to or the same as the EBUS 102 of FIG. 1. The EBUS 102 is substantially similar to or the same as the EBUS 102 of FIG. 1. For example, the EBUS 102 of FIG. 3A can include an imaging sensor 104 and an exit port including a ramp that guides the sampling needle 108 along the sampling needle trajectory 110 into a field of view 206. The imaging sensor 104 captures the FOV 206. In one example, the imaging sensor 104 is substantially similar to the imaging sensor of Figure 1. In the illustrated example, the sampling needle trajectory 110 does not pass through the nodule 202.

[0021] For example, the EBUS 102 is used to navigate to and image the nodule 202 prior to sampling the nodule 202. In one example, the nodule 202 is a pre-identified tissue from which a biopsy sample is desired. In one example, ultrasound imaging (using the imaging sensor 104) is used in the respiratory area (e.g., the airway channel 120 and airway tissue wall 122) to reliably identify the target area from which to obtain a biopsy sample of the nodule 202 because ultrasound imaging shows both the tissue and the instrument (e.g., the sample needle and / or sheath) in real time.

[0022] The FOV 206 extends from the airway channel 120 into the airway tissue wall 122. The FOV 206 may be angled slightly outward from the edge of the imaging sensor 104 depending on the curvature of the imaging sensor 104. In the illustrated example, the nodule 202 is positioned a second distance (D2) from the EBUS. The second distance (D2) is greater than the first distance (D1) of FIG. 1 . As such, the FOV 206 is sized sufficiently to capture the nodule 202 at the second distance (D2). However, because the second distance (D2) of the nodule 202 is relatively deep into the airway tissue wall 122, both the nodule 202 and the sampling needle trajectory 110 (and thus the sampling needle) are simultaneously within the FOV 206, but the sampling needle trajectory 110 does not pass through the nodule 202 within the FOV 206 while being imaged in real time. In particular, as shown, the sampling needle trajectory 110 begins at a side exit ramp of the EBUS device 102 and diverges away from the longitudinal axis of the EBUS device 102. Due to the divergence of the sample needle trajectory 110 away from the longitudinal axis while traversing from the proximal boundary of the FOV 206 to the distal boundary of the FOV 206, the needle trajectory 110 passes through the distal boundary of the FOV 206 at a certain depth. It will be recognized that a target nodule deeper in the tissue (e.g., farther from the airway wall) than the depth at which the needle trajectory passes through the distal boundary of the FOV cannot be simultaneously sampled by a needle following the needle trajectory 110 while being imaged by the imaging sensor 104.

[0023] 3B illustrates a schematic diagram 210 of the EBUS 102 of FIG. 3A targeting a nodule 212 at a second distance (D2), in accordance with at least one example of the present disclosure. In the illustrated example, the EBUS 102 is positioned within an airway channel 120. The airway channel 120 is defined by an airway tissue wall 122. The nodule 212 is located within the airway tissue wall 122. The EBUS 102 is substantially similar to or the same as the EBUS 102 of FIGS. 1 and 2A.

[0024] In the illustrated example, the imaging sensor 104 captures a first (previous) field of view (FOV) 216. The previous FOV 216 can be captured at a time when the needle trajectory has not intersected the target nodule 212 and when the target nodule 212 is within the FOV of the imaging sensor 104 (e.g., when the EBUS device 102 is positioned as shown in FIG. 3A ). The previous FOV 216 can be transmitted to an imaging display. The imaging sensor 104 is then translated through the airway channel 120 to capture a second (current) FOV 214. The current FOV 214 can be captured at a time when the needle trajectory has intersected the target nodule 212, but when the target nodule 212 is no longer within the FOV of the imaging sensor 104 (e.g., when the EBUS device 102 is positioned as shown in FIG. 3B ). The current FOV 214 can be transmitted to the imaging display in real time. The current FOV 214 and previous FOV 216 extend from the airway channel 120 into the airway tissue wall 122. The current FOV 214 and previous FOV 216 can be angled slightly outward from the edge of the imaging sensor 104 depending on the curvature of the imaging sensor 104. In the illustrated example, the nodule 212 is positioned at a second distance (D2) from the EBUS. The second distance (D2) is greater than the first distance (D1) in FIG. 1 . As such, the current FOV 214 and previous FOV 216 are sized sufficiently to capture the nodule 212 at the second distance (D2). In the illustrated example, the sampling needle 108 has passed through the nodule 212. However, because the second distance (D2) of the nodule 212 is relatively deep within the airway tissue wall 122, the sampling needle 108 does not pass through the nodule 212 while being imaged in real time within the current FOV 214. Instead, the current FOV 214 captures the sampling needle 108 extending from the EBUS 102, and the previous FOV 216 captured the nodule 212.

[0025] In this example, the current FOV 214 and the previous FOV 216 may be stitched together to generate a composite FOV. The composite FOV may then be displayed along with a real-time image of the sampling needle 108 within the current FOV 214 portion of the composite FOV. As the sampling needle 108 extends into the previous FOV 216 portion of the composite FOV, a graphical representation of the predicted needle location may then be superimposed on the composite FOV to assist the clinician in targeting the nodule 212.

[0026] FIG. 4 illustrates a schematic diagram 250 of tracking of the sampling needle of the EBUS 102 of FIG. 3A targeting the nodule 202 at a second distance (D2), in accordance with at least one example of the present disclosure. In the illustrated example, the EBUS 102 is positioned within the airway channel 120. The airway channel 120 is defined by an airway tissue wall 122. The nodule 202 is positioned within the airway tissue wall 122. The EBUS 102 is substantially similar to or the same as the EBUS 102 of FIG. 1. For example, the EBUS 102 of FIG. 4 can include an imaging sensor 104 and an exit port including a ramp that guides the sampling needle 108 along the sampling needle trajectory 110 into the FOV 206. The imaging sensor 104 captures the FOV 206. In one example, the imaging sensor 104 is substantially similar to the imaging sensor 104 of FIG. 1. The FOV 206 can be transmitted to an imaging display. The FOV 206 can be substantially similar to the FOV 206 of FIG.

[0027] The EBUS 102 includes a sampling needle. In one example, the EBUS 102 also includes a sheath that surrounds a portion of the sampling needle. For example, the sheath extends partially out of the EBUS 102 before the sampling needle is extended from the EBUS 102. Such a configuration can protect the exit port of the EBUS 102 through which the sampling needle extends from the EBUS 102. In the illustrated example, the sampling needle trajectory 110 does not pass through the node 202 in the FOV 206.

[0028] For example, the EBUS 102 is used to navigate to and image the nodule 202 prior to sampling the nodule 202. In one example, the nodule 202 is a pre-identified tissue from which a biopsy sample is desired. In one example, ultrasound imaging (using the imaging sensor 104) is used in the respiratory area (e.g., the airway channel 120 and airway tissue wall 122) to reliably identify the target area from which to obtain a biopsy sample of the nodule 202 because ultrasound imaging shows both the tissue and the instrument (e.g., the sample needle and / or sheath) in real time.

[0029] The FOV 206 is transmitted and displayed on the imaging display. In the illustrated example, a sampling needle representation 252 is generated on the imaging display to assist the clinician in collecting a sample of the nodule 202 with the sampling needle. For example, the sampling needle representation 252 is a graphical representation of the extension of the sampling needle. In one example, the sampling needle representation 252 dynamically changes as the sampling needle continues to be extended or retracted within the FOV 206. In one example, the sampling needle representation 252 is an indication perpendicular to the sampling needle trajectory 110. In another example, the sampling needle representation 252 appears substantially similar to the sampling needle to assist in identifying the sampling needle. In another example, multiple sampling needle representations 252 and 254 are generated. For example, a first sampling needle representation 252 identifies that the sampling needle is extended, and a second sampling needle representation 254 indicates the distance the sampling needle is extended.

[0030] In the illustrated example, a sampling needle trajectory representation 256 is generated on the imaging display to assist the clinician in collecting a sample of the nodule 202 with the sampling needle. For example, the sampling needle trajectory representation 256 is a graphical representation of the predicted trajectory of the sampling needle (e.g., the sampling needle trajectory 110). In one example, the sampling needle trajectory representation 256 can be an arrow (as shown), a line, or other representation of the sampling needle trajectory 110. In one example, the sampling needle trajectory representation 256 is positioned at the edge of the FOV 206. For example, the sampling needle trajectory representation 256 is positioned at the top or distal edge of the FOV 206 based on the angle of the sampling needle trajectory 110. In another example, the sampling needle trajectory representation 256 is positioned a predetermined distance from the sampling needle representations 252 or 254. For example, as the sampling needle (and thus the sampling needle representation 252 or 254) is extended or retracted, the sampling needle trajectory representation 256 moves along the sampling needle trajectory 110 at a fixed distance from the sampling needle representation 252 or 254. In some examples, multiple sampling needle trajectory representations 256 and 258 are generated on an imaging display.

[0031] 5A-5D illustrate schematic diagrams of tracking of the sampling needle 108 of the EBUS 102 of FIG. 3A at various positions 300, 320, 330, and 340 and the nodule 302 at a second distance (D2), in accordance with at least one example of the present disclosure.

[0032] In the illustrated example, the EBUS 102 is positioned within an airway channel 120. The airway channel 120 is defined by an airway tissue wall 122. The nodule 302 is positioned within the airway tissue wall 122. The EBUS 102 is substantially similar to or the same as the EBUS 102 of FIG. 1. For example, the EBUS 102 of FIGS. 5A-5D can include an imaging sensor 104 and an exit port including a ramp that directs the sampling needle along the sampling needle trajectory 110 into the FOV. The imaging sensor 104 captures the FOV. In one example, the imaging sensor 104 is substantially similar to the imaging sensor 104 of FIG. 1.

[0033] The FOV extends from the airway channel 120 into the airway tissue wall 122. The FOV may be angled slightly outward from the edge of the imaging sensor 104 depending on the curvature of the imaging sensor 104. In the illustrated example, the nodule 302 is positioned a second distance (D2) from the EBUS 102. The second distance (D2) is greater than the first distance (D1) in FIG. 1 . However, because the second distance (D2) of the nodule 302 is relatively deep into the airway tissue wall 122, both the nodule 302 and the sampling needle are not simultaneously within the current FOV 304 while being imaged in real time.

[0034] In the illustrated example, the EBUS 102 starts at a first, most distal position 300 (see FIG. 5A ) and captures a first FOV 312. The first FOV 312 is transmitted to an imaging display. The first FOV 312 may include a representation of the sampling needle trajectory 110. At the first position 300, the sampling needle trajectory 110 does not intersect the nodule 302. With the EBUS 102 in the first position 300, the nodule 302 is captured within the first FOV 312.

[0035] As the EBUS 102 is withdrawn proximally within the airway channel 120, the EBUS 102 is moved to a second position 320 (see FIG. 5B ) and captures a second FOV 314. The second FOV 314 is transmitted to an imaging display. In the illustrated example, the second FOV 314 and the first FOV 312 are combined to generate a composite image 310a. The composite image 310a is displayed on the imaging device. In the illustrated example, the first FOV 312 and the second FOV 314 have an overlap portion 306a. Because of the overlap portion 306a, the more recently acquired FOV (here, the second FOV 314) is displayed due to the overlap portion 306a. Thus, only a portion of the first FOV 312 is used for the composite image 310a. The composite image 310a can include a representation of the sampling needle trajectory 110. In the second position 320 , the sampling needle trajectory 110 does not intersect the knot 302 .

[0036] As the EBUS 102 is further withdrawn proximally within the airway channel 120, the EBUS 102 is moved to a third position 330 (see FIG. 5C ) and captures a third FOV 316. The third FOV 316 is transmitted to an imaging display. In the illustrated example, the third FOV 316, the second FOV 314, and the first FOV 312 are combined to generate a composite image 310b. In another example, the third FOV 316 and the second FOV 314 are combined to generate a composite image. For example, images are combined only distally as needed to maintain the nodule 302 (e.g., a portion of the nodule 302, a substantial portion of the nodule 302, or the entire nodule 302) in the composite image. The composite image 310b is displayed on the imaging device. In the illustrated example, the first FOV 312 and the second FOV 314 have overlapping portion 306a, and the second FOV 314 and the third FOV 316 have overlapping portion 306b. A portion of overlapping portion 306a is overlap between the first FOV 312 and the third FOV 316. Because overlapping portion 306a and overlapping portion 306b exist, the more recently acquired FOV (here, the second FOV 314) is displayed due to overlapping portion 306a (the third FOV 316 also does not overlap with the first FOV 312), and the third FOV 316 is displayed due to overlapping portion 306b. Thus, only a portion of the first FOV 312 and only a portion of the second FOV 314 are used for the composite image 310b. The composite image 310b may include a representation of the sampling needle trajectory 110. At a third position 330, the sampling needle trajectory 110 does not intersect the nodule 302.

[0037] As the EBUS 102 is further withdrawn proximally within the airway channel 120, the EBUS 102 is moved to a fourth position 340 (see FIG. 5D ) and captures a fourth FOV 304. The fourth FOV 304 is transmitted to an imaging display. In the illustrated example, the fourth FOV 304, the third FOV 316, the second FOV 314, and the first FOV 312 are combined to generate a composite image 310c. In another example, the fourth FOV 306 and the first FOV 312 are combined to generate a composite image. In the illustrated example, the first FOV 312 and the fourth FOV 304 have a non-overlapping portion 308. In one example, the non-overlapping portion 308 does not include the nodule 302 or the sampling needle trajectory 110, so the composite image leaves the non-overlapping portion 308 empty. Alternatively, the imaging display (or computing device) generates artificial data to fill the non-overlapping portion 308. For example, the imaging device (or computing device) interpolates data between the first FOV 312 and the fourth FOV 304 based on landmarks present in the first FOV 312 and the fourth FOV 304. As another example, the images are combined only distally as needed to maintain the nodule 302 (e.g., a portion of the nodule 302, a substantial portion of the nodule 302, or the entire nodule 302) in the composite image. The composite image 310c is displayed on the imaging device. In the illustrated example, the first FOV 312 and the second FOV 314 have overlapping portion 306a, the second FOV 314 and the third FOV 316 have overlapping portion 306b, and the third FOV 316 and the fourth FOV 304 have overlapping portion 306c. The overlapping portions 306a, 306b, and 306c may overlap one another. With respect to the overlapping portions 306a, 306b, and 306c, the more recently acquired FOV is displayed for the given overlapping portion 306a, 306b, and 306c.Thus, only a portion of the first FOV 312, only a portion of the second FOV 314, and only a portion of the third FOV 316 are used for the composite image 310c. The composite image 310c may include a representation of the sampling needle trajectory 110. In the illustrated example, the sampling needle trajectory 110 intersects the nodule 302 at a fourth location 340.

[0038] When the sampling needle trajectory 110 intersects the nodule 302, the sampling needle is extended. As the sampling needle is extended, the fourth FOV 304 is captured and updated in real time, causing the composite image 310c to be updated in real time. In one example, the sampling needle is extended and a graphical representation of the sampling needle is generated and displayed over the composite image 310c.

[0039] It will be appreciated that compositing or splicing adjacent and potentially overlapping ultrasound images together, as shown in FIGS. 5A-5D , provides a significant improvement over existing ultrasound sampling systems. Specifically, compositing adjacent ultrasound images together, as shown and described in connection with FIGS. 5A-5D , facilitates real-time visualization of a sampling procedure being performed on a target nodule 302 that is intersected by a needle trajectory while simultaneously being located too deep within the patient's tissue to be within the current FOV. For example, in an embodiment in which the needle trajectory traverses the FOV from the proximal end to the distal end while diverging away from the longitudinal axis of the EBUS 102, the target nodule 302 may reside above the needle trajectory whenever the target nodule 302 is within the current FOV. By withdrawing the EBUS 102 proximally, the needle trajectory 110 can be caused to intersect the target nodule 302. However, withdrawing the EBUS 102 as required to cause the needle trajectory 110 to intersect the target 302 can cause the target nodule to be positioned distal to the distal FOV boundary. The disclosed technique mitigates this drawback by stitching adjacent images together and compositing previously acquired image data onto the distal end of the current FOV, effectively extending the FOV beyond the distal FOV boundary.

[0040] FIG. 6 illustrates an example method 600 for displaying a nodule when the nodule is no longer within the EBUS field of view without navigation, according to at least one example of the present disclosure. Method 600 is used when the nodule (e.g., nodule 202, nodule 212, and / or nodule 302) is at a depth (e.g., a second distance (D2)) within tissue (e.g., airway tissue wall 112) that prevents the EBUS from imaging the nodule and the needle (e.g., sampling needle 108) within the same field of view (FOV) during collection of a nodule sample. In one example, method 600 can be implemented by a computing device (e.g., the machine of FIG. 8) communicatively coupled to the EBUS (e.g., the EBUS 102 of FIGS. 1-5). The computing device includes processing circuitry for performing various image processing tasks discussed in connection with method 600. Additionally, the computing device can include an output display, such as a monitor for displaying ultrasound images and various additional user interface elements, among other things.

[0041] At the beginning of method 600 (e.g., at the beginning of a procedure), an EBUS (e.g., EBUS 102) is positioned at a first location where a nodule is located. At 602, a computing device receives a first image of the nodule with the needle unextended. The first image does not include navigation data. The nodule is positioned at a second distance from the EBUS such that the nodule is within the EBUS FOV, but the needle and the nodule cannot be imaged simultaneously during collection of a sample from the nodule. For example, the needle and the nodule cannot be imaged simultaneously during collection of a sample because the angle at which the needle exits the EBUS and the depth of the nodule do not intersect within the EBUS FOV. In one example, the computing device receives multiple first images. In one example, the computing device receives multiple first images at multiple rotational positions. For example, the computing device receives multiple first images at multiple rotational positions and determines the ideal position of the EBUS so that the sampling needle intersects the nodule at the nodule's maximum diameter or at the center of the nodule. Due to pre-planning of the operation, the computing device knows the rotational margin for error of the EBUS. In one example, the EBUS can be moved proximally, and thus the computing device receives multiple first images including the nodule. After the first image (or multiple first images) is received, the EBUS is moved proximally.

[0042] At 604, the computing device receives a second image without the nodule. The second image does not include navigation data. The computing device receives the second image in real time. Thus, the computing device receives multiple second images. In one example, the second image includes a portion of the nodule, depending on the depth of the nodule. In other words, method 600 as described occurs in real time or near real time, with the first image being a previously captured image frame (or a composite of multiple frames) and the second image being a current real-time image (which could be multiple images coming in at the camera frame rate if the EBUS is stationary).

[0043] At 606, the computing device combines the first image and the second image to generate a composite image. Thus, the composite image includes the nodule and the current FOV (which may include the needle extended from the EBUS). In one example, the computing device combines the first image and the second image by identifying common visual landmarks between the first image and the second image. In one example, after the common visual landmarks are identified, the computing device matches the landmarks in the first image and the second image to align the first image and the second image. In one example, once the landmarks are aligned, the computing device generates the composite image. In one example, the computing device uses a portion of the first image with the second image to generate the composite image. In one example, the computing device combines multiple first images with the second image (e.g., as shown in FIGS. 5A-5D) to generate the composite image. For example, the computing device uses a portion of the most distal first image to generate the most distal portion of the composite image. For example, the computing device uses the most distal first image to be used to generate a distal-most portion of the composite image that does not include the second most distal first image. Thus, the composite image can be formed from multiple previously acquired images at different distal and proximal locations. In such an example, the computing device uses the most recent image received from that portion of the patient's anatomy to generate the composite image. Thus, the computing device generates the composite image by overwriting the first image (previously acquired image) with the second image (real-time image). Alternatively or additionally, the composite image is continuously updated as the EBUS is withdrawn, such that the composite image grows linearly as additional real-time images are captured and combined with the previously acquired images.Method 600 can be performed in real time and can continue to build a composite image as the EBUS device is withdrawn to properly position the sampling needle (as illustrated by the dotted line running from operation 614 back to operation 604).

[0044] At 608, the computing device determines the location of the nodule relative to the second image. In one example, the computing device determines the location of the nodule by identifying the nodule. For example, the computing device identifies the nodule by identifying an image signature of the nodule. As another example, the computing device identifies the nodule by receiving user input selecting the nodule. In one example, the computing device determines the location of the nodule based on the depth of the nodule in the first image based on known field of view coordinates. In one example, the computing device determines the location of the nodule relative to the second image based on the position of the first image relative to the second image in the composite image and based on the known field of view coordinates.

[0045] The needle can be extended out of the exit port of the EBUS. At 610, when the needle is extended, the computing device tracks the location of the needle. Because the distal tip of the needle will be outside the FOV when sampling the nodule, the distal tip of the needle will not be shown on the second image when the needle pierces the nodule. Thus, the computing device tracks the location of the needle. For example, the computing device tracks the length the needle is extended from the EBUS. As another example, the computing device tracks the trajectory of the needle as it exits the EBUS. As another example, the computing device tracks (or knows) the angle at which the needle exits the EBUS. As another example, the computing device tracks the length, trajectory, and angle of the needle, or a combination thereof. In one example, the computing device tracks the location of the needle by identifying the needle on the second image. In one example, the computing device tracks the location of the needle by determining the location of the needle relative to the second image. For example, the computing device identifies an image signature of the needle and tracks the location of the needle. In another example, the needle includes a visual marker to assist in tracking the location of the needle. For example, the needle includes one or more bright rings along the length of the needle. For example, the computing device identifies the bright rings, the distance between the bright rings, or the number of visible bright rings and tracks the location of the needle. In another example, the EBUS includes a sensor at the exit port, and the computing device receives information from the EBUS indicating the distance the needle has been extended. From the distance information and the known angle of exit of the needle, the computing device determines the location of the needle. In another example, the computing device tracks the location of the needle based on information received from a linear encoder in the needle actuation handle.In one example, the computing device predicts the needle trajectory of the needle whether or not the needle is extended from the exit port. In one example, the computing device knows the angle at which the needle exits the exit port of the EBUS and the distance between the exit port and the imaging sensor. Thus, the computing device can use the known angle of exit and the distance between the exit port and the imaging sensor to determine the predicted needle trajectory with the needle not extended from the EBUS. In one example, the computing device uses the determined location of the nodule to determine whether the predicted needle trajectory intersects the nodule.

[0046] At 612, the computing device generates a needle representation. The computing device generates the needle representation based on the tracked needle location. The needle representation is a virtual visual representation of the needle to be displayed on the imaging device. The needle representation can assist a clinician in collecting a sample from a nodule with the needle. In one example, the computing device generates a graphical representation of the needle extension for the needle representation. For example, the needle representation looks substantially similar to a needle to assist in identifying the needle in the image. For example, the needle representation can be a line or a needle-shaped object.

[0047] In one example, the needle representation is a representation of a predicted needle trajectory. For example, the computing device generates a graphical representation of the predicted needle trajectory. In one example, the predicted needle trajectory representation is an arrow, line, or other representation of the predicted needle trajectory. In one example, the predicted needle trajectory representation extends to or is positioned at an intersection with a nodule. In one example, the predicted needle trajectory representation extends to or is positioned at an edge of the composite image. For example, the predicted needle trajectory representation extends to or is positioned at an upper or distal edge of the composite image depending on the angle and position of the needle. In one example, the predicted needle trajectory representation is positioned a predetermined distance from the needle representation. For example, when the computing device moves the needle representation based on needle extension or retraction, the computing device moves the predicted needle trajectory representation a fixed distance from the needle representation. In one example, the computing device generates multiple predicted needle trajectory representations. For example, the multiple predicted needle trajectory representations are positioned along the same predicted needle trajectory, but the individual representations are positioned along that trajectory. In another example, the computing device determines a predicted distance between the EBUS exit port and the node. For example, the computing device uses the predicted distance to determine the length the needle should be extended from the EBUS. In one example, the computing device determines the horizontal distance between the predicted needle trajectory and the node. Thus, the computing device determines how far the EBUS needs to move so that the predicted needle trajectory (and therefore the needle) intersects the node.

[0048] At 614, the computing device displays the composite image. Despite the nodule being positioned too deep into the tissue during collection of the tissue sample to image both the nodule and the needle within a single FOV, the computing device generates a composite image with a modified FOV that includes the nodule and the needle. In one example, before the nodule moves out of the FOV (e.g., of the first image), the first image is displayed in real time without being modified by the processing described herein. In one example, the computing device allows a user to switch between multiple image display modes. For example, the computing device can include a first mode that generates a composite image when the nodule is detected beyond a certain depth and a second mode that displays an unmodified real-time image. When the needle representation is generated, the needle representation is displayed over the composite image. As the needle continues to be extended or retracted, the needle representation is dynamically updated in real time over the composite image. In one example, the computing device displays the predicted distance between the EBUS exit port and the nodule, the predicted length the needle needs to be extended to reach the nodule, and / or the distance the EBUS needs to travel for the needle to intersect the nodule. For example, displaying the predicted distance can assist a user in determining how far to extend the needle from the EBUS. Thus, the computing device can assist in identifying when the nodule has been reached for sampling.

[0049] Method 600 operates on a computing device in real time and continuously throughout the procedure. For example, while the needle is extended into the EBUS FOV, a second image continues to be acquired, and method 600 operates in real time to generate and display a composite image using the second image and the first image while displaying the nodule in relation to the needle. As another example, a first image would continue to be acquired until a second image without the nodule is identified. As another example, a first image would continue to be acquired until a second image in which the needle trajectory intersects the nodule is identified. The first and second images can be continuously received, and new composite images can be displayed throughout the procedure. Thus, the computing device continually displays the imaged biopsy target (e.g., the nodule) even after the biopsy target is no longer within the EBUS FOV.

[0050] FIG. 7 illustrates an example method for displaying a nodule when the nodule is no longer within the EBUS field of view with navigation, according to at least one example of the present disclosure. Method 700 is used when the nodule (e.g., nodule 202, nodule 212, and / or nodule 302) is at a depth (e.g., a second distance (D2)) within tissue (e.g., airway tissue wall 112) that prevents the EBUS from imaging the nodule and the needle (e.g., sampling needle 108) within the same field of view (FOV) during collection of a nodule sample. In one example, method 700 can be implemented by a computing device (e.g., the machine of FIG. 8) communicatively coupled to the EBUS (e.g., the EBUS 102 of FIGS. 1-5). The computing device includes processing circuitry for performing various image processing tasks discussed in connection with method 700. Additionally, the computing device can include an output display, such as a monitor for displaying ultrasound images and various additional user interface elements, among other things.

[0051] At the beginning of method 700 (e.g., at the beginning of a procedure), an EBUS (e.g., EBUS 102) is positioned at a first location where a nodule is to be located. EBUS devices commonly used today include navigation systems that allow for precise tracking of position and orientation within a patient. EBUS navigation systems may utilize electromagnetic (EM) tracking techniques, which allow for tracking of the EBUS within six degrees of freedom. In one example, the tracked locations include the EBUS location and EBUS orientation.

[0052] At 702, the computing device receives a first image of the nodule with the needle unextended. The first image includes navigation data. The nodule is positioned at a second distance from the EBUS such that the nodule is within the EBUS FOV, but the needle and the nodule cannot be imaged simultaneously during collection of a sample from the nodule. For example, the needle and the nodule cannot be imaged simultaneously during collection of a sample because the angle at which the needle exits the EBUS and the depth of the nodule do not intersect within the EBUS FOV. In one example, the computing device receives multiple first images. In one example, the computing device receives multiple first images at multiple rotational positions. For example, the computing device receives multiple first images at multiple rotational positions and determines an ideal position for the EBUS so that the sampling needle intersects the nodule at its maximum diameter or at its center. Due to pre-planning of the operation, the computing device knows the rotational margin for error of the EBUS. In another example, the computing device knows where and at what rotation the EBUS needs to be positioned to capture the nodule, so that the computing device knows if the EBUS is misaligned. In one example, the EBUS can be moved proximally, and thus the computing device receives multiple first images including the nodule. After the first image (or multiple first images) are received, the EBUS is moved proximally.

[0053] At 704, the computing device receives a second image without the nodule. The second image includes navigation data. The computing device receives the second image in real time. Thus, the computing device receives multiple second images. In one example, the second image includes a portion of the nodule, depending on the depth of the nodule. In other words, method 700 as described occurs in real time or near real time, with the first image being a previously captured image frame (or a composite of multiple frames) and the second image being a current real-time image (which could be multiple images coming in at the camera frame rate if the EBUS is stationary).

[0054] At 706, the computing device generates a composite image. In one example, the computing device combines the first image and the second image to generate the composite image. Thus, the composite image includes the nodule and the current FOV (which may include the needle extended from the EBUS). The computing device combines the first image and the second image using the navigation data. In one example, the computing device uses the navigation data to determine the distance between the first image and the second image. In one example, the computing device uses the distance between the first image and the second image to position the first image relative to the second image. In one example, the computing device uses a portion of the first image together with the second image to generate the composite image. In one example, the computing device combines multiple first images with the second image (e.g., as shown in FIGS. 5A-5D) to generate the composite image. For example, the computing device uses a portion of the most distal first image to generate the most distal portion of the composite image. For example, the computing device uses the most distal first image to be used to generate the distal-most portion of the composite image that does not include the second most distal first image. Thus, the composite image can be formed from multiple previously acquired images at different distal and proximal locations. Alternatively or additionally, the composite image can be continuously updated as the EBUS is withdrawn, such that the composite image grows linearly as additional real-time images are captured and combined with the previously acquired images. In such an example, the computing device uses the most recent image received from that portion of the patient's anatomy to generate the composite image. Thus, the computing device generates the composite image by overwriting a portion of the first image (previously acquired image) with the second image (real-time image). In another example, the computing device can interpolate data between the first and second images when portions of the first and second images do not overlap.For example, the computing device interpolates data between the first image and the second image by identifying common landmarks in the first image and the second image and by interpolating missing portions of the common landmarks between the first image and the second image.

[0055] In another example, the computing device does not fill the space between the first and second images if they do not overlap. For example, the first image includes a nodule, and the second image includes the current FOV, which does not include the nodule. Therefore, because the clinician needs to know the location of the nodule relative to the current FOV to properly position the EBUS to collect a sample of the nodule with a needle, the computing device displays the first and second images with a space between them based on the distance between the first and second image FOV. In one example, the computing device displays graphical representations and other information as described below, with such representations spanning the first and second images as necessary, as if data between the first and second images were not missing. In another example, rather than creating a space between the first and second images, the computing device includes an indication that the first and second images are separated by a given distance. In one example, the indication is a line or marking between the images to indicate that data is missing. In another example, the computing device displays the distance between the images on the display device and marks the edges of the images. In such an example, the computing device displays graphical representations and other information as described below, and such representations are included in the first and second images as needed, as if data between the first and second images were not missing. However, representations that would extend between the first and second images are not included. Thus, certain representations (e.g., predicted needle trajectories) may be offset from one another when the first and second images are placed side-by-side.

[0056] In another example, the computing device generates a composite image by determining the positioning between the first image and the second image. In such an example, the only data retained from the first image may be the location information and the nodule. For example, once the location of the nodule relative to the second image is determined in 708, the composite image includes the second image and the nodule. The nodule may be positioned where the nodule would have been positioned if the first image still existed. For example, the computing device may generate a representation of the nodule that is located outside the second image, inside the second image, or partially inside the second image, depending on the nodule's location. For example, the computing device may generate a representation of the nodule that matches the shape of the nodule, is circular, oval, the outline of the nodule, or any other graphical symbol to represent the nodule. In another example, the representation of the nodule may be a color that is easily distinguished from the second image.

[0057] In one example, the computing device continuously updates the composite image by updating the real-time second image in real-time as the real-time second image is received.

[0058] At 708, the computing device determines the location of the nodule relative to the second image. In one example, the computing device determines the location of the nodule by identifying the nodule. For example, the computing device identifies the nodule by identifying an image signature of the nodule. As another example, the computing device identifies the nodule by receiving user input selecting the nodule. In one example, the computing device determines the location of the nodule based on a depth of the nodule in the first image based on known field of view coordinates. In one example, the computing device determines the location of the nodule relative to the second image based on a position of the first image relative to the second image in the composite image and based on the known field of view coordinates. In another example, the computing device determines the location of the nodule by comparing navigation data from the first image with navigation data from the second image. For example, the navigation data from the first image indicates a first device pose and the navigation data from the second image indicates a second device pose, enabling the computing device to determine the relative location of the nodule relative to the second image.

[0059] The needle is extended out of the exit port of the EBUS. At 710, the computing device tracks the location of the needle as it is extended. Because the distal tip of the needle will be outside the FOV when sampling the nodule, the distal tip of the needle will not be shown on the second image as the needle pierces the nodule. Thus, the computing device tracks the location of the needle. For example, the computing device tracks the length the needle is extended from the EBUS. As another example, the computing device tracks the trajectory of the needle as it exits the EBUS. As another example, the computing device tracks (or knows) the angle at which the needle exits the EBUS. As another example, the computing device tracks the length, trajectory, and angle of the needle, or a combination thereof. In one example, the computing device tracks the location of the needle by identifying the needle on the second image. In one example, the computing device tracks the location of the needle by determining the location of the needle relative to the second image. For example, the computing device identifies an image signature of the needle and tracks the location of the needle. In another example, the needle includes a visual marker to assist in tracking the location of the needle. For example, the needle includes one or more bright rings along the length of the needle. For example, the computing device identifies the bright rings, the distance between the bright rings, or the number of visible bright rings and tracks the location of the needle. In another example, the EBUS includes a sensor at the exit port, and the computing device receives information from the EBUS indicating the distance the needle has been extended. From the distance information and the known angle of exit of the needle, the computing device determines the location of the needle. In another example, the computing device tracks the location of the needle based on information received from a linear encoder in the needle actuation handle.In one example, the computing device tracks the location of the needle using navigation data from the second image and / or the first image. In one example, the computing device predicts the needle trajectory of the needle regardless of whether the needle is extended from the exit port. In one example, the computing device knows the angle at which the needle exits the exit port of the EBUS and the distance between the exit port and the imaging sensor. Thus, the computing device can use the known angle of exit and the distance between the exit port and the imaging sensor to determine the predicted needle trajectory when the needle is not extended from the EBUS. In one example, the computing device uses the determined location of the nodule to determine whether the predicted needle trajectory intersects the nodule.

[0060] At 712, the computing device generates a needle representation. The computing device generates the needle representation based on the tracked needle location. The needle representation is a virtual visual representation of the needle to be displayed on the imaging device. The needle representation can assist a clinician in collecting a sample from a nodule with the needle. In one example, the computing device generates a graphical representation of the needle extension for the needle representation. For example, the needle representation looks substantially similar to a needle to assist in identifying the needle in the image. For example, the needle representation can be a line or a needle-shaped object.

[0061] In one example, the needle representation is a representation of a predicted needle trajectory. For example, the computing device generates a graphical representation of the predicted needle trajectory. In one example, the predicted needle trajectory representation is an arrow, line, or other representation of the predicted needle trajectory. In one example, the predicted needle trajectory representation extends to or is positioned at an intersection with a nodule. In one example, the predicted needle trajectory representation extends to or is positioned at an edge of the composite image. For example, the predicted needle trajectory representation extends to or is positioned at an upper or distal edge of the composite image depending on the angle and position of the needle. In one example, the predicted needle trajectory representation is positioned a predetermined distance from the needle representation. For example, when the computing device moves the needle representation based on needle extension or retraction, the computing device moves the predicted needle trajectory representation a fixed distance from the needle representation. In one example, the computing device generates multiple predicted needle trajectory representations. For example, the multiple predicted needle trajectory representations are positioned along the same predicted needle trajectory, but the individual representations are positioned along that trajectory. In another example, the computing device determines a predicted distance between the EBUS exit port and the node. For example, the computing device uses the predicted distance to determine the length the needle should be extended from the EBUS. In one example, the computing device determines the horizontal distance between the predicted needle trajectory and the node. Thus, the computing device determines how far the EBUS needs to move so that the predicted needle trajectory (and therefore the needle) intersects the node.

[0062] At 714, the computing device displays an image. Despite the nodule being positioned too deep into the tissue during collection of the tissue sample to image both the nodule and the needle within a single FOV, the computing device generates a composite image with a modified FOV that includes the nodule and the needle. For example, the computing device displays the composite image. In one example, before the nodule leaves the FOV (e.g., of the first image), the first image is displayed in real time without being modified by the processing described herein. In one example, the computing device allows a user to switch between multiple image display modes. For example, the computing device can include a first mode that generates a composite image when the nodule is detected beyond a certain depth and a second mode that displays an unmodified real-time image. In one example, the computing device displays an image that includes the second image and a marking indicating the location of the nodule. For example, the computing device places a marking indicating the location of the nodule outside the second image. For example, the computing device generates a marking shaped substantially like a nodule, a round marking, an oval-shaped marking, or other shape to indicate the nodule. As the needle representation is generated, the computing device displays the needle representation over the image. As the needle continues to be extended or retracted, the needle representation is dynamically updated in real time over the composite image. In one example, the computing device displays the predicted distance between the EBUS exit port and the nodule, the predicted length the needle needs to be extended to reach the nodule, and / or the distance the EBUS needs to travel for the needle to intersect the nodule. For example, displaying the predicted distance can assist a user in determining how far to extend the needle from the EBUS. Thus, the computing device can assist in identifying when the nodule has been reached for sampling.

[0063] Method 700 operates on a computing device in real time and continuously throughout the procedure. For example, while the needle is extended into the EBUS FOV, a second image continues to be acquired, and method 700 operates in real time to generate and display a composite image using the second image and the first image while displaying the nodule in relation to the needle. As another example, a first image would continue to be acquired until a second image without the nodule is identified. As another example, a first image would continue to be acquired until a second image in which the needle trajectory intersects the nodule is identified. The first and second images can be continuously received, and new composite images can be displayed throughout the procedure. Thus, the computing device continually displays the imaged biopsy target (e.g., the nodule) even after the biopsy target is no longer within the EBUS FOV.

[0064] The steps or operations of methods 600 and 700 are illustrated in a particular order for convenience and clarity. Many of the operations discussed can be performed in different sequences or in parallel without significantly affecting other operations. Methods 600 and 700 as discussed include operations performed by multiple different actors, devices, and / or systems. It will be understood that a subset of the operations discussed in methods 600 and 700 can be attributed to a single actor, device, or system and can be considered a separate, standalone process or method.

[0065] While the above invention is discussed with reference to endoscopic applications, the same concepts can be applied to endoscopic, ultrasound, and microscopic diagnostic applications. The system described herein is capable of transitioning between AI modules within a procedure, attaching metadata identifying which AI module is operating on different portions of the video stream. The system supports modular insertion of AI modules as functionality is added or updated. The system can generate reports detailing which algorithms (modules) are selected where and why throughout the procedure.

[0066] FIG. 8 illustrates a block diagram of an example machine 800 on which any one or more of the techniques (processes) discussed herein may be implemented according to some embodiments. In alternative embodiments, machine 800 may operate as a standalone device and / or may be connected (e.g., networked) to other machines. In a networked deployment, machine 800 may operate in a server-client network environment in the capacity of a server machine, a client machine, or both. In one example, machine 800 may act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. Machine 800 may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, web appliance, network router, switch, or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by the machine. Additionally, although only a single machine is illustrated, the term "machine" shall also be construed to include any collection of machines individually or collectively executing a set (or sets) of instructions to implement any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations, etc.

[0067] The machine (e.g., a computer system) 800 may include a hardware processor 802 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 804, and a static memory 806, some or all of which may communicate with each other via an interlink (e.g., a bus) 808. The machine 800 may further include a display unit 810, an alphanumeric input device 812 (e.g., a keyboard), and a user interface (UI) navigation device 814 (e.g., a mouse). In one example, the display unit 810, the input device 812, and the UI navigation device 814 may be touchscreen displays. The machine 800 may additionally include a storage device (e.g., a drive unit) 816, a signal generation device 818 (e.g., a speaker), a network interface device 820, and one or more sensors 821, such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. The machine 800 may include an output controller 828, such as a serial (e.g., Universal Serial Bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection, capable of communicating with and / or controlling one or more peripheral devices (e.g., printers, card readers, etc.).

[0068] Storage device 816 may include machine-readable medium 822 on which one or more sets of data structures or instructions 824 (e.g., software) that embody or are utilized by any one or more of the techniques or functions described herein are stored. Also, instructions 824 may reside, completely or at least partially, in main memory 804, in static memory 806, or in hardware processor 802 during execution thereof by machine 800. In one example, one or any combination of hardware processor 802, main memory 804, static memory 806, or storage device 816 may constitute a machine-readable medium.

[0069] While machine-readable medium 822 is illustrated as a single medium, the term "machine-readable medium" can include a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) configured to store one or more instructions 824. The term "machine-readable medium" can include any medium that can store, encode, or carry instructions for execution by machine 800, any medium that causes machine 800 to perform any one or more of the techniques of this disclosure, or any medium that can store, encode, or carry data structures used by or associated with such instructions. Non-limiting examples of machine-readable media can include solid-state memory, and optical and magnetic media.

[0070] The instructions 824 may further be transmitted or received over a communications network 826 using a transmission medium via a network interface device 820 utilizing any one of a number of transport protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Exemplary communications networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile telephone network (e.g., a cellular network), a Plain Old Telephone (POTS) network, a wireless data network (e.g., the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards known as WiFi®, the IEEE 802.16 family of standards known as WiMax®), the IEEE 802.15.4 family of standards, and a peer-to-peer (P2P) network, among others. In one example, network interface device 820 includes one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas and is capable of connecting to communications network 826. In one example, network interface device 820 includes multiple antennas and is capable of communicating wirelessly using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term “transmission medium” shall be taken to include any non-tangible medium capable of storing, encoding, or carrying instructions for execution by machine 800, including digital or analog communications signals or other non-tangible media for facilitating the communication of such software.

[0071] The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are also referred to herein as "examples." Such examples may include elements in addition to those shown or described. However, the inventors also contemplate examples in which only those elements shown or described are provided. Moreover, the inventors also contemplate examples that use any combination or permutation of those elements (or one or more aspects thereof) shown or described with respect to a particular example (or one or more aspects thereof), or with respect to any other example (or one or more aspects thereof) shown or described herein.

[0072] Example The following examples provide a non-limiting array of examples of the present disclosure.

[0073] Example 1 is a method for real-time combination of ultrasound image streams. The method includes receiving at least one first image of a first field of view of a patient via the ultrasound image stream, the first field of view including a nodule; receiving a plurality of second images of a second field of view of the patient via the ultrasound image stream, the plurality of second images being received after the at least one first image, the second field of view being proximal to the first field of view, and the second field of view not including the nodule; generating a plurality of composite images by combining one of the plurality of second images with at least a portion of at least one first image, the at least a portion of the at least one first image including the nodule; and displaying the plurality of composite images substantially in real time as the plurality of second images are received. In the context of these examples, "substantially in real time" is intended to mean as quickly as processing circuitry can produce the composite image.

[0074] In Example 2, the subject matter of Example 1 optionally includes determining a location of the nodule relative to at least one of the plurality of second images based on an image signature of the nodule.

[0075] In Example 3, the subject matter of Example 1 or 2 optionally includes determining a location of the nodule relative to at least one of the plurality of second images based on user input indicating the nodule received via the user device.

[0076] In Example 4, the subject matter of any one of Examples 1 to 3 optionally includes a step of determining a location of the nodule for at least one of the plurality of second images by determining a depth of the nodule in the at least one first image based on known field of view coordinates, wherein the depth of the nodule is such that the trajectory of the needle does not intersect with the nodule in the first field of view.

[0077] In Example 5, the subject matter of any one of Examples 1 to 4 optionally includes determining a location of the nodule relative to at least one of the plurality of second images by determining a location of the nodule relative to one of the plurality of second images based on a position of at least one first image relative to one of the plurality of second images in the composite image and based on known field of view coordinates.

[0078] In Example 6, the subject matter of any one of Examples 1 to 5 optionally includes determining a location of the nodule relative to at least one of the plurality of second images by comparing the set of navigation data of the at least one first image with the set of navigation data of one of the plurality of second images.

[0079] In Example 7, the subject matter of any one of Examples 1 to 6 optionally includes, wherein generating the plurality of composite images by combining one of the plurality of second images with at least a portion of the at least one first image includes positioning the at least one first image and one of the plurality of second images by identifying and matching similar landmarks between the at least one first image and one of the plurality of second images.

[0080] In Example 8, the subject matter of any one of Examples 1 to 6 optionally includes, wherein generating the plurality of composite images by combining one of the plurality of second images with at least a portion of the at least one first image includes positioning the at least one first image and one of the plurality of second images by determining a distance between the at least one first image and one of the plurality of second images using the set of navigation data of the at least one first image and the set of navigation data of one of the plurality of second images.

[0081] In Example 9, the subject matter of any one of Examples 1 to 8 optionally includes identifying that the needle is extended in a second field of view of one of the plurality of second images.

[0082] In Example 10, the subject matter of any one of Examples 1 to 9 optionally includes generating a needle representation.

[0083] In Example 11, the subject matter of any one of Examples 1 to 10 optionally includes, wherein the representation of the needle indicates a length to which the needle is extended.

[0084] In Example 12, the subject matter of any one of Examples 1 to 11 optionally includes, wherein displaying the plurality of composite images includes displaying a needle representation over the plurality of composite images.

[0085] In Example 13, the subject matter of any one of Examples 1 to 12 optionally includes generating a predicted needle trajectory representation.

[0086] In Example 14, the subject matter of any one of Examples 1 to 13 optionally includes that the step of displaying the plurality of composite images includes the step of displaying a predicted needle trajectory representation on the plurality of composite images.

[0087] In Example 15, the subject matter of any one of Examples 1 to 14 optionally includes that the step of displaying the predicted needle trajectory representation on the plurality of composite images includes the step of displaying the predicted needle trajectory representation intersecting the nodule.

[0088] In Example 16, the subject matter of any one of Examples 1 to 15 optionally includes wherein generating the plurality of composite images by combining one of the plurality of second images with at least a portion of the at least one first image includes updating one of the plurality of second images in real time.

[0089] Example 17 is a method for real-time combination of ultrasound image streams. The method includes receiving at least one first image of a first field of view of a patient via the ultrasound image stream, the first field of view including a nodule, receiving at least one second image of a second field of view of the patient via the ultrasound image stream, the second field of view being proximal to the first field of view, receiving a plurality of third images of a third field of view of the patient via the ultrasound image stream, the plurality of third images being received after the at least one first image and the at least one second image, the third field of view being proximal to the second field of view, generating a plurality of composite images by combining the plurality of third images with at least a portion of the at least one first image and at least a portion of the at least one second image, and displaying the plurality of composite images in substantially real time as the plurality of third images are received.

[0090] In Example 18, the subject matter of Example 17 optionally includes determining a location of the nodule relative to one of the plurality of third images.

[0091] In Example 19, the subject matter of any one of Examples 17 or 18 optionally includes identifying that the needle is extended in a third field of view of one of the plurality of third images.

[0092] In Example 20, the subject matter of any one of Examples 17 to 19 optionally includes a step of generating a predicted needle trajectory representation, and the step of displaying the multiple composite images includes a step of displaying the needle representation on the multiple composite images.

[0093] Example 21 is an internal medical imaging system. The system includes an imaging sensor configured to acquire medical images of an internal portion of a patient, a display device configured to display image data and associated graphical information from the imaging sensor, and a computing device including a processor and a memory device, the memory device including instructions that, when executed by the processor, cause the computing device to perform operations including receiving from the imaging sensor at least one first image of a first field of view of the patient, the first field of view including a nodule, receiving from the imaging sensor a plurality of second images of a second field of view of the patient, the plurality of second images being received after the at least one first image and the second field of view being proximal to the first field of view, generating a plurality of composite images by combining one of the plurality of second images with at least a portion of at least one first image, generating the plurality of composite images by combining one of the plurality of second images with at least a portion of the at least one first image, and displaying the plurality of composite images in substantially real time via the display device as the plurality of second images are received. The system of Example 21 can optionally implement any of the methods of Examples 1 to 20.

[0094] In Example 22, the subject matter of Example 21 optionally includes determining a location of the nodule relative to one of the plurality of second images.

[0095] In Example 23, the subject matter of any one of Examples 21 or 22 optionally includes identifying that the needle is extended in a second field of view of one of the plurality of second images.

[0096] In Example 24, the subject matter of any one of Examples 21 to 23 optionally includes receiving at least one third image of a third field of view of the patient from the imaging sensor, the third field of view being between the first field of view and the second field of view, and generating the plurality of composite images includes combining the plurality of third images with at least a portion of the at least one first image and at least a portion of the at least one second image.

[0097] In Example 25, the subject matter of any one of Examples 21 to 24 optionally includes generating a predicted needle trajectory representation, and displaying the plurality of composite images includes displaying the needle representation on top of the plurality of composite images.

[0098] Example 26 is a machine-readable medium having instructions stored thereon that, when executed by a machine, cause performing operations including receiving at least one first image of a first field of view of a patient via an ultrasound image stream, the first field of view including a nodule; receiving a plurality of second images of a second field of view of the patient via the ultrasound image stream, the plurality of second images being received after the at least one first image, the second field of view being proximal to the first field of view, and the second field of view not including the nodule; generating a plurality of composite images by combining one of the plurality of second images with at least a portion of the at least one first image, the at least a portion of the at least one first image including the nodule; and displaying the plurality of composite images in substantially real time during the step of receiving the plurality of second images.

[0099] In Example 27, the subject matter of Example 26 optionally includes instructions for determining a location of the nodule relative to at least one of the plurality of second images based on an image signature of the nodule.

[0100] In Example 28, the subject matter of Example 26 also optionally includes instructions for determining a location of the nodule relative to at least one of the plurality of second images based on user input indicating the nodule received via the user device.

[0101] In Example 29, the subject matter of Example 26 also optionally includes instructions for determining a location of the nodule for at least one of the plurality of second images by determining a depth of the nodule in the at least one first image based on known field of view coordinates, the depth of the nodule being such that the needle trajectory does not intersect with the nodule in the first field of view.

[0102] In Example 30, the subject matter of Example 26 optionally also includes instructions for determining a location of the nodule relative to at least one of the plurality of second images by determining a location of the nodule relative to one of the plurality of second images based on a position of the at least one first image relative to one of the plurality of second images in the composite image and based on the known field of view coordinates.

[0103] In Example 31, the subject matter of Example 26 optionally also includes instructions for determining a location of the nodule for at least one of the plurality of second images by comparing the set of navigation data of the at least one first image with the set of navigation data of one of the plurality of second images.

[0104] In Example 32, the subject matter of Example 26 optionally also includes, wherein the instructions for generating a plurality of composite images by combining one of the plurality of second images with at least a portion of the at least one first image include positioning the at least one first image and one of the plurality of second images by identifying and matching similar landmarks between the at least one first image and one of the plurality of second images.

[0105] In Example 33, the subject matter of Example 26 optionally also includes that the instructions for generating a plurality of composite images by combining one of the plurality of second images with at least a portion of the at least one first image include positioning the at least one first image and one of the plurality of second images by determining a distance between the at least one first image and one of the plurality of second images using the set of navigation data of the at least one first image and the set of navigation data of one of the plurality of second images.

[0106] In Example 34, the subject matter of Example 26 optionally also includes instructions for identifying that the needle is extended in the second field of view of one of the plurality of second images.

[0107] In Example 35, the subject matter of any one of Examples 26 to 34 optionally includes instructions for generating a needle representation.

[0108] In Example 36, the subject matter of any one of Examples 26 to 35 optionally includes instructions where the needle representation indicates a length to which the needle is extended.

[0109] In Example 37, the subject matter of any one of Examples 26 to 36 optionally includes, wherein the instructions for displaying the plurality of composite images include displaying a needle representation over the plurality of composite images.

[0110] In Example 38, the subject matter of any one of Examples 26 to 37 optionally includes instructions for generating a representation of the predicted needle trajectory.

[0111] In Example 39, the subject matter of any one of Examples 26 to 38 optionally includes that the instructions for displaying the plurality of composite images include displaying a predicted needle trajectory representation over the plurality of composite images.

[0112] In Example 40, the subject matter of any one of Examples 26 to 39 optionally includes that the instructions for displaying a predicted needle trajectory representation on a plurality of composite images include a step of displaying a predicted needle trajectory representation intersecting the nodule.

[0113] In Example 41, the subject matter of any one of Examples 26 to 40 optionally includes that the instructions for generating a plurality of composite images by combining one of the plurality of second images with at least a portion of at least one first image include updating one of the plurality of second images in real time.

[0114] Example 42 is a machine-readable medium having instructions stored thereon that, when executed by a machine, cause the instructions to perform operations including receiving, via the ultrasound image stream, at least one first image of a first field of view of the patient, the first field of view including a nodule; receiving, via the ultrasound image stream, at least one second image of a second field of view of the patient, the second field of view being proximal to the first field of view; receiving, via the ultrasound image stream, a plurality of third images of a third field of view of the patient, the plurality of third images being received after the at least one first image and the at least one second image, the third field of view being proximal to the second field of view; generating a plurality of composite images by combining the plurality of third images with at least a portion of the at least one first image and at least a portion of the at least one second image; and displaying the plurality of composite images in substantially real time as the plurality of third images are received.

[0115] In Example 43, the subject matter of Example 42 includes instructions for determining a location of the nodule relative to one of the plurality of third images.

[0116] In example 44, the subject matter of example 42 includes instructions for identifying that the needle is extended in a third field of view of one of the plurality of third images.

[0117] In Example 45, the subject matter of Examples 42 and 44 includes instructions for generating a predicted needle trajectory representation, and the step of displaying the plurality of composite images includes the step of displaying the needle representation over the plurality of composite images.

[0118] In the event of a conflicting usage between this document and any document so incorporated by reference, the usage in this document shall control. In this document, the terms "including" and "in which" are used as the plain English equivalents of the respective terms "comprising" and "wherein." Also, in the appended claims, the terms "including" and "comprising" are open-ended, i.e., systems, devices, articles, compositions, formulations, or processes that include elements in addition to those recited after such terms in a claim are still deemed to be within the scope of that claim.

[0119] In this document, the terms "a" or "an" are used, as is common in patent documents, to include one or more, regardless of any other instance or usage of "at least one" or "one or more." In this document, the term "or" is used to refer to non-exclusiveness, or "A or B" is used to include "A but not B," "B but not A," and "A and B," unless otherwise indicated. In this document, the terms "including" and "in which" are used as the plain-English equivalents of the respective terms "comprising" and "wherein." Also, in the appended claims, the terms "including" and "comprising" are open-ended, i.e., systems, devices, articles, compositions, formulations, or processes that include elements in addition to those recited after such terms in a claim are still deemed to be within the scope of that claim. Moreover, in the appended claims, terms such as "first," "second," and "third" are used merely as labels and are not intended to impose numerical requirements on their objects.

[0120] The above description is intended to be illustrative, not limiting. For example, the above-described examples (or one or more aspects thereof) can be used in combination with each other. Other embodiments can be used, for example, by one of ordinary skill in the art upon reviewing the above description. The Abstract is provided to comply with 37 CFR Chapter 1.72(b) to allow the reader to quickly ascertain the nature of the technical disclosure. The Abstract has been submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the above Detailed Description, various features may be grouped together to streamline the disclosure. This should not be construed as intending that an unclaimed disclosed feature is essential to any claim. Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Thus, it is contemplated that the following claims are incorporated into the Detailed Description as an example or embodiment, with each claim standing on its own as a separate embodiment, and that such embodiments can be combined with each other in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. [Explanation of symbols]

[0121] 100 EBUS102 schematic diagram 102 EBUS 104 Image sensor 106 Field of view (FOV) 108 Sampling needle 110 Sampling needle trajectory 120 Airway Channel 122 Airway Tissue Wall 124 Nodules 150 Schematic diagram of tracking of sampling needle 108 of EBUS 102 152 Sampling needle representation 154 Sampling needle trajectory representation 200 EBUS102 schematic diagram 202 Nodules 206 FOV 210 EBUS102 schematic diagram 212 Nodules 214 Current FOV 216 Previous FOV 250 Schematic diagram of EBUS102 sampling needle tracking 252 First sampling needle representation 254 Second Sampling Needle Representation 256 Sampling needle trajectory representation 258 Sampling needle trajectory representation 300 1st position 302 Nodules 304 Current FOV, 4th FOV 306a Overlapped part 306b Overlapped part 306c Overlapped part 308 Non-overlapping portion 310a Composite Image 310b Composite Image 310c composite image 312 First FOV 314 Second FOV 316 Third FOV 320 Second Position 330 Third Position 340 Fourth Position 800 machines 802 Hardware Processor 804 main memory 806 static memory 808 Interlink 810 Display Unit 812 alphanumeric input device 814 User Interface (UI) Navigation Devices 816 Storage Devices 818 Signal Generating Device 820 Network Interface Device 821 Sensor 822 Machine-readable medium 824 command 826 Communication Network 828 Output Controller D1 First distance D2 Second distance

Claims

1. 1. A method for real-time combination of ultrasound image streams, comprising: receiving at least one first image of a first field of view of a patient via the ultrasound image stream, the first field of view including a nodule; receiving, via the ultrasound image stream, a plurality of second images of a second field of view of the patient, the plurality of second images being received after the at least one first image, the second field of view being proximal to the first field of view, and the second field of view not including the nodule; generating a plurality of composite images by combining one of the plurality of second images with at least a portion of the at least one first image, the at least a portion of the at least one first image including the nodule; displaying the plurality of composite images substantially in real time during the step of receiving the plurality of second images. A method comprising:

2. The method of claim 1 , further comprising determining a location of the nodule relative to at least one of the plurality of second images based on an image signature of the nodule.

3. 10. The method of claim 1, further comprising determining a location of the nodule relative to at least one of the plurality of second images based on user input indicating the nodule received via a user device.

4. 2. The method of claim 1, further comprising determining a location of the nodule relative to at least one of the plurality of second images by determining a depth of the nodule in the at least one first image based on known field of view coordinates, wherein the depth of the nodule is such that the trajectory of the needle does not intersect with the nodule in the first field of view.

5. 2. The method of claim 1, further comprising determining a location of the nodule relative to at least one of the plurality of second images by determining the location of the nodule relative to the one of the plurality of second images based on the position of the at least one first image relative to the one of the plurality of second images in the composite image and based on known field of view coordinates.

6. 2. The method of claim 1, further comprising determining a location of the nodule relative to at least one of the plurality of second images by comparing a set of navigation data of the at least one first image with a set of navigation data of one of the plurality of second images.

7. 2. The method of claim 1, wherein generating a plurality of composite images by combining one of the plurality of second images with at least a portion of the at least one first image comprises positioning the at least one first image and one of the plurality of second images by identifying and matching similar landmarks between the at least one first image and one of the plurality of second images.

8. 2. The method of claim 1, wherein generating a plurality of composite images by combining one of the plurality of second images with at least a portion of the at least one first image includes positioning the at least one first image and one of the plurality of second images by determining a distance between the at least one first image and one of the plurality of second images using a set of navigation data of the at least one first image and a set of navigation data of one of the plurality of second images.

9. The method of claim 1 , further comprising identifying that a needle is extended in the second field of view of one of the plurality of second images.

10. The method of claim 9 further comprising generating a needle representation.

11. The method of claim 10 , wherein the needle representation indicates the length to which the needle is extended.

12. The method of claim 10 , wherein displaying the plurality of composite images comprises displaying the needle representation over the plurality of composite images.

13. The method of claim 9 further comprising generating a predicted needle trajectory representation.

14. The method of claim 13 , wherein displaying the plurality of composite images comprises displaying the predicted needle trajectory representation over the plurality of composite images.

15. The method of claim 14 , wherein displaying the predicted needle trajectory representation on the plurality of composite images comprises displaying the predicted needle trajectory representation intersecting the nodule.

16. 2. The method of claim 1, wherein generating a plurality of composite images by combining one of the plurality of second images with at least a portion of the at least one first image comprises updating one of the plurality of second images in real time.

17. 1. A method for real-time combination of ultrasound image streams, comprising: receiving at least one first image of a first field of view of a patient via the ultrasound image stream, the first field of view including a nodule; receiving at least one second image of a second field of view of the patient via the ultrasound image stream, the second field of view being proximal to the first field of view; receiving a plurality of third images of a third field of view of the patient via the ultrasound image stream, the plurality of third images being received after the at least one first image and the at least one second image, the third field of view being proximal to the second field of view; generating a plurality of composite images by combining the plurality of third images with at least a portion of the at least one first image and at least a portion of the at least one second image; displaying the plurality of composite images substantially in real time during the step of receiving the plurality of third images. A method comprising:

18. The method of claim 17 , further comprising determining a location of the nodule relative to one of the plurality of third images.

19. The method of claim 17 , further comprising identifying that a needle is extended in the third field of view of one of the plurality of third images.

20. 20. The method of claim 19, further comprising generating a predicted needle trajectory representation, wherein displaying the plurality of composite images comprises displaying the needle representation over the plurality of composite images.

21. 1. An internal medical imaging system comprising: an imaging sensor configured to acquire medical images within an interior portion of a patient; a display device configured to display image data and associated graphical information from the imaging sensor; a computing device including a processor and a memory device; Including, The memory device contains instructions that, when executed by the processor, cause the computing device to perform operations, such as: receiving at least one first image of a first field of view of the patient from the imaging sensor, the first field of view including a nodule; receiving, from the imaging sensor, a plurality of second images of a second field of view of the patient, the plurality of second images being received after the at least one first image, the second field of view being proximal to the first field of view; generating a plurality of composite images by combining one of the plurality of second images with at least a portion of the at least one first image; generating a plurality of composite images by combining one of the plurality of second images with at least a portion of the at least one first image; displaying the plurality of composite images substantially in real time upon receiving the plurality of second images via the display device; and an internal medical imaging system, including:

22. 22. The internal medical imaging system of claim 21, wherein the instructions further cause the computing device to perform an operation including determining a location of the nodule relative to one of the plurality of second images.

23. 22. The internal medical imaging system of claim 21 , wherein the instructions further cause the computing device to perform an operation including identifying that a needle is extended in the second field of view of one of the plurality of second images.

24. The instructions further cause the computing device to perform operations including receiving at least one third image of a third field of view of the patient from the imaging sensor, the third field of view being between the first field of view and the second field of view; 22. The internal medical imaging system of claim 21 , wherein generating the plurality of composite images includes combining the plurality of third images with at least a portion of the at least one first image and at least a portion of the at least one second image.

25. The instructions further cause the computing device to perform operations including generating a predicted needle trajectory representation; 24. The internal medical imaging system of claim 23, wherein displaying the plurality of composite images includes displaying the needle representation over the plurality of composite images.

Images